Do Not Crawl a Website Before You Know What Is in It

The first version of a website ingestion workflow usually has one setting: crawl everything.

I understand the temptation. If an agent might need a page later, why not collect it now?

Because most websites contain far more than the agent needs. Marketing archives, tag pages, translated duplicates, old announcements, legal text, and navigation loops all compete with the useful documentation. A bigger corpus is not automatically a better one.

I would rather see the shape of the site first.

Discovery should be a separate step

2md Map does one job. It starts from a URL, follows links to a chosen depth, and returns the site structure without converting the pages.

That separation is the useful part.

You can set a page limit, inspect the tree, and export the result as text, Markdown, or JSON. Nothing has entered the knowledge base yet. Nobody has spent tokens cleaning pages that will be discarded five minutes later.

I ran it against Jevvellabs with a depth of one and a ten-page limit. It returned ten URLs and the corresponding tree. That was enough to spot the profile, blog, category pages, and recent articles before deciding what an agent-facing bundle should contain.

No account or API key was needed for that discovery run.

The agent integration is almost boring

Map uses the existing crawl endpoint with treeOnly enabled:

{
  "url": "https://example.com",
  "depth": 2,
  "maxPages": 50,
  "treeOnly": true
}

That makes it easy to place in front of a RAG ingestion job or documentation agent. The agent maps the site, presents the candidate sections, and only converts the URLs that belong in scope.

The human can approve the boundary. The machine can handle the tedious link discovery.

This is much better than asking somebody to paste fifty documentation URLs into a spreadsheet.

A map is not a crawl

There is an important limit: Map only sees pages it can discover through links. It will not magically find an orphaned page, pass an authenticated area, or decide which version of a document is authoritative.

That is fine. It is a planning tool, not an oracle.

The output answers three practical questions:

• How large is this site likely to be?
• Which sections deserve extraction?
• What should stay outside the agent’s context?

Those questions should come before conversion, embedding, or indexing.

Smaller context starts with a better boundary

Teams spend a lot of effort tuning chunk sizes and retrieval thresholds after ingestion. Sometimes the larger improvement is to stop irrelevant pages from entering the corpus at all.

Map makes that decision visible. You can review the structure, argue about scope, and keep a record of what was included.

It is a small tool. That is why it fits so well in an agentic workflow.

The agent does not need another giant platform. It needs a cheap way to ask, “What is here?” before it decides what to learn.