Crawlspace

Why this crawler is deliberately slow

A crawler that anonymous strangers can point at any address is a denial-of-service amplifier waiting to happen. A hundred people each politely staying inside their own rate limit still add up to a hundred simultaneous crawls of whichever site they all decided to check.

So there are two limits here, not one. You get a budget, and the site being crawled gets a budget of its own regardless of who asked. There is a quarter-second pause between requests, a page ceiling, a depth ceiling, and links to other hosts are listed but never followed. A faster crawler would be easy to write and a bad thing to put on the internet.

robots.txt is honoured

Paths disallowed for * are skipped and reported. This is not a legal requirement, and plenty of crawlers ignore it — but a tool that does gets its address blocked everywhere, and then it works for nobody.

A practical consequence worth knowing: if your own robots.txt is stricter than you remember, the crawl will come back smaller than expected. That is a useful finding in itself, since search engines are reading the same file.

What the sitemap includes, and what it leaves out

Pages marked noindex are crawled but kept out of the generated sitemap. Listing a noindex page in a sitemap is a direct contradiction — you are asking a search engine to crawl something you have told it not to index — and Search Console reports it as an error.

The generated file is a starting point rather than a finished artefact. It has no priorities and no change frequencies, because search engines have said for years that they ignore both. The lastmod dates are the date of the crawl, which is honest but not useful; replace them with real modification dates if your site knows them, and leave them out entirely rather than lie.

Broken links are found from the inside

Every internal link that returns an error is reported with the page it was found on. The usual culprits are a renamed page nobody updated the menu for, a trailing slash that used to work, and a staging address that escaped into production.

Depth matters more than page count for finding these. A depth of one checks only what the homepage links to; two reaches most of a small site. Raising the page limit without raising the depth just crawls more of the same shallow layer.

Crawl a site to a set depth, produce a valid sitemap.xml, and list every link that leads nowhere. Crawling is deliberately unhurried and depth-limited: a checker that hammers the site it is checking is a worse problem than the broken links it finds.

It's one of the free tools in the util.quest collection — nothing to install, and no account needed. Found a bug or want a feature? Reach out at [email protected].