Web Crawler
Crawl billions of pages politely, avoid duplicates, and keep content fresh.
Medium45 minURL frontierPolitenessDedup
A web crawler downloads pages, extracts links, and keeps going, building the raw corpus for a search engine. The difficulty is not fetching a page; it is crawling billions of pages politely, avoiding duplicates and traps, and deciding what to fetch next.
Clarify requirements
Functional
- Start from seed URLs, fetch pages, extract links, and enqueue new URLs.
- Store page content (HTML) and metadata for downstream indexing.
- Respect robots.txt and per-site crawl delays.
- Recrawl pages periodically based on how often they change.
- Scope: HTML only (mention images/PDFs as extensions).
Non-functional
- Crawl 1B pages per week (plus recrawls).
- Politeness: at most one concurrent connection per host and a delay between requests.
- Robust to malformed HTML, slow servers, spider traps.
- Scalable horizontally; resumable after failures.
Try it first
Set a timer, sketch your own design on paper, then compare it with the reference walkthrough.
45:00
Defend it out loud
The AI interviewer will probe your design for Web Crawler and score it.