← All drills

Build a Concurrent Link Checker

What you'll be able to do

Build a concurrent network tool that is fast, polite and survives hung hosts and rate limits — and know, with numbers, when async is the right answer and when it is not.

⌁ Link checkers and crawlers (lychee, Screaming Frog, the W3C checker, every search crawler) are this shape: a bounded, polite, retrying frontier. The same patterns run any high-fanout API client or webhook dispatcher.
Start this internship
Create an account to unlock the 11 sections, the workbench, and askThili.
Begin

Sections

1. Build a Concurrent Link Checker
🔒 locked
2. Lesson 1 - The Synchronous Baseline
🔒 locked
3. Lesson 2 - Coroutines and the Event Loop
🔒 locked
4. Lesson 3 - Bounded Concurrency
🔒 locked
5. Lesson 4 - Timeouts and Cancellation
🔒 locked
6. Lesson 5 - Retries Done Right
🔒 locked
7. Lesson 6 - The Crawl Frontier
🔒 locked
8. Lesson 7 - Per-Host Politeness
🔒 locked
9. Lesson 8 - Structured Concurrency
🔒 locked
10. Lesson 9 - When Async Is the Wrong Tool
🔒 locked
11. Lesson 10 - Test It, Then Break It
🔒 locked

Dig deeper

🔗PEP 492 — Coroutines with async and await syntax
docs
🔗PEP 654 — Exception Groups and except*
docs
🔗asyncio — official documentation (TaskGroup, timeout, Queue, Semaphore)
docs
🔗HTTPX — Async support and timeouts
docs
🔗RFC 9309 — Robots Exclusion Protocol
docs
🔗RFC 9110 §10.2.3 — Retry-After
docs
🔗Exponential Backoff And Jitter (AWS Architecture Blog)
docs
🔗thilicrawl — the reference implementation you build in this course
code