← All drillsBuild a Concurrent Link Checker
What you'll be able to do- ✓Explain what the event loop does, and why async speeds up waiting but not computing
- ✓Bound concurrency with a semaphore, and find the right limit by measurement rather than by guessing
- ✓Put a deadline on every network call, and cancel work cleanly without swallowing CancelledError
- ✓Retry only failures that can change, with exponential backoff, jitter and Retry-After
- ✓Build a queue-based crawler, and recognise the deadlock a bounded frontier causes
- ✓Add per-host limits, a rate limit and robots.txt handling, and report an unreachable site as unreachable
- ✓Handle errors with TaskGroup and ExceptionGroup, leaving no orphan tasks and keeping partial results on Ctrl-C
- ✓Recognise a blocked event loop, move CPU work off it, and say when async is the wrong tool
- ✓Test async code deterministically with a fake clock and a mock network, and against a seeded ground truth
Build a concurrent network tool that is fast, polite and survives hung hosts and rate limits — and know, with numbers, when async is the right answer and when it is not.
⌁ Link checkers and crawlers (lychee, Screaming Frog, the W3C checker, every search crawler) are this shape: a bounded, polite, retrying frontier. The same patterns run any high-fanout API client or webhook dispatcher.
Start this internshipCreate an account to unlock the 11 sections, the workbench, and askThili.
BeginSections
1. Build a Concurrent Link Checker
🔒 locked2. Lesson 1 - The Synchronous Baseline
🔒 locked3. Lesson 2 - Coroutines and the Event Loop
🔒 locked4. Lesson 3 - Bounded Concurrency
🔒 locked5. Lesson 4 - Timeouts and Cancellation
🔒 locked6. Lesson 5 - Retries Done Right
🔒 locked7. Lesson 6 - The Crawl Frontier
🔒 locked8. Lesson 7 - Per-Host Politeness
🔒 locked9. Lesson 8 - Structured Concurrency
🔒 locked10. Lesson 9 - When Async Is the Wrong Tool
🔒 locked11. Lesson 10 - Test It, Then Break It
🔒 lockedDig deeper
🔗PEP 492 — Coroutines with async and await syntax
docs🔗PEP 654 — Exception Groups and except*
docs🔗asyncio — official documentation (TaskGroup, timeout, Queue, Semaphore)
docs🔗HTTPX — Async support and timeouts
docs🔗RFC 9309 — Robots Exclusion Protocol
docs🔗RFC 9110 §10.2.3 — Retry-After
docs🔗Exponential Backoff And Jitter (AWS Architecture Blog)
docs🔗thilicrawl — the reference implementation you build in this course
code