← All drillsBuild a Document Search Engine
What you'll be able to do- ✓Build an inverted index and answer keyword queries over thousands of documents
- ✓Rank results with TF-IDF and BM25 and explain what each term in the score does
- ✓Measure a ranker with recall at k and mean reciprocal rank, with the spread across query samples, instead of eyeballing results
- ✓Test whether lowercasing, stop words, stemming, k1 and b help, and show how the choice of test queries changes the verdict
- ✓Handle typos with character n-grams and measure what that costs in memory and speed
- ✓Write a search note that states what the test cannot tell you
Build the retrieval layer under a search box or a "chat with your documents" system — an index, a ranker, and a test harness that tells you whether a change helped.
⌁ The keyword search inside Elasticsearch, Apache Lucene and Solr, and Algolia — the retrieval baseline that embedding-based and hybrid search systems are measured against.
Start this internshipCreate an account to unlock the 4 sections, the workbench, and askThili.
BeginSections
1. Build a Document Search Engine
🔒 locked2. Lesson 1 - The inverted index
🔒 locked3. Lesson 2 - Is it any good?
🔒 locked4. Lesson 3 - Real queries are messy
🔒 lockedDig deeper
📄A Statistical Interpretation of Term Specificity and Its Application in Retrieval (Sparck Jones, 1972, Journal of Documentation 28(1))
paper📄The Probabilistic Relevance Framework: BM25 and Beyond (Robertson & Zaragoza, 2009, Foundations and Trends in Information Retrieval)
paper🔗Project Gutenberg — the public-domain books the lessons search
article