Topic
7 videos.
How does a search engine find the right documents among billions? Information retrieval is the field of computer science that studies this. It covers how text is stored, how a query is matched against it and how results are scored. Search engines are built on its ideas, so knowing a little of it helps when SEO talk turns technical.
Most of the talks here come from university courses, so expect theory more than website advice. 2.6 Inverted index is a short lecture on the structure behind search. A no nonsense intro to BM25 is a technical piece with code on one scoring formula.
For the marketing angle, Mike King on ARGDW is a conference session that argues many SEO tools lean on older methods.
Sells in this field
Mike King argues that search now works on meaning while many SEO tools still lean on BM25 and TF-IDF, and goes on to relevance engineering and fan-out queries.
Sells in this field
Dawn Anderson explains what the AI era changes for crawling, indexing and technical SEO, and argues RAG works from a web index rather than live data.
Teaching or conference talk
BM25 is a ranking formula for scoring how well documents match a query, and this video breaks down term frequency, document length, the k1 and b settings and a code demo.
Teaching or conference talk
How does a search engine build its index? This is the first of three parts on index construction from an ETH Zurich information retrieval course, spring 2022.
Older: check current guidanceTeaching or conference talk
The inverted index is the data structure behind search engines, and this eight minute lecture from a University of Amsterdam course explains it.
Older: check current guidanceTeaching or conference talk
Web crawlers are responsible for a large share of internet traffic, according to this lecture, which explains how they work, why crawling is hard and gives a short look at Nutch.
Older: check current guidanceTeaching or conference talk
Why is web crawling hard? This brief talk goes through being polite to servers, keeping data fresh, covering enough pages, handling scale and dodging traps.
Older: check current guidance
