Challenges in Web Crawling

This video is about 6 years old. Search advice changes quickly, so check this against current guidance.

About this video

Why is web crawling hard? This brief talk goes through being polite to servers, keeping data fresh, covering enough pages, handling scale and dodging traps.

Good to know: Very short overview only

Why watch it

Three minutes on why crawling the web is hard: politeness, freshness, coverage, scale and traps. A quick way in before the longer lecture.