Official source
1 How Google Search crawls pages
How does Google Search crawl pages? This walk-through goes from finding URLs to fetching and rendering them, and explains what Googlebot is and why sitemaps can help.
Older: check current guidanceTopic
Crawling
19 videos.
A page cannot appear in search until a crawler has found it and fetched it. Crawling is that first step. Sitemaps, internal links and server logs all play a part, and a crawler that wastes time on the wrong URLs can leave good pages unvisited.
Beginners can start with Google's own introduction to how it crawls pages, recorded in 2024 and quick to watch. For a recent Google conversation about sitemaps, there is Do sitemaps still matter? with Martin Splitt and John Mueller. Readers ready for logs could try Crawling, indexing and logfiles from brightonSEO. Sources range from Google Search Central to tool vendors and a university lecture on how crawlers are built. Anything from 2020 or earlier is worth checking against today's guidance.
Practise with: Tools for crawling and site audits
Order
First steps
Sells in this field
2 How Search Engine Indexing Works?
Search engine indexing is broken into crawling, indexing and ranking, followed by ways to control what gets indexed so that content is found.
Sells in this field
3 Webinar: Breaking Down Technical SEO | Wix SEO
Technical SEO problems are the focus as experts from Wix and Deepcrawl go through the fundamentals of keeping a site healthy for search engines.
Older: check current guidanceSells in this field
4 Wix | Live Webinar: How to Get Indexed in 2024
Getting pages crawled and indexed faster is the aim of this Wix webinar, where Fabrice Canel from Bing presents IndexNow and Crystal Carter answers common indexing questions.
Older: check current guidanceIndependent practitioner
5 How do you deal with Crawl Budget? | BrightonSEO 2021
Crawl budget is how many pages crawlers can get through in a set time, according to this BrightonSEO 2021 interview with Joanna Beech, which also covers how to deal with it.
Older: check current guidance
Once the basics are clear
Trade press or conference
6 Google's Martin Splitt on how the search engine crawls and indexes - Live with Search Engine Land
How Google crawls and indexes pages is the topic as Barry Schwartz of Search Engine Land puts live questions to Martin Splitt of Google about the technical side of search.
Older: check current guidanceOfficial source
7 Crawling smarter, not harder | Search Off the Record
Does crawling more often mean a better site? John Mueller, Lizzi Sassman and Gary Illyes untangle ideas about crawl frequency and site quality, and ask how crawling could be more efficient.
Official source
8 Do sitemaps still matter?
Do XML sitemaps still matter? Martin Splitt and John Mueller look back at their history, ask whether crawlers can work out site structure alone, and discuss why fetch warnings in Search Console worry owners.
Sells in this field
9 Sitemaps are often treated as an afterthought - Carlos Manuel Sánchez Donate
Sitemaps deserve more attention than they get, argues Carlos Manuel Sánchez Donate, who says they tell search engines and LLMs which parts of a site matter and suggests splitting them by content type.
Official source
10 Curing crawling chaos
Crystal Carter talks with Martin Splitt about a site that no crawling tool could crawl fully, the steps and tools used to fix it, and how monitoring can head it off.
Older: check current guidanceSells in this field
11 How to Do Log File Analysis | Lesson 4/7 | SEMrush Academy
Log file analysis gets a short lesson in Semrush Academy's course, part four of seven, using the Log File Analyzer and its Googlebot activity graph to see how Googlebot crawls a site.
Older: check current guidanceTrade press or conference
12 Crawling, indexing and logfiles: The beauty and the beast - Andor Palau - brightonSEO October 2024
Andor Palau goes through crawling, indexing and log files using real log file analysis data, with ideas for improving each area on your own site.
Trade press or conference
13 065: Optimizing “Crawl Budget” To Improve Your Web Traffic w/Vanessa Fox
Does optimising crawl budget improve web traffic? Vanessa Fox joins this podcast episode to talk about how search engines discover, crawl and index content.
Teaching or conference talk
14 IR Lecture 10: Crawlers
Web crawlers are responsible for a large share of internet traffic, according to this lecture, which explains how they work, why crawling is hard and gives a short look at Nutch.
Older: check current guidanceTeaching or conference talk
15 Design a Web Crawler: FAANG Interview Question
How would you design a web crawler for a system design interview? This walkthrough takes engineers through one possible answer.
Going further
Sells in this field
16 Log file Analysis - What You Should Be Looking For
Log file analysis for SEO gets a panel discussion, with three technical SEOs talking through what to look for in server logs, hosted by Jason Barnard.
Older: check current guidanceSells in this field
17 Internal linking: a practical approach for big websites | Serge Bezborodov | Brighton SEO April 2025
Internal linking on big websites is harder when thousands of pages are thin or duplicated, the speaker says, and links are what turn a bunch of pages into a connected site.
Sells in this field
18 News SEO is the cutting edge for all SEO - Barry Adams | #YoastCon2023
News SEO exposes a lot about how Google crawls, indexes and judges authority, argues Barry Adams, because Google must find and rank new articles quickly.
Older: check current guidanceSells in this field
19 Dawn Anderson on DSCRI, RAG, and the Rebirth of Technical SEO | Kalicube Summit 2026
Dawn Anderson explains what the AI era changes for crawling, indexing and technical SEO, and argues RAG works from a web index rather than live data.
