Official source
1 How Google Search crawls pages
How does Google Search crawl pages? This walk-through goes from finding URLs to fetching and rendering them, and explains what Googlebot is and why sitemaps can help.
Older: check current guidanceTopic
Googlebot
10 videos.
Googlebot is the program Google uses to fetch pages so they can be added to search. Site owners see it in their server logs. Developers care because it decides what content Google gets to see, including on sites built with JavaScript.
For a short official introduction, try How Google Search crawls pages. Advanced readers who debug how a page looks to a crawler could try Giacomo Zecchini on rendering and layout problems. Martin Splitt of Google appears in several talks here, aimed at developers. A group of them are from 2019, the year Googlebot moved to an up to date version of Chrome, so some rendering details may have moved on. The log file lessons come from software companies, which sell tools for the job.
Order
First steps
Once the basics are clear
Trade press or conference
2 Google's Martin Splitt on how the search engine crawls and indexes - Live with Search Engine Land
How Google crawls and indexes pages is the topic as Barry Schwartz of Search Engine Land puts live questions to Martin Splitt of Google about the technical side of search.
Older: check current guidanceSells in this field
3 Jamie Alberico - Think Like a Bot, Rank Like a Boss: How Googlebot Renders
Why might content on a page fail to rank? Jamie Alberico explains Googlebot rendering, the difference between the page a visitor sees and the one Google processes.
Older: check current guidanceTrade press or conference
4 Martin Splitt | Technical SEO for JS Web Apps | ComponentsConf 2019
Martin Splitt explains to web developers how Googlebot and other crawlers see JavaScript pages, and which common pitfalls to avoid when building for search.
Older: check current guidanceSells in this field
5 How to Do Log File Analysis | Lesson 4/7 | SEMrush Academy
Log file analysis gets a short lesson in Semrush Academy's course, part four of seven, using the Log File Analyzer and its Googlebot activity graph to see how Googlebot crawls a site.
Older: check current guidance
Going further
Official source
6 How to make your content shine on Google Search (Chrome Dev Summit 2019)
Martin Splitt looks inside the evergreen Googlebot and goes through rendering best practice and debugging steps for developers who want their content found in Search.
Older: check current guidanceSells in this field
7 Webinar: "Rendering SEO" with Martin Splitt (Google)
Martin Splitt of Google joins Bartosz Goralewicz and Tomek Rudzki to talk rendering SEO, the future of technical optimisation and why SEO should look beyond JavaScript.
Older: check current guidanceTrade press or conference
8 Rendering, Style, and Layout When Things Go Wrong - Giacomo Zecchini
Why can crawlers see a page differently from users? Giacomo Zecchini explains rendering, style and layout, Googlebot's viewport expansion and implementation mismatches that hide content.
Sells in this field
9 Mastering Bot Logs for Effective Crawl Budget Management - JetOctopus Webinar
Crawl budget management through bot logs gets a JetOctopus webinar, covering how to block malicious bots and check log health, from load times to status codes.
Older: check current guidanceTrade press or conference
10 When Server Logs Lie: The Wire-Level Truth Behind Googlebot’s Ghost Errors - Jori Ford
Googlebot errors in server logs may not be what they seem, according to Jori Ford, who explains what happens at the wire level to produce so-called ghost errors.
