The Common Crawl Foundation presented a Stanford Institute for Human-Centered Artificial Intelligence (HAI) Seminar entitled "Preserving Humanity's Knowledge and Making it Accessible: Addressing Challenges of Public Web Data". The seminar was presented to a full house at Stanford and an actively engaged audience both in-person and online. Following the seminar and Q&A, we spent several hours in follow-up conversations with attendees and meeting with partners and friends who attended.
The presentation provided an introduction to Common Crawl and our data, and covered topics around crawler politeness and the Robots Exclusion Protocol, legal and policy issues, and web data and language coverage. You can download a PDF of the presentation slides via GitHub.
We would like to thank Patrick Hynes, Stanford HAI’s Senior Manager of Research Communities, for hosting us, Professor Diyi Yang for meeting with us, and to everyone who attended.
Erratum:
Content is truncated
Some archived content is truncated due to fetch size limits imposed during crawling. This is necessary to handle infinite or exceptionally large data streams (e.g., radio streams). Prior to March 2025 (CC-MAIN-2025-13), the truncation threshold was 1 MiB. From the March 2025 crawl onwards, this limit has been increased to 5 MiB.
For more details, see our truncation analysis notebook.
.svg)
