Common Crawl - Open Repository of Web Crawl Data

Latest Blog Post

The Common Crawl Foundation at the TPDL Conference

The Common Crawl Foundation participated in the 30th TPDL (Theory and Practice for Digital Archives) conference in Faro, Portugal presenting two contributions on Crawl theory and on archival data enrichment.

Michael Paris

Michael is a Senior Research Engineer at the Common Crawl Foundation.

Common Crawl maintains a free, open repository of web crawl data that can be used by anyone.

Common Crawl is a 501(c)(3) non–profit founded in 2007.‍We make wholesale extraction, transformation, and analysis of open web data accessible to researchers.

Over 300 billion pages spanning 15 years.

Free and open corpus since 2007.

Cited in over 10,000 research papers.

3–5 billion new pages added each month.

Featured Papers

Nathan Godey, et al.

Gaperon: A Peppered English-French Generative Language Model Suite

Sheriff Issaka, et al.

The African Languages Lab: A Collaborative Approach to Advancing Low-Resource African NLP

Badal Nyalang

Stereotyped by Silence: How LLMs Erase Northeast Indian Languages Through Omission and Orthographic Corruption

Xiangyu Wu, et al.

MirrorCAPTCHA: Wild CAPTCHA, Wild Distribution, Wild Web-based Platform Meet Multimodal LLM Agents

Michael Paris, Hande Celikkanat, Luca Foppiano

Measuring What the Crawler Sees: Discovery Curves, Core Persistence, and Shell Dynamics in Longitudinal Web Crawls

Julio Garbers, Terry Gregory

The Diffusion of Artificial Intelligence Across Firms: Evidence from Europe

Pedro Ortiz Suarez, et al.

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

Asier Gutiérrez-Fandiño, et al.

esCorpius: A Massive Spanish Crawling Corpus

Latest Blog Post

The Common Crawl Foundation at the TPDL Conference

The Data

Resources

Community

About

Common Crawl is a 501(c)(3) non–profit founded in 2007.
‍
We make wholesale extraction, transformation, and analysis of open web data accessible to researchers.