May 2024 Crawl Archive Now Available
June 3, 2024
The crawl archive for May 2024 is now available. The data was crawled between May 18th and May 31st, and contains 2.7 billion web pages (or 377 TiB of uncompressed content). This is our 100th crawl!
Read More...Host- and Domain-Level Web Graphs November/December 2023, February/March 2024, and April 2024
May 5, 2024
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of November, February, April 2024.
Read More...April 2024 Crawl Archive Now Available
May 1, 2024
We are pleased to announce that the crawl archive for April 2024 is now available. The data was crawled between April 12th and April 25th, and contains 2.7 billion web pages (or 386 TiB of uncompressed content). Page captures are from 47.24 million hosts or 37.65 million registered domains and include 0.98 billion new URLs not visited in any of our prior crawls.
Read More...March/April 2024 Newsletter
March 26, 2024
We're excited to share an update on some of our recent projects and initiatives in this newsletter!
Read More...Host- and Domain-Level Web Graphs September/October, November/December 2023 and February/March 2024
March 14, 2024
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of September, November, February 2023-24.
Read More...February/March 2024 Crawl Archive Now Available
March 11, 2024
The crawl archive for February/March 2024 is now available. The data was crawled between February 20th and March 5th, and contains 3.16 billion web pages (or 424.7 TiB of uncompressed content).
Read More...Web Archiving File Formats Explained
March 1, 2024
In the ever–evolving landscape of digital archiving and data analysis, it is helpful to understand the various file formats used for web crawling. From the early ARC format to the more advanced WARC, and the specialised WET and WAT files, each plays an important role in the field of web archiving. In this post, we explain these formats, exploring their unique features, applications, and the enhancements they offer.
Read More...A Further Look Into the Prevalence of Various ML Opt–Out Protocols
February 22, 2024
This post details some experiments that we have done regarding Machine Learning Opt–Out protocols. We decided to investigate the prevalence of some of these protocols, by taking a deeper look at our WARC files, and finding which proportions of domains are using which opt–out protocols.
Read More...Balancing Discovery and Privacy: A Look Into Opt–Out Protocols
February 13, 2024
What opt–out protocols are, their importance, how you can use them, how we respect them, and what the emerging initiatives are that surround them.
Read More...Host- and Domain-Level Web Graphs May/Sep/Nov 2023
December 22, 2023
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of May, September, and November of 2023.
Read More...November/December 2023 Crawl Archive Now Available
December 15, 2023
The crawl archive for November/December 2023 is now available. The data was crawled between November 28th and December 12th, and contains 3.35 billion web pages (or 454 TiB of uncompressed content).
Read More...Oct/Nov 2023 Performance Issues
November 15, 2023
Our datasets have become very popular over time, with downloads doubling every 6 months for several years in a row. This post details some steps to take if you are impacted by performance issues.
Read More...Host- and Domain-Level Web Graphs Mar/May/Oct 2023
October 18, 2023
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of March, May, and October 2023. The host-level graph consists of 378.7 million nodes and 2.6 billion edges, and the domain-level graph has 94.2 million nodes and 1.7 billion edges.
Read More...September/October 2023 crawl archive now available
October 12, 2023
The crawl archive for September/October 2023 is now available! The data was crawled Sept 21 – October 5 and contains 3.4 billion web pages or 456 TiB of uncompressed content.
Read More...Bridging Digital Exploration and Scientific Frontiers
October 10, 2023
This month Common Crawl Foundation members had the privilege of attending 5th International Open Search Symposium at CERN in Geneva, Switzerland.
Read More...May/June 2023 crawl archive now available
June 21, 2023
The crawl archive for May/June 2023 is now available! The data was crawled May 27 – June 11 and contains 3.1 billion web pages or 390 TiB of uncompressed content. Page captures are from 44 million hosts or 35 million registered domains and include 1.0 billion new URLs, not visited in any of our prior crawls.
Read More...March/April 2023 crawl archive now available
April 6, 2023
The crawl archive for March/April 2023 is now available! The data was crawled March 20 – April 2 and contains 3.1 billion web pages or 400 TiB of uncompressed content. Page captures are from 43 million hosts or 34 million registered domains and include 1.2 billion new URLs, not visited in any of our prior crawls.
Read More...Host- and Domain-Level Web Graphs September/October, November/December 2022 and January/February 2023
March 15, 2023
We are pleased to announce a new release of host-level and domain-level web graphs based on the September/October, November/December 2022 and January/February 2023 crawls. For more information about the data formats and the processing pipeline, please see the announcements of previous webgraph releases.
Read More...January/February 2023 crawl archive now available
February 16, 2023
The crawl archive for January/February 2023 is now available! The data was crawled January 26 – February 9 and contains 3.15 billion web pages or 400 TiB of uncompressed content. Page captures are from 40 million hosts or 33 million registered domains and include 1.3 billion new URLs, not visited in any of our prior crawls.
Read More...November/December 2022 crawl archive now available
December 14, 2022
The crawl archive for November/December 2022 is now available! The data was crawled November 26 – December 10 and contains 3.35 billion web pages or 420 TiB of uncompressed content. Page captures are from 44 million hosts or 34 million registered domains and include 1.2 billion new URLs, not visited in any of our prior crawls.
Read More...September/October 2022 crawl archive now available
October 11, 2022
The crawl archive for September/October 2022 is now available! The data was crawled September 24 – October 8 and contains 3.15 billion web pages or 380 TiB of uncompressed content. Page captures are from 44 million hosts or 34 million registered domains and include 1.3 billion new URLs, not visited in any of our prior crawls. This crawl includes improvements made in extracting clean text in WET files and WAT anchor texts.
Read More...Host- and Domain-Level Web Graphs May, June/July and August 2022
September 23, 2022
We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of May, June/July and August 2022. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases.
Read More...August 2022 crawl archive now available
August 22, 2022
The crawl archive for August 2022 is now available! The data was crawled August 7 – 20 and contains 2.55 billion web pages or 295 TiB of uncompressed content. Page captures are from 46 million hosts or 37 million registered domains and include 1.3 billion new URLs, not visited in any of our prior crawls.
Read More...June/July 2022 crawl archive now available
July 13, 2022
The crawl archive for June/July 2022 is now available! The data was crawled June 24 – July 7 and contains 3.1 billion web pages or 370 TiB of uncompressed content. Page captures are from 44 million hosts or 35 million registered domains and include 1.4 billion new URLs, not visited in any of our prior crawls.
Read More...May 2022 crawl archive now available
June 2, 2022
The crawl archive for May 2022 is now available! The data was crawled May 16 – 29 and contains 3.45 billion web pages or 420 TiB of uncompressed content. Page captures are from 45 million hosts or 36 million registered domains and include 1.4 billion new URLs, not visited in any of our prior crawls.
Read More...