The July crawl of 2014 is now available! The new dataset is over 266TB in size containing approximately 3.6 billion webpages. The new data is located in the commoncrawl bucket at /crawl-data/CC-MAIN-2014-23/.
To assist with exploring and using the dataset, we’ve provided gzipped files that list:
- all segments (CC-MAIN-2014-23/segment.paths.gz)
- all WARC files (CC-MAIN-2014-23/warc.paths.gz)
- all WAT files (CC-MAIN-2014-23/wat.paths.gz)
- all WET files (CC-MAIN-2014-23/wet.paths.gz)
By simply adding either s3://commoncrawl/ or https://data.commoncrawl.org/ to each line, you end up with the S3 and HTTP paths respectively.
We've also released a Python library, gzipstream, that should enable easier access and processing of the Common Crawl dataset. We'd love for you to try it out!
Thanks again to blekko for their ongoing donation of URLs for our crawl!
Note: the original estimate for this crawl was 4 billion, but after full analytics were run, this estimate was revised.
This release was authored by:
No items found.
Erratum:
WARC Content-Type header in revisit records
Originally reported by:
Sebastian Nagel
Common Crawl's WARC revisit records use Content-Type: message/http (following the WARC 1.1 spec's example), but per iipc/warc-specifications#55 it should be application/http;msgtype=response for consistency with other HTTP response records.
Erratum:
Columnar Index Subsets with Fewer than 900 Partitions per Crawl
Originally reported by:
Sebastian Nagel
Older crawls from CC-MAIN-2013-20 including CC-MAIN-2016-30 do not have the robotstxt and crawldiagnostics subset partitions in the Columnar Index.
Erratum:
WARC-Target-URI May Include Non-ASCII Characters
Originally reported by:
The WARC-Target-URI header in WARC record, but also corresponding WAT, WET and URL index records may include non-ASCII characters, not encoded using percent-encoding or Punycode.
Erratum:
Content is truncated
Originally reported by:
Some archived content is truncated due to fetch size limits imposed during crawling. This is necessary to handle infinite or exceptionally large data streams (e.g., radio streams). Prior to March 2025 (CC-MAIN-2025-13), the truncation threshold was 1 MiB. From the March 2025 crawl onwards, this limit has been increased to 5 MiB.
Erratum:
No Truncation Indicator in WARC Records
Originally reported by:
Henry Thompson
Due to an issue with our crawler, not all truncations were indicated correctly. A workaround to detect length truncation is to be suspicious if the length of the content is exactly 1048576 bytes. Truncations for time or network do not have such a workaround. In the WARC files this indicator is called "WARC-Truncated".
Erratum:
Missing content_truncated flag in URL indexes
Originally reported by:
The flag in our URL indexes (CDX and columnar) that indicates whether or not a WARC record payload was truncated was added in CC-MAIN-2019-47. This indicator is missing in our indexes for all previous crawl releases. In the CDX index this is referred to as "truncated", and the columnar index refers to this as "content_truncated".
Erratum:
WAT data: repeated WARC and HTTP headers are not preserved
Originally reported by:
Repeated HTTP and WARC headers were not represented in the JSON data in WAT files.
Erratum:
Erroneous title field in WAT records
Originally reported by:
Robert Waksmunski
The title field in WAT record is extracted from last but not first <title> element in an HTML page
Erratum:
Charset Detection Bug in WET Records
Originally reported by:
Javier de la Rosa
The charset detection required to properly transform non-UTF-8 HTML pages in WARC files into WET records didn't work before November 2016 (CC-MAIN-2016-50) due to a bug in IIPC Web Archive Commons.
Erratum:
Missing Language Classification
Originally reported by:
Starting with crawl CC-MAIN-2018-39 we added a language classification field (‘content-languages’) to the columnar indexes, WAT files, and WARC metadata for all subsequent crawls.
.svg)
