Common Crawl - Blog - February/March 2024 Crawl Archive Now Available

Erratum:

WARC Content-Type header in revisit records

Originally reported by:

Sebastian Nagel

Common Crawl's WARC revisit records use Content-Type: message/http (following the WARC 1.1 spec's example), but per iipc/warc-specifications#55 it should be application/http;msgtype=response for consistency with other HTTP response records.

Erratum:

WARC-Target-URI May Include Non-ASCII Characters

Originally reported by:

More details

The WARC-Target-URI header in WARC record, but also corresponding WAT, WET and URL index records may include non-ASCII characters, not encoded using percent-encoding or Punycode.

Erratum:

Redirect target URL in URL indexes may be a relative URL

Originally reported by:

Sebastian Nagel

More details

When the HTTP “Location” header includes a relative URL, the corresponding “redirect” field in the CDX index and “fetch_redirect” field in the columnar index will also store a relative URL. In all other cases, redirect targets in the URL indexes should be recorded as absolute URLs.

Erratum:

Content is truncated

Originally reported by:

More details

Some archived content is truncated due to fetch size limits imposed during crawling. This is necessary to handle infinite or exceptionally large data streams (e.g., radio streams). Prior to March 2025 (CC-MAIN-2025-13), the truncation threshold was 1 MiB. From the March 2025 crawl onwards, this limit has been increased to 5 MiB.

Erratum:

SURT URLs do not properly encode non-UTF-8 percent-encoded characters

Originally reported by:

Tom Morris

More details

When constructing SURT (Sort-friendly URI Reordering Transform) URLs, percent-encoded characters that are not valid UTF-8 sequences were not being correctly handled. This could lead to inconsistencies in URL normalization and sorting, potentially causing incorrect deduplication or retrieval issues in datasets that rely on SURT-based indexing.

Erratum:

WAT data: repeated WARC and HTTP headers are not preserved

Originally reported by:

More details

Repeated HTTP and WARC headers were not represented in the JSON data in WAT files.

Erratum:

WARC revisit metadata records

Originally reported by:

More details

The revisit records in the Common Crawl WARC archives (since Aug 2018) lack the metadata record which is attached to all response records.

Erratum:

Erroneous title field in WAT records

Originally reported by:

Robert Waksmunski

More details

The title field in WAT record is extracted from last but not first <title> element in an HTML page

	File List	#Files	Total Size Compressed (TiB)
Segments	segment.paths.gz	100
WARC	warc.paths.gz	90000	90.36
WAT	wat.paths.gz	90000	20.97
WET	wet.paths.gz	90000	8.40
Robots.txt	robotstxt.paths.gz	90000	0.16
Non-200 responses	non200responses.paths.gz	90000	3.38
URL index	cc-index.paths.gz	302	0.23
Columnar URL index	cc-index-table.paths.gz	900	0.27

Erratum:

WARC Content-Type header in revisit records

Erratum:

WARC-Target-URI May Include Non-ASCII Characters

Erratum:

Redirect target URL in URL indexes may be a relative URL

Erratum:

Content is truncated

Erratum:

SURT URLs do not properly encode non-UTF-8 percent-encoded characters

Erratum:

WAT data: repeated WARC and HTTP headers are not preserved

Erratum:

WARC revisit metadata records

Erratum:

Erroneous title field in WAT records

The Data

Resources

Community

About