Search historical URLs
Enumerate exact URLs, hosts, domains and time ranges across indexed Common Crawl snapshots.
CIRCL archive intelligence
Search a decade of the public web.
Common Crawl wGET turns large-scale web archives into targeted, reproducible lookups. Find the record you need, then retrieve only the selected WARC object.
Built for focused investigation · metadata-first search · selective retrieval
2013+web history coverage
WARCstandard archive objects
SHA-1content digest lookup
Byte-rangetargeted object retrieval
What you can do
Start with a question.
Keep the download precise.
Follow content by digest
Use hexadecimal or Base32 SHA-1 digests to locate matching archived content without guessing its URL.
Retrieve the WARC header and web object
Inspect metadata first, then retrieve the WARC header and the matching archived web object without transferring the full archive.
The archive behind the service
Open web history at planetary scale.
Common Crawl maintains a free and open repository of web crawl data. The corpus now contains more than 300 billion pages collected across 15 years, with roughly 3–5 billion new pages added each month. That scale is powerful for research and investigations, but it also makes targeted indexing and selective retrieval essential.
Common Crawl wGET provides a focused way to work with that archive: search the metadata, understand the matching record, then retrieve the relevant WARC header and archived web object.
A deliberate path through big data
Index first.
Object second.
The client sends a compact query to the CIRCL service. Matching metadata gives you the coordinates and provenance needed to make a narrow retrieval.
- 01
Describe the target
URL, FQDN, path, query, date range or SHA-1 digest.
- 02
Review the record
Check capture time, WARC filename, offset and length.
- 03
Retrieve the evidence
Download the selected archived response for analysis or preservation.
A small client for a very large archive
Useful from the shell.
CCWget is open source and designed for scripted, auditable workflows.
# Search a host through the service
python ccwget.py --list-fqdn example.org --alltime
# Look up a content digest
python ccwget.py --sha1 <sha1-hex-or-base32> --alltimeAccess policy
Need an account?
Access is provided free of charge for appropriate use cases. To request credentials, contact info@circl.lu with a short description of your intended use.
Requests are reviewed individually and access is granted at CIRCL’s discretion, taking operational capacity, responsible use and the needs of the community into account.
EU Funded Project
HOPLITE
HOPLITE is an EU-funded project to provide Law Enforcement Agencies (LEAs) and Judicial Authorities (JAs) with an intuitive platform for launching OSINT campaigns, receiving threshold-level incident alerts, and exchanging threat intelligence with trusted partners.