The Wayback Machine for developers and researchers: reading a site's past through the web archive

Websites change and disappear faster than we think. Pages get deleted, domains expire, startups shut down, and official statements get quietly edited. The Internet Archive's Wayback Machine is one of the few places that keeps snapshots of that past at scale, readable by anyone.

In OSINT for developers the Wayback Machine showed up as a single command. This post goes deeper: how the archive gets built, how to read snapshot URLs, how to use the CDX API for research, and what you can't trust about an archive.

A short history and how it works

Brewster Kahle founded the Internet Archive in 1996 as a nonprofit digital library. Its web archive opened to the public as the Wayback Machine in 2001 and has since grown to hundreds of billions of pages.

The archive is built by crawlers: programs that visit a page, store its HTTP response (HTML, images, CSS, JavaScript), then follow links to the next page. Crawl output is stored in the WARC format (Web ARChive, standardized as ISO 28500), which records not just page content but request and response headers along with their timestamps. The sources vary: the Internet Archive's own broad crawls, partnerships with national libraries and projects such as Archive-It, and user requests through Save Page Now.

The consequence matters: the Wayback Machine only stores what its crawlers have seen. Pages behind a login, content rendered entirely by JavaScript, or pages never linked from anywhere are often not archived at all.

Reading a snapshot URL

Every snapshot has an address following the same pattern:

https://web.archive.org/web/20190615083012/https://your-domain.com/about
                            └─────┬──────┘ └───────────┬───────────┘
                             time (UTC)          original URL
                           YYYYMMDDhhmmss

A few useful tricks:

• The timestamp doesn't have to be complete. /web/2019/https://your-domain.com redirects to the snapshot closest to the start of 2019. Check where you land: the closest snapshot might be months away.
• An asterisk gives you a listing. /web/*/your-domain.com opens the calendar of every snapshot, and /web/*/your-domain.com/* lists every URL under that domain.
• The id_ suffix gives you the original file. Normally the Wayback Machine rewrites links in the page and adds a toolbar. With /web/20190615083012id_/https://… you get the raw response as it was archived, which is what you want for comparing or hashing.

Timestamps in the URL are always UTC. When matching a snapshot to an event in your own time zone, convert first.

The CDX API: the archive as data

The calendar on the website is nice to look at, but research needs raw data. The CDX API returns one line per capture, with its timestamp, HTTP status, MIME type, and a digest of the content:

curl -s 'https://web.archive.org/cdx/search/cdx?url=your-domain.com&output=json&fl=timestamp,statuscode,mimetype,digest&limit=20'

The most commonly used parameters:

The digest field is a SHA-1 hash (in base32) of the response body. That makes collapse=digest very useful: a crawler might visit the same page hundreds of times, but what matters is when its content changed.

curl -s 'https://web.archive.org/cdx/search/cdx?url=your-domain.com/privacy&fl=timestamp,digest&filter=statuscode:200&collapse=digest'

To map old files across a whole domain, combine matchType=domain with a filter on MIME type or extension:

curl -s 'https://web.archive.org/cdx/search/cdx?url=your-domain.com&matchType=domain&fl=original&collapse=urlkey&filter=original:.*\.(pdf|json|env|bak|sql|zip)$' \
  | sort -u

For large domains the output can get very long. Use limit and time ranges, or tools like waybackurls and gau that handle pagination for you. And pause between requests: the Internet Archive is a nonprofit, and its API is rate limited.

Comparing two versions of a page

The most common research question is "what changed?". Once you have two timestamps from CDX, fetch the raw versions with id_ and compare:

A=20230101000000; B=20240101000000
URL=https://your-domain.com/terms
curl -sL "https://web.archive.org/web/${A}id_/${URL}" -o a.html
curl -sL "https://web.archive.org/web/${B}id_/${URL}" -o b.html
diff <(sed 's/<[^>]*>//g' a.html | tr -s ' \n') \
     <(sed 's/<[^>]*>//g' b.html | tr -s ' \n')

Stripping HTML tags with sed is crude, but good enough to spot sentences that were added or removed. For complex pages, extract the text with a real HTML parser before diffing.

Journalists use this pattern to track changes to terms of service, press releases, and team pages, and researchers use it to see when a product claim started or stopped appearing.

Preserving evidence with Save Page Now

The archive isn't only for reading. Anyone can ask for a page to be archived right now, through the form on web.archive.org or directly by URL:

curl -s -o /dev/null -w '%{http_code}\n' 'https://web.archive.org/save/https://your-domain.com/announcement'

Then confirm the snapshot actually exists with the availability API:

curl -s 'https://archive.org/wayback/available?url=your-domain.com/announcement&timestamp=20261102' \
  | jq '.archived_snapshots.closest'

If you're collecting evidence, such as a phishing page, a post that might be deleted, or an official statement, also keep a local copy with its hash. A public snapshot helps others verify, but you still need your own record:

[2026-11-02 13:15 UTC] https://your-domain.com/announcement
  archive: https://web.archive.org/web/20261102131502/https://your-domain.com/announcement
  local  : announcement-20261102.html  sha256=3f9a…

Limits to keep in mind

Web archives are a powerful source, but not a perfect record.

• Incomplete. Crawlers don't visit every page, and not every day. The absence of a snapshot is not proof that a page never existed.
• Mixed timestamps. A single page snapshot can include images, CSS, or scripts archived at other times. What renders on screen isn't necessarily exactly what visitors saw at that moment. Check each asset's timestamp when the details matter.
• Dynamic content. Pages that load their content through JavaScript from an API are often archived as an empty shell.
• Exclusions. Site owners can ask the Internet Archive to hide their domain's captures, so a snapshot that existed yesterday may be gone from public view today.
• The service can go down. In October 2024 the Internet Archive was hit by DDoS attacks and a breach of user account data, taking the service offline for days. For important research, cross-check with other archives such as archive.today, or a Memento aggregator like Time Travel that searches many archives at once.

Wayback Machine snapshots are often submitted as evidence in court, but they usually still need to be authenticated, for example with an affidavit from the Internet Archive or a witness who can explain how the capture was made. The principle is the same as OSINT in general: an archive is a strong lead that has to be verified, not a fact that stands on its own.

From the site owner's side

The Wayback Machine is useful for your own sites too:

• Recovering a lost site. With no backup, old pages and assets can often still be downloaded from the archive, one at a time or with a tool like wayback_machine_downloader.
• Preventing link rot. When you link to a source in a blog post or documentation, archive the source too. Wikipedia does this automatically for millions of its references.
• Auditing old leaks. Run the CDX queries above against your own domain. A .env file, a database backup, or an internal document that was briefly public may still be in the archive. Remove the file from the server, rotate any credentials it exposed, then consider requesting an exclusion from the Internet Archive.

That last step matters: deleting a file from your server doesn't delete its copy in the archive. Just as in OPSEC for developers, assume anything that was ever public has been copied somewhere, and rotate leaked secrets instead of hoping nobody saw them.