When government data disappears: How to cross-check, preserve, and investigate

IRE 2025 and AccessFest 2025 Tip Sheet

Anna Massoglia, anna@sunlightresearch.net, X, LinkedIn, BlueSky

Jay Hunter, jay@hunterindex.org, X, LinkedIn, Substack

Michael Nolan, michael@sunlightsearch.net, LinkedIn

Jason Leopold, jleopold15@bloomberg.net, LinkedIn

Elizabeth Clemons, elizabeth@sunlightresearch.net, LinkedIn

This tip sheet can be used to find information on the disappearance of government data and explore what you can do to keep track of data relevant to your research and reporting.

---

Examples of Government Data and Information Archives

• Data Rescue Project

• Backups List

• SciOp Datasets

• Data Lumos 

• Internet Archive

  • Wayback Machine

  • GovWayback

• End of Term Archive

• Source Cooperative 

  • Harvard Library Innovation Lab’s Data.gov Archive

• Harvard Dataverse 

• Data Liberation Project

• GW’s National Security Archive

  • Cyber Vault Library

  • Virtual Reading Room

  • Climate Change Transparency Project 

• AWS Open Data Sponsorship Program

• Hunter Index (politicians’ personal financial information)

• MuckRock

  • Requests (FOIA) 

  • Document Cloud

• ProPublica

  • News Apps

  • 527 Explorer (IRS 

  • Nonprofit Explorer (IRS Form 990s)

  • Data Store (static, historical)

• OpenSecrets.org (campaign finances, lobbying)

• The FOIA Project 

• DOGE Track

---

Tools for Archiving Content

• Internet Archive’s Wayback Machine

• Archive.ph 

• Zotero 

• Document Cloud by MuckRock

• Perma.cc (free for journalists)

• Conifer Webrecorder by Rhizome (more complex websites)

• ArchiveBox (self-hosted)

---

Tools for Archiving Data

• GitHub

  • Scraping code: https://github.com/m-nolan/doge-scrape 

  • BLN Updating code: https://github.com/biglocalnews/sync-doge-scrape/

• Creating web scrapers

  • Finding undocumented APIs: https://inspectelement.org/apis.html

---

Tools for Change Detection

• Visualping – Visual/text change tracking with alerts, Chrome extension (freemium)

• WebSite-Watcher – Advanced local tool for Windows ($$)

• Distill.io – Content tracking with local app, browser, Chrome extension  (freemium)

• PageCrawl.io – Team-friendly archiving and alerts (freemium)

• Wachete – Tracks private/password-protected pages (freemium)

• ChangeTower – Keyword and content change alerts (freemium)

• Fluxguard – HTML, visual and text tracking, translation (freemium)

• Follow That Page – Basic email alerts for text changes

• SiteDelta – Firefox-only in-browser tracker

• KeyCDN Tools  – check HTTP header for when page was last modified

---

Tips for Archiving

• Data Curation Network

---

Creating Web Scrapers

• Finding undocumented APIs: https://inspectelement.org/apis.html

---

DOGE Scraper Github Links

• Scraping code: https://github.com/m-nolan/doge-scrape 

• BLN Updating code: https://github.com/biglocalnews/sync-doge-scrape/

---

Articles writen with BLN DOGE tables

• University of Minnesota: Cuts, halted grant reviews could be ‘absolutely crippling’ to research

  • https://www.minnpost.com/health/2025/03/university-of-minnesota-cuts-halted-grant-reviews-could-be-absolutely-crippling-to-research/

  • MinnPost

  • 2025-03-03

• Unpacking DOGE’s plan to cut 14 federal office leases across Minnesota

  • https://www.minnpost.com/national/2025/03/unpacking-doges-plan-to-cut-14-federal-office-leases-across-minnesota/

  • MinnPost

  • 2025-03-19

• How much federal money flows into Minnesota for health care, education, agriculture?

  • https://www.minnpost.com/state-government/2025/03/how-much-federal-money-flows-into-minnesota-for-health-care-education-agriculture/

  • MinnPost

  • 2025-03-31

• Evanston woman says her contract is among DOGE website errors

  • https://www.nbcchicago.com/investigations/evanston-woman-says-her-contract-is-among-doge-website-errors/3691899/

  • NBC Chicago

  • 2025-03-07

• 'We got un-DOGE'd': Inaccurate info discovered in claimed taxpayer savings

  • https://www.nbcboston.com/news/local/we-got-un-doged-inaccurate-info-discovered-in-claimed-taxpayer-savings/3658923/

  • NBC Boston

  • 2025-03-14

• From DOGE cuts to tariffs, see Trump's first 100 days by the numbers

  • https://www.nbcphiladelphia.com/news/national-international/doge-cuts-tariffs-and-immigration-crackdown-trumps-first-100-days-by-the-numbers/4171722/

  • NBC10 Philadelphia

  • 2025-04-29

• DEI, Project 2025 and the Constitution: Tracking Trump's impact in his first 100 days

  • https://www.nbcwashington.com/investigations/dei-project-2025-and-the-constitution-tracking-trumps-impact-in-his-first-100-days/3903435/

  • NBC4 Washington

  • 2025-04-30

• DOGE claims at least $117 million in Bay Area contract cuts, spurring layoffs and uncertainty

  • https://www.nbcbayarea.com/investigations/doge-claims-millions-bay-area-contract-cuts/3864516/

  • NBC Bay Area

  • 2025-05-08

• EV uncertainty: How Trump's charging station funding freeze in affecting the DC area

  • https://www.nbcwashington.com/investigations/ev-uncertainty-how-trumps-charging-station-funding-freeze-is-affecting-the-dc-area/3914753/

  • NBC4 Washington

  • 2025-05-15

---

Questions or comments about Sunlight's workshops and resources? Contact Elizabeth at elizabeth@sunlightresearch.net.