Scraping 107M rows of data to build this
Ben’s session #5
TL;DR
- AI agents were employed to scrape 107 million rows of UK council spending data, focusing on payments over £500.
- The project involved cataloging official data sources, downloading data from multiple councils, and consolidating it into a single format.
- Various prototypes were built to visualize the data, including maps, ledgers, and comparison cards, with a focus on an 'Apple Maps' style interface.
- Technical challenges included handling large datasets, leading to the adoption of Parquet and DuckDB for efficient data management.
- The final stages involved data cleaning, vendor classification, duplicate merging, and refining the user experience and data validity.
- The resulting web application aims to make public council spending data accessible and understandable to the public.