dataseries.org is now ready for production. In a previous post from August, I promised this if all data stayed green for 90 days. It did, and dataseries.org is now 1.0.
Uptime monitoring on GitHub: UPTIME.md.
Most importantly, as you can see from the first row, the daily pipeline completed every single day over the last 3 months.
The middle row shows the days when a data provider was down or changed something on its side. This is expected to happen from time to time. The FSO changed the file format of a data cube, the KOF retired an API version, and the Adecco job index was down for some days. Note that this never caused stale data. The pipeline keeps the last version of a dataset and simply tries again. In two cases the scraper had to be fixed; in the third, the source came back online.
The bottom row went red once on August 14, but this was the monitoring’s fault, not the data’s: it expected a monthly update of a quarterly series.
I am confident we can keep this up in the future. As explained in more detail in the relaunch post, the scraping logic is stored in simple markdown files, and the code is derived from them by AI. That keeps the maintenance load reasonable.
A few other things happened since August. Spatial data is now available on maps.dataseries.org, by canton, municipality or 100 m grid. The site is now also available in German, French and Italian, and supports dark mode. There is a Python package, and the R package 1.1.0 on CRAN can search and label in all four languages.
As always, if you miss a dataset or have an idea, let me know!