Denmark Statistics and large health data: We don’t know what we’re doing
September 25, 2025
Who am I? 👋 👋
History
MSc and PhD in Nutritional Science in Toronto, Canada
(Previous) Research in diabetes epidemiology
Team leader at SDCA for the Seedcase Project, an NNF funded software project to simplify building FAIR data
My work on large data
ukbAid: R package and website
DARTER Project: Website of application to and documentation on a DST project
Rationale for this talk
We are woefully behind on data engineering and programming practices
This impacts how we do effective and accurate research
This is important because:
Validity of results
Speed impacts time to results
Ability to do more complex analysis with more data
Resources (which cost money)
Aim of this talk
Highlight issues with Denmark Statistics and the research using the registers (and any other large data)
Spread awareness of how critical programming skills are, especially for large data
Showcase a few tools for doing research faster, so you can focus on doing science
General “roadmap” 🗺️
Issues with DST
Need for dedicated programming expertise
Questionable reproducibility and validity
Denmark Statistics, the good and the bad
The good: Amazing resource, gold mine of data
And the bad…
Everyone works on same server
No queue system for analyses
Can crash your analysis if others use more memory
Everyone works in same folder in a project
Data is stored in a proprietary SAS format
For example, BEF register:
Takes many minutes to load one year of data (in R)
Data updates make more work for us
Can you see the issue?
Metadata is confusing and poorly documented
Variables are not consistent across years
Finding the metadata is difficult
DST is either unaware of or indifferent to improving things
Highlights lack of engineering and design expertise at DST
- Puts tech burden onto researchers
Need programming expertise, especially for large data
Two tools as examples: Parquet and DuckDB
Parquet should be used to store large data
Most data formats are row-based, like CSV or SAS. Newer formats tend to be column-based like Parquet.
Column-based storage has many advantages
Parquet is 50-75% smaller than other formats
| File type | Size (MB) |
|---|---|
SAS (.sas7bdat) |
1.45 Gb |
CSV (.csv) |
~90% of SAS |
Stata (.dta) |
745 Mb |
Parquet (.parquet) |
398 Mb |
Can partition data by a value (e.g. year) and load all at once
DuckDB is a recent SQL engine designed for analytical queries
SQL, or Structured Query Language, is a language for managing and querying databases
DuckDB is impressively fast
Faster than almost all other tools
Relatively complex queries (joins, group by, aggregates) on 55 Gb takes < 7.5 min 1
Generally, simpler queries take < 10 seconds for massive datasets
Easily connects with Parquet datasets
Example in DST
- Load all 45 years of BEF
- Drop all missing PNR
- Group by year
- Count sex
Takes < 6 seconds
So… why is this important? Seems like easy stuff!
- Because it highlights a lack of expertise and understanding of some basic things
Reproducibility and verification of research
Reproducibility: same data + same code = same results?
Non-reproducibility is a big, though mostly unknown, issue
Analyzing, and reproducing, large data requires programming skills
With massive data, you really need to know how to code and programming
But, researchers are not trained for this kind of skill
No one reviews code, so, no one knows if it’s correct
- This problem is bigger than people realize.
“Science as amateur software development”
“But the code runs!”
Working in DST makes reproducibility harder
Hard to review code
Hard to collaborate
No version control
Use proprietary data formats
No queue system
No training materials or courses
Science is not about trust, it’s about verification
It’s difficult to trust what researchers do in DST
What to do?
If reviewing papers, request or demand code is accessible
Recognize, value, and reward those with programming expertise
Pressure DST to improve things
Licensed under CC-BY 4.0.
Slides at slides.lwjohnst.com