Denmark Statistics and large health data: We don’t know what we’re doing

Luke W. Johnston

September 25, 2025

Who am I? 👋 👋

History

• MSc and PhD in Nutritional Science in Toronto, Canada

• (Previous) Research in diabetes epidemiology

• Team leader at SDCA for the Seedcase Project, an NNF funded software project to simplify building FAIR data

I started out doing more “traditional” research but have gradually been moving towards more computationally technical work, like software development and data engineering.

As I continued my research career, I got more and more into building software and creating courses to teach researchers about better programming practices. This also meant that I had less and less time to publish papers, because, there’s only so much time in a day.

As I go along, keep this in mind, because it highlights a big issue we have in research.

My work on large data

• ukbAid: R package and website

• DARTER Project: Website of application to and documentation on a DST project

Rationale for this talk

Aside from being asked to present on it 🤪

We are woefully behind on data engineering and programming practices

Especially when working with massive data

This impacts how we do effective and accurate research

This is important because:

• Validity of results

• Speed impacts time to results

• Ability to do more complex analysis with more data

• Resources (which cost money)

Aim of this talk

1. Highlight issues with Denmark Statistics and the research using the registers (and any other large data)

2. Spread awareness of how critical programming skills are, especially for large data

3. Showcase a few tools for doing research faster, so you can focus on doing science

Also keep this very informal, ask questions at any point.

There’s a lot of issues with how DST is organised and structured. I won’t go into all of them, but instead will focus on some of the more basic, fundamental ones.

General “roadmap” 🗺️

• Issues with DST

• Need for dedicated programming expertise

• Questionable reproducibility and validity

Denmark Statistics, the good and the bad

Anyone do research with DST?

The good: Amazing resource, gold mine of data

Nearly unique in the world!

And the bad…

Application alone is a massive barrier to working with the data, but I won’t talk about that in this talk, just to highlight that it is a big issue.

All of these things are important to remember later on in the presentation.

For this talk, only basic things

Everyone works on same server

• No queue system for analyses

• Can crash your analysis if others use more memory

So that basically means, it’s best to work on the weekends or evenings for simple stuff. And hope that for longer analyses, no one else hoards all the memory and crashing your analysis.

Remember how I mentioned about why this is important? This is a massive waste of resources and time and effort, if analyses just crash because someone else is using too much memory.

It also slows down research because of something so stupid as this.

Waste of resources, slows research

Everyone works in same folder in a project

project/
├── luke/
│   └── analysis/
└── omar/
    └── paper/

Collaboration is difficult:

• Anyone can edit anything
• Can’t know who changed what and when (no “version control”)
• Not easy to review and improve other’s code

No version control or separation of sub-projects.

I’ll come back to this reviewing thing later.

Slows research down

Data is stored in a proprietary SAS format

For example, BEF register:

bef2018.sas7bdat
bef2019.sas7bdat
bef2020.sas7bdat
bef2021.sas7bdat
bef2022.sas7bdat

Takes many minutes to load one year of data (in R)

This means, if you use R, or Python, or Stata, you have to load these, which can take many minutes per file, just to load it.

These are important to know for later.

Waste of time and resources

Data updates make more work for us

bef2021.sas7bdat
bef2022.sas7bdat
December_2023/bef2022.sas7bdat
December_2023/bef2023.sas7bdat

Can you see the issue?

One problem, sometimes there’s a new version of a year you already had. But you don’t know what’s been changed. You have to spend time checking what changed and if it messes things up for you. The second problem is, the updates are in a new folder. So trying to build an automated pipeline to load the data in is a bit of a pain because the structure changes for each update.

These seem like small things, but these are the things that mess up how code works and runs. These things are what make it hard to build effective analyses and effective software to ideally simplify your life while working with the registers.

Again, waste of time

Metadata is confusing and poorly documented

• Variables are not consistent across years

• Finding the metadata is difficult

Metadata is a big problem. Documentation is relatively poor for most of the variables, it’s in another location that requires you to dig into it. Values in some variables that are numbers but actually are categories… but the documentation for what those numbers mean isn’t in the same place. So requires searching.

Slows research down, wastes time

DST is either unaware of or indifferent to improving things

There’s so much potential there, but DST doesn’t seem to value research and improvements. Or at least, from their actions and infrastructure so far, it doesn’t seem like it.

Highlights lack of engineering and design expertise at DST

• Puts tech burden onto researchers

Need programming expertise, especially for large data

An organization should be responsible for having a well-designed setup, otherwise, its on the researcher. But that’s not their expertise.

Two tools as examples: Parquet and DuckDB

https://parquet.apache.org/

https://duckdb.org/

Parquet should be used to store large data

Most data formats are row-based, like CSV or SAS. Newer formats tend to be column-based like Parquet.

Row-based

name,sex,age
Tim,M,30
Jenny,F,25

Column-based

name,Tim,Jenny
sex,M,F
age,30,25

Column-based storage has many advantages

Compression

name,Tim,Sam,Jenny
sex,M,F,F
age,30,30,25

…becomes…

name,Tim,Jenny,Sam
sex,M,F{2}
age,30{2},25

Loading

• Computers read by lines
• Per line = same data type
• Only read needed columns

sex,M,F
age,30,25

Parquet is 50-75% smaller than other formats

File size between CSV, Parquet, Stata, and SAS for bef register for 2017.

File type	Size (MB)
SAS (.sas7bdat)	1.45 Gb
CSV (.csv)	~90% of SAS
Stata (.dta)	745 Mb
Parquet (.parquet)	398 Mb

Why doesn’t DST use Parquet? It’s a clearly better format. Unless they keep SAS because it uses so much storage, which they charge for.

e.g. when someone uses Stata, they need to copy the data to the folder and convert to Stata format… which basically doubles the amount of storage used… thus increasing costs for the researcher.

My own experience: 500 GB of SAS data converted to Parquet is only 80 GB.

Can partition data by a value (e.g. year) and load all at once

bef/year=2018/part-0.parquet
bef/year=2019/part-0.parquet
bef/year=2020/part-0.parquet
bef/year=2021/part-0.parquet
bef/year=2022/part-0.parquet

Load in R with arrow package:

bef <- arrow::open_dataset("bef")

Load all years in < 1 sec, compared to ~5 min for one year from SAS format

DuckDB is a recent SQL engine designed for analytical queries

https://duckdb.org/

SQL, or Structured Query Language, is a language for managing and querying databases

DuckDB is impressively fast

• Faster than almost all other tools

  • Relatively complex queries (joins, group by, aggregates) on 55 Gb takes < 7.5 min ¹

  • Generally, simpler queries take < 10 seconds for massive datasets

• Easily connects with Parquet datasets

1. From benchmark page.

Example in DST

1. Load all 45 years of BEF
2. Drop all missing PNR
3. Group by year
4. Count sex

Takes < 6 seconds

So… why is this important? Seems like easy stuff!

• Because it highlights a lack of expertise and understanding of some basic things

Reproducibility and verification of research

Reproducibility: same data + same code = same results?

Background on reproducibility, explain what

Which ties into reproducibility, which is when the same data and the same code are run independently of the original authors, to get the same results.

But if the expertise for coding, for basic understandings of computeres and data, are lacking, it’s difficult to actually ensure reproducibility.

Non-reproducibility is a big, though mostly unknown, issue

DOI: 10.1093/gigascience/giad113

Analyzing, and reproducing, large data requires programming skills

• With massive data, you really need to know how to code and programming

• But, researchers are not trained for this kind of skill

Hoping your skills and knowledge are enough to do the analysis, without actually knowing if they are good enough.

No one reviews code, so, no one knows if it’s correct

• This problem is bigger than people realize.

Who has reviewed code here, like gone through all of it, not just looked at it?

I’ve done a few formal code reviews and … it’s not good. I had to stop reviewing things because it was just too much work and too stressful. Until a cultural change happens, I’m very pessimistic about things. Given code that I’ve seen so far, I think this is a bigger problem to the accuracy of research from registers than a lot of people realise.

I get it, researchers are not taught these things. Nor do they have a lot of time, they have so many things to do. In my team, where we build software, we review every single change or addition to the code. It’s hard work. It takes a lot of time to learn to make some changes that are easy to review. And it takes a lot of time to understand what the person is trying to do with the code.

And this is with three of us, including me, who have a lot of programming and software development experience.

Or rather, very few formally review code.

“Science as amateur software development”

“But the code runs!”

The state of programming and software development in research is abysmal to say the least.

The problem that researchers aren’t even really aware of this problem No funding goes to it, there are few courses on it, it isn’t required, we cannot hire skilled personnel for this either because it isn’t funded nor valued. What ends up happening? You get numbers from running the analysis… But now do you even know it’s right? Programming is hard, way harder than doing research.

From this academic presentation.

Working in DST makes reproducibility harder

• Hard to review code

• Hard to collaborate

• No version control

• Use proprietary data formats

• No queue system

• No training materials or courses

So makes it harder to have several people working on a project, for example, if one person has more expertise in programming/coding.

Science is not about trust, it’s about verification

The whole point of science is verification. Otherwise we are no different from pseudoscience. If we can’t verify that the results are correct, by looking at the code and reviewing it and running it, how can we know and trust it?

Mention tho that I believe knowledge generated from science is still better than things made up out of thin air.

Some trust is needed, but shouldn’t be dependent on it.

It’s difficult to trust what researchers do in DST

What to do?

If reviewing papers, request or demand code is accessible

You don’t have to review it, just ask for it.

Recognize, value, and reward those with programming expertise

Pressure DST to improve things