The forgotten child in research: Data engineering and infrastructure
September 13, 2023
Some questions before starting 🤔
🙋 How many of you have worked with or tried to discover data for projects?
🙋 How many of you wish you spent less time on menial/manual tasks related to data?
Who am I? 👋
- Team Leader at Steno Diabetes Center Aarhus and Aarhus University, Denmark
- Research/work:
- Teach how to do open and reproducible science
- Build software to automate research
- Do epidemiological research
Two main goals of this (informal) talk 🔈
Spreading awareness…
… on the vital importance of the foundation of our data-driven world
What is that foundation? The data itself and the engineering of and around it.
… that innovation and commercialization can come from anywhere
Not to limit thinking to only traditional areas like results from research.
The stages and lifecycle of research are like a big family
Unmet and (often) unaware basic needs in health research
Personal past experiences: Data management tasks often given to unskilled MSc/PhD students
In research organizations: Focused on beginning and end of lifecycle (collecting data and publishing), not the middle
In small- to medium-sized companies: Don’t have maturity and/or funds to have internal data engineering team
NovoNordiskFonden: Increase impact of funding by expanding use of data from funded projects
Many, substantial negative effects of this unmet need
Examples often distill down to wasted time and money
Retractions because of data processing error (e.g. with Excel)
Wasted time looking for data or resolving issues with data
Time spent learning niche skill to understand or structure data
Unusable data because of lack of documentation (e.g. units of measure)
Limited options and solutions for data infrastructure within research world
… they are often custom-built
… they are often designed for industry, expensive, or “over-engineered”
… they are often heavy on the tech jargon
Our solution: A framework for building a modern data infrastructure
Seedcase: Improving discoverability, structure, and management of research data
Designed for typical use cases of doing research
From Seedcase Project Design Docs
Central philosophies and value
Follow FAIR (Findable, Accessible, Interoperable, and Reusable), open, and transparent principles
Openly licensed and re-usable
Use state-of-the-art principles and tools
Friendly to beginners and non-tech people
Who are we: The team
DD2 Data Manager
Database Administrator
Research Software Engineer
Research Software Engineer
Future steps: Ensuring financial sustainability
Creating a company around Seedcase
Offer research software development and data engineering services, keeping product free and open source
- Consulting
- Training and education
- Adding features
- Sponsorship from industry
- Embedding it early in newly funded projects
- Cloud-based products?
Licensed under CC-BY 4.0.
Slides at slides.lwjohnst.com