How (and why) to become more reproducible and open in your research

.title[
# How (and why) to become more reproducible and open in your research
]
.author[
### Luke W. Johnston
]

---

</style>

.footer-left[
Slides: [slides.lwjohnst.com/au/2022-08-11](https://slides.lwjohnst.com/au/2022-08-11/)  
Licensed under CC-BY <svg aria-hidden="true" role="img" viewBox="0 0 496 512" style="height:1em;width:0.97em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#990033;overflow:visible;position:relative;"><path d="M314.9 194.4v101.4h-28.3v120.5h-77.1V295.9h-28.3V194.4c0-4.4 1.6-8.2 4.6-11.3 3.1-3.1 6.9-4.7 11.3-4.7H299c4.1 0 7.8 1.6 11.1 4.7 3.1 3.2 4.8 6.9 4.8 11.3zm-101.5-63.7c0-23.3 11.5-35 34.5-35s34.5 11.7 34.5 35c0 23-11.5 34.5-34.5 34.5s-34.5-11.5-34.5-34.5zM247.6 8C389.4 8 496 118.1 496 256c0 147.1-118.5 248-248.4 248C113.6 504 0 394.5 0 256 0 123.1 104.7 8 247.6 8zm.8 44.7C130.2 52.7 44.7 150.6 44.7 256c0 109.8 91.2 202.8 203.7 202.8 103.2 0 202.8-81.1 202.8-202.8.1-113.8-90.2-203.3-202.8-203.3z"/></svg>
]

---

<div>
<style type="text/css">.xaringan-extra-logo {
width: 60px;
height: 128px;
z-index: 0;
background-image: url(../../common/sdca-logo.png);
background-size: contain;
background-repeat: no-repeat;
position: absolute;
top:1em;right:1em;
}
</style>
<script>(function () {
  let tries = 0
  function addLogo () {
    if (typeof slideshow === 'undefined') {
      tries += 1
      if (tries < 10) {
        setTimeout(addLogo, 100)
      }
    } else {
      document.querySelectorAll('.remark-slide-content:not(.title-slide):not(.inverse):not(.hide_logo)')
        .forEach(function (slide) {
          const logo = document.createElement('a')
          logo.classList = 'xaringan-extra-logo'
          logo.href = 'https://www.stenoaarhus.dk/'
          slide.appendChild(logo)
        })
    }
  }
  document.addEventListener('DOMContentLoaded', addLogo)
})()</script>
</div>

## <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#990033;overflow:visible;position:relative;"><path d="M40 48C26.7 48 16 58.7 16 72v48c0 13.3 10.7 24 24 24H88c13.3 0 24-10.7 24-24V72c0-13.3-10.7-24-24-24H40zM192 64c-17.7 0-32 14.3-32 32s14.3 32 32 32H480c17.7 0 32-14.3 32-32s-14.3-32-32-32H192zm0 160c-17.7 0-32 14.3-32 32s14.3 32 32 32H480c17.7 0 32-14.3 32-32s-14.3-32-32-32H192zm0 160c-17.7 0-32 14.3-32 32s14.3 32 32 32H480c17.7 0 32-14.3 32-32s-14.3-32-32-32H192zM16 232v48c0 13.3 10.7 24 24 24H88c13.3 0 24-10.7 24-24V232c0-13.3-10.7-24-24-24H40c-13.3 0-24 10.7-24 24zM40 368c-13.3 0-24 10.7-24 24v48c0 13.3 10.7 24 24 24H88c13.3 0 24-10.7 24-24V392c0-13.3-10.7-24-24-24H40z"/></svg> Outline

- Learning objectives

- Main take-home message

- Introduce and describe reproducibility and open science

- Activities to discuss and brainstorm (~40 min)

- Resources

---

## Learning objectives of this workshop:

1. Become aware of what reproducible and open practices are

2. Hear and learn how others in your group work and on their experiences and thoughts

3. Learn about some simple ways you can become more open and reproducible in your work

4. Identify actionable ways to adopt some of these practices in your group at NCRR (1)

---

**Take-home message:**

## Publicly share your code from your research

???

And I will add, share you code at *any* stage of your research. From beginning,
to end, to middle.

---

## Why are reproducible and open practices important right now? 🤔

**They are part of multiple large trends like team science, computing, meta-research, higher quality/rigor**

???

(Raise of hands) How many could describe the difference between reproducibility
and replicability?

---

## Reproducibility: Same data + analysis = same result?

- It's like baking: Data are the ingredients and the analysis is the recipe
- Known as analytic/computational reproducibility or reproducible data analysis.
- *Independently* reproduce results in paper based on same analysis and data.

---

## Reproducibility is a spectrum: *Any* improvement is better than no improvement

???

Part of this workshop and a lot of my teaching and work is to get that us as a
community to move closer to this end.

Reproducibility is about HOW EXACTLY a finding was found in a study.

---

## 6 min activity: Think 💭about your research workflow, then share <svg aria-hidden="true" role="img" viewBox="0 0 640 512" style="height:1em;width:1.25em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#da9100;overflow:visible;position:relative;"><path d="M208 0C322.9 0 416 78.8 416 176C416 273.2 322.9 352 208 352C189.3 352 171.2 349.7 153.9 345.8C123.3 364.8 79.13 384 24.95 384C14.97 384 5.93 378.1 2.018 368.9C-1.896 359.7-.0074 349.1 6.739 341.9C7.26 341.5 29.38 317.4 45.73 285.9C17.18 255.8 0 217.6 0 176C0 78.8 93.13 0 208 0zM164.6 298.1C179.2 302.3 193.8 304 208 304C296.2 304 368 246.6 368 176C368 105.4 296.2 48 208 48C119.8 48 48 105.4 48 176C48 211.2 65.71 237.2 80.57 252.9L104.1 277.8L88.31 308.1C84.74 314.1 80.73 321.9 76.55 328.5C94.26 323.4 111.7 315.5 128.7 304.1L145.4 294.6L164.6 298.1zM441.6 128.2C552 132.4 640 209.5 640 304C640 345.6 622.8 383.8 594.3 413.9C610.6 445.4 632.7 469.5 633.3 469.9C640 477.1 641.9 487.7 637.1 496.9C634.1 506.1 625 512 615 512C560.9 512 516.7 492.8 486.1 473.8C468.8 477.7 450.7 480 432 480C350 480 279.1 439.8 245.2 381.5C262.5 379.2 279.1 375.3 294.9 369.9C322.9 407.1 373.9 432 432 432C446.2 432 460.8 430.3 475.4 426.1L494.6 422.6L511.3 432.1C528.3 443.5 545.7 451.4 563.5 456.5C559.3 449.9 555.3 442.1 551.7 436.1L535.9 405.8L559.4 380.9C574.3 365.3 592 339.2 592 304C592 237.7 528.7 183.1 447.1 176.6L448 176C448 159.5 445.8 143.5 441.6 128.2H441.6z"/></svg>

- For 4 min, *to yourself*, think what your workflow is *exactly* like for doing
research. As you think of what you do, write them down on the stickies, until the time runs out.
    - Where do you save your files? How do you name your files?
    - What apps do you use?
    - How do you know which files to work on and where you left off?
    - How do you keep track of tasks to do?
    - How do you collaborate and coordinate with others? Email? Shared folders?

- For 2 min, as the whole group we'll briefly go over some of the workflow items.

???

This helps to get us thinking and mentally primed for later activities.

---

## Reproducibility as a spectrum: *Any* improvement is better than no improvement

\+ One folder per manuscript (with associated files)  
\+ Use relative file paths (`data/project-data.csv` vs `C:/User1/some-data-file.csv`)  
\+ Version controlled (like with Git)  
\+ Reproducible document system
\+ Automated and explicit pipeline management (re-generate results with single command)

???

- I wanted you to think about your workflow so you can start appreciating
how even small things can be changed to improve reproducibility.

- And if more reproducible, if things change, its easier to update later work
with literally a push of a button (or single command)

- Some things are easier to do than others, and some require more technical
knowledge and skill than most researchers have the time or motivation to
acquire. (More about encouraging these skills).

- But, we're missing key component here...

---

## Key <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#da9100;overflow:visible;position:relative;"><path d="M336 352c97.2 0 176-78.8 176-176S433.2 0 336 0S160 78.8 160 176c0 18.7 2.9 36.8 8.3 53.7L7 391c-4.5 4.5-7 10.6-7 17v80c0 13.3 10.7 24 24 24h80c13.3 0 24-10.7 24-24V448h40c13.3 0 24-10.7 24-24V384h40c6.4 0 12.5-2.5 17-7l33.3-33.3c16.9 5.4 35 8.3 53.7 8.3zm40-176c-22.1 0-40-17.9-40-40s17.9-40 40-40s40 17.9 40 40s-17.9 40-40 40z"/></svg> component: Using the same data and analysis to *independently* get the same result

- Can someone else bake the same food without either of the ingredients or the recipe?

???

If you were to give me your code and I had access to your data, would 
I know how you got any given result presented in your paper?

---

## Open science occurs at all stages of research

???

Components of open science are in all stages of research.

---

## Open science is also a spectrum

\+ Open access (like preprints)  
\+ Open protocol  
\+ **Open data/data format**  
\+ **Open analysis plan/code**  
\+ **Open source (like software used)** (1)

.footnote[(1): Example, AU institutionally approves and supports a closed source
software (Stata), reducing reproducibility.]

???

Focus down to reproducibility side (data, code).

E.g. AU has institutionally approved a closed source statistical software
(Stata), which by definition, makes work less open and reproducible (someone
else needs a Stata license to run your software), unlike open source software
like R where anyone can install and run it.

---

## 5 min activity: Think 💭 and share <svg aria-hidden="true" role="img" viewBox="0 0 640 512" style="height:1em;width:1.25em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#da9100;overflow:visible;position:relative;"><path d="M208 0C322.9 0 416 78.8 416 176C416 273.2 322.9 352 208 352C189.3 352 171.2 349.7 153.9 345.8C123.3 364.8 79.13 384 24.95 384C14.97 384 5.93 378.1 2.018 368.9C-1.896 359.7-.0074 349.1 6.739 341.9C7.26 341.5 29.38 317.4 45.73 285.9C17.18 255.8 0 217.6 0 176C0 78.8 93.13 0 208 0zM164.6 298.1C179.2 302.3 193.8 304 208 304C296.2 304 368 246.6 368 176C368 105.4 296.2 48 208 48C119.8 48 48 105.4 48 176C48 211.2 65.71 237.2 80.57 252.9L104.1 277.8L88.31 308.1C84.74 314.1 80.73 321.9 76.55 328.5C94.26 323.4 111.7 315.5 128.7 304.1L145.4 294.6L164.6 298.1zM441.6 128.2C552 132.4 640 209.5 640 304C640 345.6 622.8 383.8 594.3 413.9C610.6 445.4 632.7 469.5 633.3 469.9C640 477.1 641.9 487.7 637.1 496.9C634.1 506.1 625 512 615 512C560.9 512 516.7 492.8 486.1 473.8C468.8 477.7 450.7 480 432 480C350 480 279.1 439.8 245.2 381.5C262.5 379.2 279.1 375.3 294.9 369.9C322.9 407.1 373.9 432 432 432C446.2 432 460.8 430.3 475.4 426.1L494.6 422.6L511.3 432.1C528.3 443.5 545.7 451.4 563.5 456.5C559.3 449.9 555.3 442.1 551.7 436.1L535.9 405.8L559.4 380.9C574.3 365.3 592 339.2 592 304C592 237.7 528.7 183.1 447.1 176.6L448 176C448 159.5 445.8 143.5 441.6 128.2H441.6z"/></svg> about potential *benefits*

> What might be some ***benefits*** to being more reproducible and open at NCRR
in your research group and/or individual level (from the trainee to those more
established)?

- For 1 min, *to yourself*, think about the question.

- For 2 min, discuss with your neighbour what you've thought about.

- For 2 min, we'll have a group-wide sharing of some of the thoughts.

---

> What might be some ***barriers*** to being more reproducible and open at NCRR
in your research group and/or individual level (from knowledge and technical
capacity, to what gets supported and what doesn't)?

- For 1 min, *to yourself*, think about the question.

- For 2 min, discuss with a different neighbour what you've thought about.

- For 2 min, we'll have a group-wide sharing of some of the thoughts.

---

## Possible social and technical actions to moving toward openness and reproducibility

- Reviewing (pre-analysis)
    - Data cleaning plan 
    - Data analysis plan

- Agreeing to all write and publish protocols with analysis plans for projects
]
    
--

.pull-left[
- Decide on a standard folder and file structure for each project/manuscript for your
group

- Have frequent informal "lightning round" of code/skill sharing

- **Publicly link your code with your research output** using GitHub and Zenodo
]

???

Now that we're getting more into thinking about benefits and barriers, we
can get into thinking about potential actions to take to be more reproducible and 
open. Before getting into brainstorming activities, I want to give you some
ideas for actions to help prime your thinking.

- Work with/pressure DST to make it easier to download code from server to
eventually share (something we'll start doing more at SDCA)
  
---

## 10 min activity: Think 💭, brainstorm, then share <svg aria-hidden="true" role="img" viewBox="0 0 640 512" style="height:1em;width:1.25em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#da9100;overflow:visible;position:relative;"><path d="M208 0C322.9 0 416 78.8 416 176C416 273.2 322.9 352 208 352C189.3 352 171.2 349.7 153.9 345.8C123.3 364.8 79.13 384 24.95 384C14.97 384 5.93 378.1 2.018 368.9C-1.896 359.7-.0074 349.1 6.739 341.9C7.26 341.5 29.38 317.4 45.73 285.9C17.18 255.8 0 217.6 0 176C0 78.8 93.13 0 208 0zM164.6 298.1C179.2 302.3 193.8 304 208 304C296.2 304 368 246.6 368 176C368 105.4 296.2 48 208 48C119.8 48 48 105.4 48 176C48 211.2 65.71 237.2 80.57 252.9L104.1 277.8L88.31 308.1C84.74 314.1 80.73 321.9 76.55 328.5C94.26 323.4 111.7 315.5 128.7 304.1L145.4 294.6L164.6 298.1zM441.6 128.2C552 132.4 640 209.5 640 304C640 345.6 622.8 383.8 594.3 413.9C610.6 445.4 632.7 469.5 633.3 469.9C640 477.1 641.9 487.7 637.1 496.9C634.1 506.1 625 512 615 512C560.9 512 516.7 492.8 486.1 473.8C468.8 477.7 450.7 480 432 480C350 480 279.1 439.8 245.2 381.5C262.5 379.2 279.1 375.3 294.9 369.9C322.9 407.1 373.9 432 432 432C446.2 432 460.8 430.3 475.4 426.1L494.6 422.6L511.3 432.1C528.3 443.5 545.7 451.4 563.5 456.5C559.3 449.9 555.3 442.1 551.7 436.1L535.9 405.8L559.4 380.9C574.3 365.3 592 339.2 592 304C592 237.7 528.7 183.1 447.1 176.6L448 176C448 159.5 445.8 143.5 441.6 128.2H441.6z"/></svg> *short-term, easier* options

> What are some potential options that you and your research group could
implement ***relatively easily and within the short-term (<6 months)*** to be more
open and reproducible?

- For 2 min, think about ideas to this question (and write down if you want to on the stickies)

- For 3 min, brainstorm with in your group (I'll make them) on what you've all thought about

- For 2 min, decide on one person to write down some of the ideas on the sheet of paper

- For 3 min, we'll briefly go over and discuss some of the ideas

---

> What are some potential options that you and your research group could implement
that are ***more difficult and within the longer-term (>1 year)*** to be more open
and reproducible?

- For 2 min, think about ideas to this question (and write down if you want to on the stickies)

- For 3 min, brainstorm with your group (I'll make them) on what you've all thought about

- For 2 min, decide on one person to write down some of the ideas on the sheet of paper

- For 3 min, we'll briefly go over and discuss some of the ideas

---

## Resources

- Introduction course to Reproducible Research in R: 
[r-cubed.rostools.org](https://r-cubed.rostools.org/)
    - [Further learning resources](https://r-cubed.rostools.org/resources.html#further-learning)

- Intermediate course to Reproducible Research in R:
[r-cubed-intermediate.rostools.org](https://r-cubed-intermediate.rostools.org/)
    - [Further learning resources](https://r-cubed-intermediate.rostools.org/resources.html#for-continued-learning)