Workshop on reproducibility and open science

# Workshop on reproducibility and open science
### Luke W. Johnston

---

</style>

.footer-left[
Slides: [slides.lwjohnst.com/au/2022-04-25](https://slides.lwjohnst.com/au/2022-04-25/)  
Licensed under CC-BY <svg aria-hidden="true" role="img" viewBox="0 0 496 512" style="height:1em;width:0.97em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#990033;overflow:visible;position:relative;"><path d="M314.9 194.4v101.4h-28.3v120.5h-77.1V295.9h-28.3V194.4c0-4.4 1.6-8.2 4.6-11.3 3.1-3.1 6.9-4.7 11.3-4.7H299c4.1 0 7.8 1.6 11.1 4.7 3.1 3.2 4.8 6.9 4.8 11.3zm-101.5-63.7c0-23.3 11.5-35 34.5-35s34.5 11.7 34.5 35c0 23-11.5 34.5-34.5 34.5s-34.5-11.5-34.5-34.5zM247.6 8C389.4 8 496 118.1 496 256c0 147.1-118.5 248-248.4 248C113.6 504 0 394.5 0 256 0 123.1 104.7 8 247.6 8zm.8 44.7C130.2 52.7 44.7 150.6 44.7 256c0 109.8 91.2 202.8 203.7 202.8 103.2 0 202.8-81.1 202.8-202.8.1-113.8-90.2-203.3-202.8-203.3z"/></svg>
]

---

<div>
<style type="text/css">.xaringan-extra-logo {
width: 60px;
height: 128px;
z-index: 0;
background-image: url(../../common/sdca-logo.png);
background-size: contain;
background-repeat: no-repeat;
position: absolute;
top:1em;right:1em;
}
</style>
<script>(function () {
  let tries = 0
  function addLogo () {
    if (typeof slideshow === 'undefined') {
      tries += 1
      if (tries < 10) {
        setTimeout(addLogo, 100)
      }
    } else {
      document.querySelectorAll('.remark-slide-content:not(.title-slide):not(.inverse):not(.hide_logo)')
        .forEach(function (slide) {
          const logo = document.createElement('a')
          logo.classList = 'xaringan-extra-logo'
          logo.href = 'https://www.stenoaarhus.dk/'
          slide.appendChild(logo)
        })
    }
  }
  document.addEventListener('DOMContentLoaded', addLogo)
})()</script>
</div>

<!-- 
70 min:

15
+ 6 (activity) = 21
+ 10 = 31
+ 5*2 (activity) = 41
+ 5 = 46
+ 10*2 (activity) = 66

- 2 * 10 min activity
- 2 *

Introduce myself and my work.

- My work and R courses?
    - DDA
    - I know that several people from NCRR have taken my R course

-->

## <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#990033;overflow:visible;position:relative;"><path d="M80 368H16a16 16 0 0 0-16 16v64a16 16 0 0 0 16 16h64a16 16 0 0 0 16-16v-64a16 16 0 0 0-16-16zm0-320H16A16 16 0 0 0 0 64v64a16 16 0 0 0 16 16h64a16 16 0 0 0 16-16V64a16 16 0 0 0-16-16zm0 160H16a16 16 0 0 0-16 16v64a16 16 0 0 0 16 16h64a16 16 0 0 0 16-16v-64a16 16 0 0 0-16-16zm416 176H176a16 16 0 0 0-16 16v32a16 16 0 0 0 16 16h320a16 16 0 0 0 16-16v-32a16 16 0 0 0-16-16zm0-320H176a16 16 0 0 0-16 16v32a16 16 0 0 0 16 16h320a16 16 0 0 0 16-16V80a16 16 0 0 0-16-16zm0 160H176a16 16 0 0 0-16 16v32a16 16 0 0 0 16 16h320a16 16 0 0 0 16-16v-32a16 16 0 0 0-16-16z"/></svg> Outline

- Learning objectives

- Main take-home message

- Introduce and describe reproducibility and open science

- Activities to discuss and brainstorm (~40 min)

- Resources

---

## Learning objectives of this workshop:

1. Become aware of what reproducible and open practices are

2. Hear and learn about what the experiences and thoughts of your colleagues are

3. Learn about some simple ways to become more open and reproducible

4. Identify ways to adopt some of these practices at NCRR (1)

---

**Take-home message:**

## Publicly share your code from your research

???

And I will add, share you code at *any* stage of your research. From beginning,
to end, to middle.

---

## Why are reproducible and open practices important right now? 🤔

**They are part of multiple large trends like team science, computing, meta-research, higher quality/rigor**

???

(Raise of hands) How many could describe the difference between reproducibility
and replicability?

---

## Reproducibility: Same data + analysis = same result?

- Also called analytic/computational reproducibility or reproducible data
analysis.
- *Independently* reproduce results in paper based on same analysis and data.

---

## Reproducibility is a spectrum

???

Part of this workshop and a lot of my teaching and work is to get that us as a
community to move closer to this end.

Reproducibility is about HOW EXACTLY a finding was found in a study.

---

## 6 min activity: Think 💭about your research workflow, then share <svg aria-hidden="true" role="img" viewBox="0 0 576 512" style="height:1em;width:1.12em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#da9100;overflow:visible;position:relative;"><path d="M416 192c0-88.4-93.1-160-208-160S0 103.6 0 192c0 34.3 14.1 65.9 38 92-13.4 30.2-35.5 54.2-35.8 54.5-2.2 2.3-2.8 5.7-1.5 8.7S4.8 352 8 352c36.6 0 66.9-12.3 88.7-25 32.2 15.7 70.3 25 111.3 25 114.9 0 208-71.6 208-160zm122 220c23.9-26 38-57.7 38-92 0-66.9-53.5-124.2-129.3-148.1.9 6.6 1.3 13.3 1.3 20.1 0 105.9-107.7 192-240 192-10.8 0-21.3-.8-31.7-1.9C207.8 439.6 281.8 480 368 480c41 0 79.1-9.2 111.3-25 21.8 12.7 52.1 25 88.7 25 3.2 0 6.1-1.9 7.3-4.8 1.3-2.9.7-6.3-1.5-8.7-.3-.3-22.4-24.2-35.8-54.5z"/></svg>

- For 4 min, *to yourself*, think what your workflow is *exactly* like for doing
research. As you think of what you do, add them to the Mentimeter, until the time runs out.
    - Where do you save your files? How do you name your files?
    - What apps do you use?
    - How do you know which files to work on and where you left off?
    - How do you keep track of tasks to do?
    - How do you collaborate and coordinate with others? Email? Shared folders?

- For 2 min, as the whole group we'll briefly go over some of the workflow items.

???

This helps to get us thinking and mentally primed for later activities.

---

## Reproducibility as a spectrum

\+ One folder per manuscript (with associated files)  
\+ Relative file paths used (`data/project-data.csv` vs `C:/User1/some-data-file.csv`)  
\+ Version controlled (like with Git)  
\+ Automated and explicit pipeline management (re-generate results with single command)  
\+ Reproducible document system

???

- I wanted you to think about your workflow so you can start appreciating
how even small things can be changed to improve reproducibility.

- And if more reproducible, if things change, its easier to update later work
with literally a push of a button (or single command)

- Some things are easier to do than others, and some require more technical
knowledge and skill than most researchers have the time or motivation to
acquire. (More about encouraging these skills).

- But, we're missing key component here...

---

## Key <svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#da9100;overflow:visible;position:relative;"><path d="M512 176.001C512 273.203 433.202 352 336 352c-11.22 0-22.19-1.062-32.827-3.069l-24.012 27.014A23.999 23.999 0 0 1 261.223 384H224v40c0 13.255-10.745 24-24 24h-40v40c0 13.255-10.745 24-24 24H24c-13.255 0-24-10.745-24-24v-78.059c0-6.365 2.529-12.47 7.029-16.971l161.802-161.802C163.108 213.814 160 195.271 160 176 160 78.798 238.797.001 335.999 0 433.488-.001 512 78.511 512 176.001zM336 128c0 26.51 21.49 48 48 48s48-21.49 48-48-21.49-48-48-48-48 21.49-48 48z"/></svg> component: Using the same data and analysis to *independently* get the same result

???

If you were to give me your code and I had access to your data, would 
I know how you got any given result presented in your paper?

---

## Open science occurs at all stages of research

???

Components of open science are in all stages of research.

---

## Open science is also a spectrum

\+ Open access (like preprints)  
\+ Open protocol  
\+ **Open data/data format**  
\+ **Open analysis plan/code**  
\+ **Open source (like software used)** (1)

.footnote[(1): Example, AU institutionally approves and supports a closed source
software (Stata), reducing reproducibility.]

???

Focus down to reproducibility side (data, code).

E.g. AU has institutionally approved a closed source statistical software
(Stata), which by definition, makes work less open and reproducible (someone
else needs a Stata license to run your software), unlike open source software
like R where anyone can install and run it.

---

## 5 min activity: Think 💭 and share <svg aria-hidden="true" role="img" viewBox="0 0 576 512" style="height:1em;width:1.12em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#da9100;overflow:visible;position:relative;"><path d="M416 192c0-88.4-93.1-160-208-160S0 103.6 0 192c0 34.3 14.1 65.9 38 92-13.4 30.2-35.5 54.2-35.8 54.5-2.2 2.3-2.8 5.7-1.5 8.7S4.8 352 8 352c36.6 0 66.9-12.3 88.7-25 32.2 15.7 70.3 25 111.3 25 114.9 0 208-71.6 208-160zm122 220c23.9-26 38-57.7 38-92 0-66.9-53.5-124.2-129.3-148.1.9 6.6 1.3 13.3 1.3 20.1 0 105.9-107.7 192-240 192-10.8 0-21.3-.8-31.7-1.9C207.8 439.6 281.8 480 368 480c41 0 79.1-9.2 111.3-25 21.8 12.7 52.1 25 88.7 25 3.2 0 6.1-1.9 7.3-4.8 1.3-2.9.7-6.3-1.5-8.7-.3-.3-22.4-24.2-35.8-54.5z"/></svg> about potential *benefits*

> What might be some ***benefits*** to being more reproducible and open at NCRR
at the organizational, group, and/or individual level (from the trainee to those
more established)?

- For 1 min, *to yourself*, think about the question.

- For 2 min, discuss with your neighbour what you've thought about.

- For 2 min, we'll have a group-wide sharing of some of the thoughts.

---

> What might be some ***barriers*** to being more reproducible and open at NCRR
at the organizational, group, and/or individual level (from knowledge and technical
capacity, to what gets supported and what doesn't)?

- For 1 min, *to yourself*, think about the question.

- For 2 min, discuss with your neighbour what you've thought about.

- For 2 min, we'll have a group-wide sharing of some of the thoughts.

---

## Possible social and technical actions to moving toward openness and reproducibility

- Reviewing (pre-analysis)
    - Data cleaning plan 
    - Data analysis plan

- Discuss with DST to support reproducible/open systems (within law)
]
    
--

- Decide on standard folder and file structure for each project/manuscript
]

???

Now that we're getting more into thinking about benefits and barriers, we
can get into thinking about potential actions to take to be more reproducible and 
open. Before getting into brainstorming activities, I want to give you some
ideas for actions to help prime your thinking.

- Work with/pressure DST to make it easier to download code from server to
eventually share (something we'll start doing more at SDCA)
  
---

## 10 min activity: Think 💭, brainstorm, then share <svg aria-hidden="true" role="img" viewBox="0 0 576 512" style="height:1em;width:1.12em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#da9100;overflow:visible;position:relative;"><path d="M416 192c0-88.4-93.1-160-208-160S0 103.6 0 192c0 34.3 14.1 65.9 38 92-13.4 30.2-35.5 54.2-35.8 54.5-2.2 2.3-2.8 5.7-1.5 8.7S4.8 352 8 352c36.6 0 66.9-12.3 88.7-25 32.2 15.7 70.3 25 111.3 25 114.9 0 208-71.6 208-160zm122 220c23.9-26 38-57.7 38-92 0-66.9-53.5-124.2-129.3-148.1.9 6.6 1.3 13.3 1.3 20.1 0 105.9-107.7 192-240 192-10.8 0-21.3-.8-31.7-1.9C207.8 439.6 281.8 480 368 480c41 0 79.1-9.2 111.3-25 21.8 12.7 52.1 25 88.7 25 3.2 0 6.1-1.9 7.3-4.8 1.3-2.9.7-6.3-1.5-8.7-.3-.3-22.4-24.2-35.8-54.5z"/></svg> *short-term, easier* options

> What are some potential options that you, your group, and NCRR could
implement ***relatively easily and within the short-term (<1 year)*** to be more
open and reproducible?

- For 2 min, think about ideas to this question (and write down if you want to)

- For 3 min, brainstorm with your neighbour(s) on what you've all thought about

- For 2 min, decide on one person to write down some of the ideas in the Mentimeter

- For 3 min, we'll briefly go over and discuss some ideas from the Mentimeter

---

> What are some potential options that you, your group, and NCRR could implement
that are ***more difficult and within the longer-term (>1 years)*** to be more open
and reproducible?

- For 2 min, think about ideas to this question (and write down if you want to)

- For 3 min, brainstorm with your neighbour(s) on what you've all thought about

- For 2 min, decide on one person to write down some of the ideas in the Mentimeter

- For 3 min, we'll briefly go over and discuss some ideas from the Mentimeter

---

## Resources

- Introduction course to Reproducible Research in R: 
[r-cubed.rostools.org](https://r-cubed.rostools.org/)
    - [Further learning resources](https://r-cubed.rostools.org/resources.html#further-learning)

- Intermediate course to Reproducible Research in R:
[r-cubed-intermediate.rostools.org](https://r-cubed-intermediate.rostools.org/)
    - [Further learning resources](https://r-cubed-intermediate.rostools.org/resources.html#for-continued-learning)