# Reproducibility and Repeatability

While writing a paper recently, I started wondering if I should use reproducible or repeatable when talking about results and experiments. Knowing their purpose and relevance, I was curious if there were definitions that were agreed upon in the scientific community.

After some digging, I found definitions from ACM. Then, I found someone raising concerns about it.

Then, I found another related source titled Replicability vs. reproducibility — or is it the other way around? Again, with concerns about if the terms were “right”. While there were definitions, there was no consensus.

A bit disappointed that there were no agreed upon definitions, I decided to go to the basics — cracked open a dictionary to get the definitions of these verbs.

Repeat: do (something) again, either once or a number of times

Reproduce: produce again

In these basic definitions,

• repeat is used in the context of (sequence of) actions or dynamic “things” while reproduce is used in the context of objects or static “things”.
• repeat focuses on performing actions and not on observed results/objects while reproduce focuses on observed results/objects and not on the actions leading to results/objects.
• neither repeat nor reproduce impose any constrain the subject performing the actions or producing the results.

In a scientific setting, experiments involve actions while results are objects. So, I think we should use repeat in conjunction with experiments and reproduce in conjunction with results.

Further, observe that repeating an experiment does not imply reproducing a result. Likewise, reproducing a result does not imply repeating an experiment.

Here are few example situations described using the above definitions.

1. All 10 subsequent repetitions of our experiment X reproduced result R.
2. Group H could (not) reproduce result R by repeating Group G’s experiment X.
3. We reproduced result R by repeating Group G’s experiment X by replacing module M with N (or data set K with L).
4. We were able to reproduce result R from Group G’s experiment X in a new experiment Y. **

While I understand the reasons to be specific about who conducted the experiment, what data set was used, and who observed the results, we don’t need a dedicated term to capture these specifics. I think we can do just fine with two well-defined terms and few more short phrases.

** I think a modified version of an experiment is a different experiment; be it one or more different steps, one or more different parameter values, or a modified/different data set. So, I would prefer to write 4 as We reproduced result R from experiment X with experiment Y, a modified version of experiment X, and here is the description of Y.

Written by