While writing a paper recently, I started wondering if I should use reproducible or repeatable when talking about results and experiments. Knowing their purpose and relevance, I was curious if there were definitions that were agreed upon in the scientific community.
After some digging, I found definitions from ACM. Then, I found someone raising concerns about it.
Then, I found another related source titled Replicability vs. reproducibility — or is it the other way around? Again, with concerns about if the terms were “right”. While there were definitions, there was no consensus.
A bit disappointed that there were no agreed upon definitions, I decided to go to the basics — cracked open a dictionary to get the definitions of these verbs.
Repeat: do (something) again, either once or a number of times
Reproduce: produce again
In these basic definitions,
- repeat is used in the context of (sequence of) actions or dynamic “things” while reproduce is used in the context of objects or static “things”.
- repeat focuses on performing actions and not on observed results/objects while reproduce focuses on observed results/objects and not on the actions leading to results/objects.
- neither repeat nor reproduce impose any constrain the subject performing the actions or producing the results.
In a scientific setting, experiments involve actions while results are objects. So, I think we should use repeat in conjunction with experiments and reproduce in conjunction with results.
Experiments are/should be repeatable and results are/should be reproducible.
Further, observe that repeating an experiment does not imply reproducing a result. Likewise, reproducing a result does not imply repeating an experiment.
Here are few example situations described using the above definitions.
- All 10 subsequent repetitions of our experiment X reproduced result R.
- Group H could (not) reproduce result R by repeating Group G’s experiment X.
- We reproduced result R by repeating Group G’s experiment X by replacing module M with N (or data set K with L).
- We were able to reproduce result R from Group G’s experiment X in a new experiment Y. **
While I understand the reasons to be specific about who conducted the experiment, what data set was used, and who observed the results, we don’t need a dedicated term to capture these specifics. I think we can do just fine with two well-defined terms and few more short phrases.
** I think a modified version of an experiment is a different experiment; be it one or more different steps, one or more different parameter values, or a modified/different data set. So, I would prefer to write 4 as We reproduced result R from experiment X with experiment Y, a modified version of experiment X, and here is the description of Y.