Would theoreticians of the past do more evaluations/measurements in today’s world?
In a recent discussion on social media about “would Turing have won the Turing aware?” article, someone mentioned that he would not get published in good venues today as his work was too theoretical and lacked measurements.
I disagree with this remark cos’ it does not factor in evolution of the environment and its effect on efforts and contributions.
Compared to 2017, Turing did his work prior to 1954 when he had access to far fewer computing resources and operated in a world that was limited in both physical and information mobility. So, if he had access to the kind of resources and mobility we have now, then he would have likely done more measurements to supplement and support his theoretical work.
I have similar disagreement when CS folks defend shallow evaluations/measurements. Here are few common supporting reasons for such defense and why they are wrong.
- Evaluations take a long time. This is not true given the amount of compute resource available at low cost (and the assumption that the new contributions improve over past contributions). [Amazon Web Services, Microsoft Azure, Google Cloud Platform]
- Real-world software test subjects are unavailable. This is not true given the amount of open source software available freely in public and the variety they offer in different aspects, e.g., structure, purpose, and sizes. [OpenHub, Internet Archive, GitHub, Apache]
- Industry-strength software test subjects are unavailable. This is not true given the number of companies embracing open source and interested in contributing to software studies. The only exception being when evaluation needs access to classified software artifacts.
- Human test subjects are unavailable. This is not true given the increase in the number of software developers and users.
- Efforts involving human subjects are hard. Yes, they are harder but they are not harder than they were in the past. This is supported by the number of tools, technologies, and platforms available to enroll and engage human subjects in qualitative studies.
- Real-world data is unavailable. This is not true given the number of freely and publicly available data sets. The only exception being when evaluation needs access to private data of citizens. [Kaggle, Quandl, Internet Archive]
Now, in many of the above cases, it is true the task of getting the subjects and/or data can be hard. But, is that sufficient reason for shallow evaluation? I’d say no.
In today’s world, there are far few CS scenarios (e.g., quantum computing) in which lack of resources or subjects can be the reason for shallow evaluations. So, instead of being confined by the nature of one’s work (theory or practice) and limited by past approaches to our kind of work, we should strive to use currently available resources to conduct deeper and meaningful evaluations. I believe past pioneers and mavericks did so.