Guidelines to Write Useful (Academic) Reviews

I recently had the opportunity to provide inputs to improve the academic reviewing process in CS. I used this opportunity to jot down what reviewers should do to provide useful reviews.

Review Content

1. The review should be limited to the merits and limitations of the work described in the paper. It should not consider what could have been done.
2. For any alternatives (e.g., methodology, techniques, data sets, visualization) that a reviewer thinks should have been used, the reviewer should provide a sound argument why the considered/described work is incorrect and why the alternative would be correct. Such feedback should provide references to the alternatives.
3. Suppose the effort described in the paper uses approach B in context X. The review should not penalize the paper for not using the next best approach A if no prior work has used the approach B in a context similar to X or the current work is evaluating approach B.
4. If a reviewer remarks that some prior work has tackled a similar problem, then the reviewer should provide references to such prior work.
   This advice applies to remarks about the novelty of the work.
5. If a reviewer has feedback about specific claims made in the paper, then, in the review, the reviewer should quote snippets from the paper that imply such claims along with her comments, specifically, when the feedback is negative.
6. If a reviewer believes a specific contribution in a paper can/should be improved, he should provide a couple of suggestions to improve.
7. If a reviewer wants to expose her student to the reviewing process, then both she and her student should review the paper and submit their reviews as two different reviews (by a reviewer and a sub-reviewer). In such cases, the reviewer should review her student’s review with her student before submitting the reviews. She should not delegate the reviewing task to her student when she is the reviewer.

If you have additional guidelines, then please suggest them by leaving a comment :)

Review Structure (added Jul-17-2020)

As a supplement to the guidelines on review content, Tim Menzies suggested the following general guidelines to structure reviews. (Thank you!!)

1. Start by summarizing the contributions of the paper. Use key sentences from the abstract of the paper. May be, mix it with other content from the paper.
2. Provide general comments about the contributions.
3. List the pros of the described work.
4. List the cons of the described work. List these in decreasing order of importance.
5. Describe what needs to change before you accept the paper; maybe, something as simple as “fix the first three issues and limitations listed earlier.”

The above is a good structure for a review. My reviews have a similar structure with minor changes.

1. Summary of the paper. Often this mentions the main contributions that caught my attention in the paper.
2. List of broad-stroke comments about the pros and cons of the described work along with suggestions for improvements. These comments should convey primary reasons for the reviewer’s decision, and they should each be supported by detailed comments. Also, they should suggest what should be fixed in future revisions.
3. List of detailed (often, line-level) comments about specific concerns about the paper. These comments are almost always a copy-n-paste of the notes about concerns and improvement opportunities that I made while reading the paper. (So, I have to make sure my notes are good enough to be copied as is :) BTW, I use this Groovy script to extract my notes from PDFs. )