Expectations for paper reviewing should be more clearly communicated. It is not a complicated process, but it is a time consuming one. Through my Ph.D. I have reviewed papers for 8 different conference/journal venues (and many of them multiple times) at the intersection of machine learning and robotics. I may have overdone it on reviewing, but it should feel like the responsible thing to do.
Here is the process I have converged at for reviewing (with a few knobs being fine tuned). Let me know if there is something I should add!
My process is split into two phases, a deep reading phase (1-2 hours) where I try to have an open mind and be curious (though at the end I am normally like WTF??? WHY? UGH) and a clear feedback phase (1-4 hours). I do not start writing anything in a review until I have completely read the paper and am starting to develop an understanding for its point of view and execution.
The cumulative mental energy in both of them is similar, though I find the deep reading phase to be concentrated in a shorter time period of a couple hours. It is really crucial for it to be hyper-focused, else I end up having read a paper without anything to say. Thankfully, this paper reading process is similar to what I do when a friend asks for advice on a paper. It’s less of a cognitive load to have one mode of deep paper reading, so you don’t have to switch hats between friend, collaborator, and reviewer.
The clear feedback phase takes a lot of time, but normally having done the full read (or two or three) correctly makes the actual write up of the review a formality. The clear read generates feedback and ideas in the margin that you should translate to coherent and meaningful feedback. Though, papers should generally be reviewable by you without much additional searching if you are an expert. Clear feedback takes the form of line numbers, quotes, specific questions, links to relevant papers, and more. The text documents can get quite lengthy (I have maxed out a couple review windows for RAS papers).
You’ll give a good review if you read the paper as if you would a close friend or colleague who asked for feedback. There is no reason to put on a tough face for a review because it happens to be in a conference. Also, you should and can be supportive of what authors have done well in a paper review.
If you don’t have any markup in the margins for about half of a page, you probably did not read closely enough, or you are going to nominate this paper for an Oral Presentation and it’s near perfect.
Here is the specific markup format I use (colors are your choice):
Setup-wise, I use PDF Expert on an iPad Pro with an Apple Pencil. Tablet or physical paper recommended strongly. Zoom in, nit pick, write questions in the margin, draw the diagrams the paper should’ve included.
There are some things that I don’t think are normal in reviews, but I think should be required:
I am figuring out which template I like to use for my reviews still. They are similar, but one breaks down the review into per-section comments after laying out the key points. The other is more high level (let me know which you think is better!).
The key is to write down almost all your thoughts. It takes a ton of time to get all these thoughts down, but it is worth it. Later in the writing process, you realize the pain points of your reading and analysis. This pain points then often take your central points / weakness of the paper. Distinctly summarizing this points is crucial to a good review (writing a lot of points without linking them together is not very useful).
Your review will be way better if you take the time to edit it after you went all the way through, but I understand normally these are deadline pushes so it may slip through the cracks.
You can download my template here.
This is a review in the style of how I would help a colleague or friend when preparing a paper. I think it could be a little over the top when reviewing for conferences, but all of the comments are suggestions, so the authors should be happy with more opportunities to improve.
You can download my template here.
This is what I think most reviewers at comments do (barring the worst reviews that do more harm than good). It’s a general paper summary, and main review, and a review summary.
Choosing a template: honestly I normally do this after my first read. While my default is to do full (and some conferences give you their own format, grrr), if a paper is a clear reject you likely should do the shorter review format. It'll save you a lot of time. In addition, you can do the shorter review template for a clear accept, but go into detail if you're going to nominate it for an award or an oral presentation.