I am a standing reviewer for Transactions of ACL (TACL) journal.  A few weeks ago the journal asked me if I was happy to continue as a reviewer, and I instantly agreed, because I actually enjoy reviewing for TACL!   Which is a complete contrast to reviewing for conferences such as ACL, which is a chore that I only do out of a sense of responsibility (and indeed I am doing less of this than I used to).

Below I explain some of the reasons why I find that TACL reviewing is a much better experience for me as a reviewer than ACL-type reviewing.    Perhaps there are some lessons here on the reviewing process.

Proper interaction with authors means I can improve papers

TACL is a journal, which means that I have proper interaction with the authors.    The authors know that they need to address reviewers concerns in order to get their paper published; their paper will not be published until we are happy with their changes.  So authors take reviewer comments very seriously, and often put a lot of “blood, sweat, and tears” into making the changes that we requested.   Usually this process really improves the paper; there are exceptions but most of the time addressing reviewers comments does improve a paper, especially if the reviewers are knowledgeable.  Certainly many of my papers have been **much** improved by reviewers comments!

With ACL reviewing, in contrast, I can make suggestions and comments in my reviews, but authors can and do ignore them.  Partially because they dont have the time to make big changes (usually only a few weeks between receiving reviews and submitting final version of paper), but also because many authors dont see the point (they’ve got an ACL paper on their CV, so why bother spending time tweaking the paper to improve it, best thing CV-wise is to move on to the next paper).  Which is demotivating to me as a reviewer, and indeed makes me less likely to make suggestions in my reviews.

In other words, when I review for TACL, my efforts can really improve the paper,  and make a difference.  When I review for ACL, I am simply recommending accept/reject, which feels somewhat pointless since rejected papers will probably just appear in a different venue (authors will keep on resubmitting them until they get lucky somewhere).

Other reviewers are knowledgable, so I learn more

I feel that I learn things when I review TACL papers.  Partially because the papers themselves are interesting and broaden my horizons (which is perhaps a tribute to the editors who decide which papers I am asked to review), and also partially because the other reviewers are usually very knowledgeable, so that I learn things from reading their reviews and interacting with them.

In ACL reviewing, in contrast. I dont feel that I learn much, because (A) the papers are often boring and (B) the other reviewers are often clueless.  The papers are boring because too many of them focus on winning a “leaderboard” contest.  Ie, they tweak a model, apply it to a known problem, and show a small improvement in a usually dubious metric (often evaluated on a problematic data set).  They also usually make no effort to draw linguistic or computational insights from their work.  And the other reviewers are often junior people who have little experience or knowledge about the field, eg PhD students who have never themselves published in ACL.   My understanding is that everyone who submits a paper to ACL can be asked to be a reviewer, regardless of background; TACL, in contrast, is very selective about its reviewers.

So anyways,  when I review for TACL, I usually learn from both from the papers and the other reviewers. This can happen when I review for ACL, but it is much rarer.

Year-round reviewing is much pleasanter than batch reviewing

Last but not least, TACL reviewing is pleasanter because reviewing is spread throughout the year. I was asked to review 4 papers for TACL in 2019, but I got one paper every 3 months, not 4 papers in a batch.  So I could focus on one paper at a time, which was nice.    This is because TACL is a journal with year-round submission, not a conference where all papers are submitted at the same time.

Also the reviewing forms are simple and the process involves a lot of discussion.  So I feel that I am focusing on the merits of the work and how to improve it, not on filling out a bureaucratic form.

With ACL venues, in contrast, I get a bunch of papers which I need to review in a short time period, usually when I need to also do zillions of other things.  So not very pleasant and indeed somewhat stressful.  Some of the reviewing forms are also quite bureaucratic, which is also off-putting.

So from a process perspective, reviewing for TACL is relatively pleasant and allows me to focus on science; reviewing for ACL venues is less pleasant and feels more bureaucratic.

Final Thoughts

I think the TACL reviewing model works well, for reviewers as well as authors, it would be great if such reviewing was more common in NLP!  This requires moving from conference papers to journal papers, so it wont happen overnight, but hopefully over time the “TACL model” will spread.