Why Have an External Review Committee?

Background: back in 2007 I established a “review committee” to complement the ISMM’08 program committee.  The back story is that my real target was not ISMM, but rather, the larger conferences such as PLDI and particularly ASPLOS, which I felt at the time were suffering from serious problems in their reviewing process.   I was particularly concerned about a) the lack of depth and breadth within any given committee (a function of a fast growing field and reasonable limits on committee size), and b) a lack of accountability and/or motivation for external reviewers under the conventional mechanisms in place at that time.

The process has since been more widely adopted and at the time of writing is used at ISMM, PLDI, ASPLOS, POPL and ISCA.  My colleague, Kathryn McKinley proposed using the review committee to review program committee submissions.   This idea was taken up by Alex Aiken and used in PLDI 2010, opening up program committee submissions to PLDI for the first time in many years.

I wrote the following back in June 2008, when I pitched the idea of an external review committee to the then ASPLOS chairs:

Overview

A review committee (RC) is intended to replace the role currently filled by ad hoc external review assignments (at PLDI and ASPLOS, for example) in a more structured and accountable way.  The committee is set up ahead of time with members invited in a similar way to PC invites, only that RC members will not attend the PC meeting and will review only a modest  number (3, 4 or 5) of papers.  The RC can be established after the PC, with input from the PC.  The RC is attractive because it is effective, accountable and practical.

Motivation

The motivation for the review committee is threefold: 1) anecdotal evidence of mounting concern at the quality of reviews at PLDI & ASPLOS over the past few years, 2) logistical nightmares associated with ad hoc reviewing, particularly when combined with double-blind submission, and 3) the lack of rigor and accountability associated with existing ad hoc review processes.

Size and Composition

The size of the RC is determined by the number of external reviews required (180, say) and the desired number of reviews per RC member (4, say, which would mean 45 members).  Since the RC is much bigger than the PC, if well composed, it should substantially increase the depth and breadth of expertise among the reviewers.  To help ensure breadth, the PC (established first) can be used to identify potential RC members.  The RC would also include people from recent ASPLOS PCs, thus systematically ensuring some level of continuity of standards while allowing the PC to remain fresh and free from perceptions of insularity.

Accountability

RC members are signed on in a similar way to PC members, agreeing to some set of rules and expectations.  RC members review multiple papers, so are better calibrated than reviewers who review just a single paper.  RC members can participate/observe the author response process, so can improve their review in light of author comments and are held to the accountability standards that author response encourages.  Since there is no strong reason not to allow people onto the RC from one year to the next, good reviewers could be re-used from year to year.

Practicality

Since the RC is composed ahead of time (but after the PC), it takes the lengthy process of identifying external reviewers out of the time-critical loop.  The PC chair can therefore make the invites in a less pressured setting.  The total number of invites is substantially lower than if ad hoc external reviews are used (by a factor of around 3 or 4, since the average number of reviews per external reviewer is around 3 or 4 times higher).  RC members use the conference management software (in START, the RC fits nicely as a “track”), and can therefore bid for papers, likely improving the quality of assignments.  Conflicts are detected by the software, just as they are for the PC (addressing a major logistical headache associated with ad hoc assignments).

Opportunity

The RC creates a new opportunity: this week the SIGPLAN EC discussed using the RC to review PLDI PC submissions in future years (perhaps PLDI 2010, after trialing the RC in 2009), since this allows reviewing without the conflict of interest issues normally involved with PC submissions.   In addition to allowing busy senior members of the community a low-cost opportunity to participate even if they can’t commit to the PC, the RC also allows junior researchers to cut their teeth.   A job well done could lead to a subsequent invite to the PC.

Q&A:

Won’t it mean more work for the community?

Using an RC does not change the number of reviews that are written for the conference, just the means of assigning those reviews.  It is (much) less work for the PC chair and gives the reviewers more credit for their work.

It’s a lot of work to put together a PC; won’t it be more work to do an RC?

The total number of people involved is much less than when using ad hoc review requests, so the total number of requests is significantly reduced.  The PC can contribute to the formation of the RC.  I had 100% acceptance rate on RC invites for ISMM’08 (which I realize is a much smaller concern).  I think this is because most members of our community enjoy doing some reviewing, particularly if it involves getting credit, does not involve travel and involves a modest number of papers.

What if we don’t have an expert in the PC or RC for some paper?

The use of an RC does not preclude ad hoc review assignments, but should make it exceptional, rather than the rule.