Chair’s guide

Submission time

Follow these steps to prepare to accept paper submissions.

1. Set up PC member accounts. Many PCs are divided into classes, such as “heavy” and “light”, or “PC” and “ERC”. Mark these classes with user tags. It’s also useful to configure tag colors so that PC member names are displayed differently based on class (for instance, heavy PC member names might appear in bold).

2. Set submission policies, including whether submission is blind.

3. Set submission deadlines. Authors first register, then submit their papers, possibly multiple times; they choose for each submitted version whether that version is ready for review. Normally, HotCRP allows authors to update their papers until the deadline, but you can also require that authors “freeze” each submission explicitly; only administrators can update frozen submissions. The only deadline that really matters is the paper submission deadline, but HotCRP also supports a separate paper registration deadline, which will force authors to register a few days before they submit. An optional grace period applies to both deadlines: HotCRP reports the deadlines, but allows submissions and updates post-deadline for the specified grace period. This provides some protection against last-minute server overload and gives authors some slack.

4. Set up the submission form, including whether abstracts are required, whether authors check off conflicted PC members (“Collect authors’ PC conflicts with checkboxes”), and whether authors must enter additional non-PC collaborators, which can help detect conflicts with external reviewers (“Collect authors’ other collaborators as text”). The submission form also can include:

   • PDF format checker. This adds a “Check format” link to the Edit Paper screen. Clicking the link checks the paper for formatting errors, such as going over the page limit. Papers with formatting errors may still be submitted, since the checker itself can make mistakes, but the automated checker leaves cheating authors no excuse.

   • Additional fields such as checkboxes, selectors, freeform text, and uploaded attachments. Checkbox fields might include “Consider this paper for the Best Student Paper award” or “Provide this paper to the European shadow PC.” Attachment fields might include supplemental material. You can search for papers with or without each field.

   • Topics. Authors can select topics, such as “Applications” or “Network databases,” that characterize their paper’s subject areas. PC members express topics for which they have high, medium, and low interest, improving automatic paper assignment. Although explicit preferences (see below) are better than topic-based assignments, busy PC members might not specify their preferences; topic matching lets you do a reasonable job at assigning papers anyway.

5. Take a look at a paper submission page to make sure it looks right.

6. Open the site for submissions. Submissions will be accepted only until the listed deadline.

Assignments

After the submission deadline has passed:

1. Consider checking the papers for anomalies. Withdraw and/or delete duplicates or update details on the paper pages (via “Edit paper”). Also consider contacting the authors of papers that were never officially submitted, especially if a PDF document was uploaded; sometimes a user will uncheck “The paper is ready for review” by mistake.

2. Check for formatting violations (optional). Search > Download > Format check will download a summary report. Serious errors are also shown on paper pages (problematic PDFs are distinguished by an “X”).

3. Prepare the review form. Take a look at the templates to get ideas.

4. Set review policies and deadlines, including reviewing deadlines, whether review is blind, and whether PC members may review any paper (usually “yes” is the right answer).

5. Prepare tracks (optional). Tracks give chairs fine-grained control over PC members’ access rights for individual papers. Example situations calling for tracks include external review committees, PC-paper review committees, and multi-track conferences.

6. Collect review preferences from the PC. PC members can rank-order papers they want or don’t want to review. They can either set their preferences all at once, or (often more convenient) page through the list of submitted paperssetting their preferences on the paper pages.

   If you’d like, you can collect review preferences before the submission deadline. Select “PC can see all registered papers until submission deadline”, which allows PC members to see abstracts for registered papers that haven’t yet been submitted.

7. Check for missing conflicts. HotCRP does not automatically confirm all conflicts, such as conflicts indicated by PC members’ “Collaborators and other affiliations” or their review preferences. Use the conflict assignment tool to search for and confirm such conflicts.

8. Assign reviews. You can make assignments by paper, by PC member, by uploading an assignments file, or, even easier, automatically. PC review assignments can be “primary” or “secondary”; the difference is that primary reviewers are expected to complete their review, but a secondary reviewer can delegate their review to someone else. You can also assign PC “metareviews”. Unlike normal reviewers, a metareviewer can view all other reviews before submitting their own.

   The default assignments pages apply to all submitted papers. You can also assign subsets of papers obtained through search, such as papers with fewer than three completed reviews.

9. Open the site for reviewing.

Chair conflicts

Chairs and system administrators can access any information stored in the conference system, including reviewer identities for conflicted papers. It is easiest to simply accept such conflicts as a fact of life. Chairs who can’t handle conflicts fairly shouldn’t be chairs. However, HotCRP does offer other mechanisms for conflicted reviews.

A PC member can manage the reviewing and discussion process for specific papers. This PC member is called the paper administrator. Use the left-hand side of a paper’s assignment page to enter its administrator. (You may need to “Override conflicts” to access the assignment page.)

Paper administrators have full privilege to assign and view reviews for their papers, and can, for example, use the autoassignment tool. They cannot change conference settings.

Normally, a conflicted chair can easily override their conflict. When a paper has an administrator, however, chair conflicts cannot be overridden.

Paper administrators make life easy for PC reviewers and greatly restrict conflicted chairs’ access. Usually this suffices. For additional privacy, use review tokens, which are completely anonymous review slots. To create a token, an administrator goes to an assignment page and clicks on “Request review” without entering a name or email address. This reports the token, a short string of letters and numbers such as “9HDZYUB”. Any user who knows the token can enter it on HotCRP’s home page, after which the system lets them view the paper and anonymously modify the corresponding “Jane Q. Public” review. True reviewer identities will not appear in HotCRP’s database or its logs. For even more privacy, the paper administrator could collect offline review forms via email and upload them using review tokens; then even web server access logs store only the administrator’s identity.

Before the meeting

1. Collect authors’ responses to the reviews (optional). Authors’ responses (also called rebuttals) let authors correct reviewer misconceptions before decisions are made. Responses are entered into the system as comments. On the decision settings page, update “Collect responses to the reviews,” then send mail to authors informing them of the response deadline. PC members can still update their reviews up to the review deadline; authors are informed via email of any review changes.

2. Set PC can see all reviews if you haven’t already, allowing the program committee to see reviews and scores for non-conflicted papers. (During most conferences’ review periods, a PC member can see a paper’s reviews only after completing their own review for that paper. This supposedly reduces bias.)

3. Examine paper scores, either one at a time or en masse, and decide which papers will be discussed. The tags system can group papers and prepare discussion sets. Use search keywords to, for example, find all papers with at least two overall merit ratings of 2 or better.

4. Assign discussion orders using tags (optional). Common discussion orders include sorted by overall ranking (high-to-low, low-to-high, or alternating), sorted by topic, and grouped by PC conflicts. Explicit tag-based orders make it easier for the PC to follow along.

5. Assign discussion leads (optional). Discussion leads are expected to be able to summarize the paper and the reviews. You can assign leads either paper by paper or automatically.

6. Define decision types (optional). By default, HotCRP has two decision types, “accept” and “reject,” but you can add other types of acceptance and rejection, such as “accept as short paper.”

7. The night before the meeting, download all reviews onto a laptop (Download > All reviews) in case the Internet explodes and you can’t reach HotCRP from the meeting place.

At the meeting

1. The meeting tracker can keep the PC coordinated. Search for papers in whatever order you like (you may want an explicit discussion order). Then open a browser tab to manage the tracker, navigate to the first paper in the order, and select “☟” to activate the tracker. From that point on, PC members see a banner with the tracker tab’s current position in the order:

   

   You can also view the discussion status on the discussion status page.

2. Scribes can capture discussions as comments for the authors’ reference.

3. Paper decisions can be recorded on the paper pages or en masse via search. Use decision settings to expose decisions to PC members if desired.

4. Shepherding (optional). If your conference uses shepherding for accepted papers, you can assign shepherds either paper by paper or automatically.

After the meeting

1. Enter decisions and shepherds if you didn’t do this at the meeting.

2. Give reviewers some time to update their reviews in response to PC discussion (optional).

3. Set “Who can see decisions?” to “Authors, PC members, and reviewers.” This will also make shepherd names visible to authors.

4. Send mail to authors informing them that reviews and decisions are available. The mail can also contain the reviews and comments themselves.

5. Collect final papers (optional). If you’re putting together the program yourself, it can be convenient to collect final versions using HotCRP. Authors upload final versions just as they did submissions. You can then download all final versions as a .zip archive. (The submitted versions are archived for reference.)