Statistics

• 271 submitted papers
• 66 (=24%) papers accepted (all conditionally)
• 52 program committee members (from 15 countries)
• 130 external reviewers
• up to 16 papers reviewed per PC member
• 1.1M words of review text written

Reviewing process: Things that did not change

• “Lightweight” double-blind reviewing process
  • initial reviews written without authorship knowledge
  • authors revealed during discussion period
• Reviewing broken into two rounds
  • Some submissions rejected after first round (2 reviews): 18%
• Authors wrote rebuttals to reviews

Things that changed from 2017

• Automatic assignment of reviews based on machine learning algorithm
• No physical PC meeting—electronic discussion only
• One large PC—no EPC or ERC
• All papers accepted conditionally and shepherded (for PACM)

Automatic review assignments

The process

• Previous practice: reviewers express preferences on all papers, conference system tries to match reviewers to papers accordingly.
  Problem: scanning almost 300 papers is a lot of work for the commitee.
• Experiment for POPL: automatically assign reviews using Toronto Paper Matching System (TPMS), run by Laurent Charlin at U. Toronto
  
  1. Reviewers provide a link to their list of publications instead of preferences, publication PDFs gathered by crawling web sites.
  2. TPMS extracts text from all pubs, converts to “bag of words”
  3. Unsupervised machine learning algorithm generates preference scores based on similarity between documents.
  4. Assignments chosen to optimize preference scores while respecting conflicts.

T-SNE plot of all submissions

T-SNE plot of all reviewers

(different projection)

Appropriateness

Assignments were about as well suited to expertise as with manual assignment. Reviewers rated themselves as “expert” on 44% of their reviews, the same percentage as in 2017.

Effort from PC

Similar or less work for most PC members

Diversity

Diversity of topics mostly okay, possibly a little too high.

Overall

Range of opinions, skewing positive.

Online PC discussion

Effect on recruitment

A significant positive factor for many PC members

Engagement

Not much effect on perceived engagement

Discussion committee

Assigning a PC member to push discussion along helped

Review quality

Review quality was perceived as about the same. Actually, 15% more review text was generated per paper than in 2017.

Overall reaction

Caveat: online poll evaluating online reviewing could have selection bias

Selected comments from PC (edited)

• I thought the online discussion worked well, provided the PC members and discussion leads were proactive. The trouble is it's far too easy to agree to do lots of other stuff during the discussion period
• I meet interesting people I haven't met before at PC meetings...in a way this is a reward for the hard work you put into it.
• I strongly appreciate the opportunity to gather my thoughts, reread others reviews and explain myself in writing.
• In-person discussion is more efficient because you get a lot more information about how PC members really feel about the paper from informal cues...perhaps schedule Skype calls for specific papers?
• It was good that the reviewing process was in two stages, this forced me (and everyone else) to do and submit reviews in a timely manner and increased the quality of the reviews and discussion IMO.
• The problem was the lack of an overall view of the submissions. With a normal PC meeting, everyone gets that view just by showing up
• Online PC meetings worked pretty well for POPL... make it very very clear to PC members that they must be reactive and participate every single day of the discussion period.

Thank you!

• Everyone who submitted papers
• General chair Ranjit Jhala and other organizers
• Andy Gordon (POPL'17 PC Chair) for great advice
• All reviewers (PC and external)
  Thorsten Altenkirch

  Nada Amin

  Gilles Barthe

  Mark Batty

  Lars Birkedal

  Pavol Cerny

  James Cheney

  Koen Claessen

  William Cook

  Karl Crary

  Loris D'Antoni

  Mila Dalla Preda

  Isil Dillig

  Sophia Drossopoulou

  Constantin Enea

  Azadeh Farzan

  Xinyu Feng

  Atsushi Igarashi

  Laura Kovacs

  Neel Krishnaswami

  Sorin Lerner

  Xavier Leroy

  Conor McBride

  Mira Mezini

  Matthew Might

  Peter Müller

  Santosh Nagarakatte

  Aleksandar Nanevski

  Klaus Ostermann

  Ruzica Piskac

  Nadia Polikarpova

  Zvonimir Rakamaric

  Aseem Rastogi

  Grigore Rosu

  Andrey Rybalchenko

  Amr Sabry

  David Sands

  Davide Sangiorgi

  Peter Sewell

  Zhong Shao

  Yannis Smaragdakis

  Zhendong Su

  Éric Tanter

  Zachary Tatlock

  Tachio Terauchi

  Viktor Vafeiadis

  Philip Wadler

  David Walker

  Stephanie Weirich

  Jean Yang

  Danfeng Zhang
• Thanks for attending!