STOC/FOCS PC meetings: Does nature of decisions justify cost?

I am just back from the FOCS 2017 program committee (PC) meeting. This is my second time attending a physical PC meeting, both times for FOCS. In the past I was sorry that I had to decline some invitations for personal reasons. Just like the previous instance, I was impressed with the depth and thoroughness of the discussion. I also had a great time sitting in the same room with so many esteemed colleagues, chatting about what’s cool in the new research papers, and cracking nerdy jokes. It is also a privilege for me to hear what everyone has to say about what is going on. I even managed to have some quick research meetings here and there. During the flight I even proved a great theorem watched Trainspotting 2. And I had never been to Caltech: it is a great place with a moving history as told by the book in the drawer next to my bed in the Athenaeum.

However in the rest of the post I want to address the title. Throughout, by “PC meetings”, I mean “physical” meetings as opposed to “virtual.” Before I start, I want to stress two things. First, what I am about to say has absolutely nothing to do with this or any specific STOC/FOCS PC meeting, rather it is aimed to PC meetings in general. Second, I am not saying that the decisions made are “bad.” Indeed I don’t think a substantially better way to make decisions exists (here and in many other cases).

First, the experiment. As mentioned in a previous post, this time around I ran a little experiment: Right before the meeting started, I saved the ranking of the papers, based on the reviews and the offline discussion, weighted by confidence. Then I compared it with the final decisions at the end of the meeting (with the knowledge of the total number of papers accepted). There is a handful of papers that I have a conflict with and so I don’t know about. Those I know about number 88. The two lists of 88 papers have edit distance 19, meaning that you need to change 19 papers in one list to get the other list. This is a 0.216 fraction of the papers. I consider this number negligible. Note that it is based on information that was entered in expectation of resolving many things during the meeting: extra rounds of offline discussion would have probably calibrated this much better. Think also how this compares with the NIPS experiment. In that case, the two lists of 37 papers have edit distance 21 (plus or minus one), which is a lot larger than the above. However STOC/FOCS decisions may be more “stable” than NIPS (would be fun to have any data on this).

A common experience is also that a large fraction of papers is a clear reject, a small fraction is a clear accept, and the rest are more or less rated like novels or movies; and I am not aware of a better rating scheme.

Second, how the decisions are made. I am not sure that decisions made during the meeting are significantly better than decisions made offline (I will use the word “offline” to mean “not in a physical meeting”). A meeting can start at 8:30AM and go past 6PM, with total break time about 1 hour. Many people are also jet lagged or exhausted. Working offline may be better. Sure, in a physical meeting you can literally grab someone by the arm and ask: Do you really think X should be accepted given Y? But in general there is very little time for this. On the other hand, in an offline discussion you can say: OK, we need a little extra expertise on papers x, y, z: reviewer w, please dig into it and report (happened to me). This supposedly should also take place before the physical meeting, but I think it’s done less than it should be, because the feeling is that we are going to have a meeting anyway so we might as well resolve this then. Instead what happens at times during the meeting is that expertise is missing, and there’s just no time to remedy. Sometimes, the in-person decision process can be rather chaotic. I want to stress again that I am not saying that *any* PC is not run well. My point is that this is the nature of the hard decisions that we need to make. Of course, email conversations can get very tedious. And you can also say that if there’s no threat of a physical meeting people feel less pressure to write good reviews and make good decisions, though that’s not my experience from SODA/CCC program committees. A good model could be to have a virtual meeting, with all the participants present at the same time. This could be done early on, and in fact having two such meetings could be the best thing. At my institutions we have many, many meetings virtually. During such meetings, you can still stare at someone in the camera and ask: Do you really think X should be accepted given Y?

There are many venues where decisions are made offline, such as CCC/CRYPTO/SODA/JACM/SICOMP. One can say that STOC/FOCS are more important than CCC/CRYPTO/SODA, but it’s hard to say that they are more important than say JACM. An argument is then that journal decisions are different because they involve a lot more time and care. I tend to disagree with this last point. My impression is that journal and conference reviews are rather indistinguishable when it comes to computing the accept/reject bit. The difference is only in the second-order term: journals can spend more time on details and style.

Third, the location. This meeting was held at Caltech. The Theory festival at Montreal ended with a full day of amazing tutorials on friday. The festival is promoted as a “must-attend” event. So what we are telling the program committee is this: “Look, you really gotta attend the theory festival. If you need to attend anything at all this year, that’s the meeting! Oh, and right after that (4PM) please catch a red-eye Montreal-LAX, and be ready to start at 8:30 AM on Saturday.” Personally, I wanted to attend the theory fest, but I just couldn’t to the above. So I skipped it in favor of the PC meeting.

If we really need to have physical meetings, I think we should combine them with some other popular event, or at least put them where the center of mass is. Typically, none of this happens, and the tradition seems to be that the meeting is held where the PC chair works, which may not even be where the conference is. I am not the only one who feels so, by the way, about this point and the others. In particular a past program committee in an act of rebelion organized a conference for themselves and co-located it with the PC meeting. But that’s unusual, and it’s clearly better to co-locate with the existing events, since attendance is already an issue. One argument is that the PC chair can offer a nice room with lots of support, which is hard to find elsewhere. I think this argument does not stand. The “support” consists in everybody bringing their laptops (an example of “downfall of mankind” discussed earlier), and wireless. I don’t think it’s so hard to get a room with a table and wireless in a hotel or another university. Finally, I think these meetings favor the United States even more disproportionately so than the conference itself. Indeed, STOC/FOCS are at least once in a while held elsewhere (one was in Rome, for example — I didn’t go). I’ve never heard of a meeting done abroad, would be curious to know.

I am told that this year TCC has a physical meeting (not always the case) but that it is co-located with CRYPTO, a conference typically attended by most of the crypto community. This makes sense to me.

Look at some recent STOC/FOCS PC, and think of the cost of flying everybody to the meeting, and think of the result.

Fourth, the time. As discussed earlier, do we really have to hold such meetings during week-ends? Especially in the summer, I don’t know of a reason for this. Why not Tuesday to Thursday? People don’t teach, departments are empty, and flying is less chaotic. Plus maybe you want to spend your summer week-ends going to the beach instead of being shut in a windowless room with dozens of laptops? An argument is that international flights are sometimes cheaper if you stay during the week-end. I am not sure this really makes a difference, for a number of reasons (including: not so many people from abroad, and they typically stay extra days anyway).


My impression is that things are done this way mostly because of inertia: This is how it’s been done, and now good luck changing it. Also, being a STOC/FOCS chair is (or is close to) a once-in-a-lifetime appointment, which clearly one doesn’t want to screw up. Another argument I heard is that physical meetings are a better way to preserve tradition. That is, a young member of the PC learns better the trade through physical interaction.

If you put all the above things together, my impression is that the answer to the title is “no.” The resources are probably enough for at least a fraction of a postdoc, or they could be used to allow more people to actually attend the conference. My concrete proposal is to have the next meeting offline, with a mixture of desynchronized discussion and virtual synchronized meetings. At least, we should try.


7 thoughts on “STOC/FOCS PC meetings: Does nature of decisions justify cost?

  1. I am not sure what you mean by “One can say that STOC/FOCS are more important than CCC/CRYPTO/SODA”. A STOC/FOCS reject appearing in any of other conferences is no big deal for science though it may affect the career of a young researcher.

    I agree with the sentiments in the following para. “This supposedly should also take place before the physical meeting, but I think it’s done less than it should be, because the feeling is that we are going to have a meeting anyway so we might as well resolve this then. Instead what happens at times during the meeting is that expertise is missing, and there’s just no time to remedy. Sometimes, the in-person decision process can be rather chaotic.”

  2. I would even go further and expand your title till “Does Impact of TCS Conferences Justify Costs?”.
    I mean here, of course, not the impact of results submitted/accepted. I rather mean: just seeing physical (mostly much too short) presentations of them, in a condensed row one after another, in few hectic days. And all this in times when one can already have an entire paper printed on the table before this “physical meeting”. Well, I understand pretty well that people (especially young people) want to be accepted at FOCS/STOC (and other “important” conferences) just for their Careers. But is this again not just an inertia nowadays? An extremely costly inertia, in terms of money, time and – not forget – ecology.

    Of course, personal meetings are very important. In this respect, workshops, seminars, schools – with ample time for discussions – are something very different from conferences. But is the impact/cost fraction of “physical presence” at conferences larger than one? Almost no time for discussions, just for hearing short abstracts “in person”. Has someone seen an acknowledgment like “ideas of this paper were born during discussions at XXX conference”? I have seen some for, say, Dagstuhl-Seminars …

    As for PC work: I agree, if properly organized, virtual meetings here fully suffice. Besides
    the money spent (which could be used to sent young people to “more discussion friendly”
    meeting I’ve mentioned above), a big advantage is then the possibility to ask sub-referees once again in case of big deviations. This worked perfect at several PCs I was involved (not “most important” in the “current scale”, but it worked).

    “Sure, in a physical meeting you can literally grab someone by the arm and ask: Do you really think X should be accepted given Y?” Here I completely disagree: PC decisions cannot be “emotional”, in any sense. “Grab someone by the arm” or even “stare at someone in the camera” is a pressure :-)

  3. You forgot the most important reason to avoid weekends (for some of us): time with family!

    Also, while physical PC meetings may be due to “inertia”, it is nowhere near the inertia of the conference system itself. That is, I think having PC meetings in person is just due to no one changing it, rather than any sort of “force” holding it in place. As you point out, many conferences already do not have in-person PC meetings, so what is holding the rest back from switching? I don’t think there are large community / societal forces keeping it that way. Good for you for bringing it up publicly! Have you raised it with the conference organizers / steering committees? If they won’t listen, maybe you should start a petition…

  4. Thanks for the comments everybody! I prefer to disassociate this post from the role of conferences in computer science. That’s a larger discussion that I don’t expect to be resolved soon (and I haven’t said what my own preference is) while I cling to the hope that changes such as the one in this post could occur in the near future.

    1. Re “disassociation”: yes, this inertia does not seem to disappear soon. There were many attempts to turn the wheel, e.g. this by Lance Fortnow, and many others. Nothing big happened. But something still happened: we now have “theory days”. A very interesting change. Hopefully, more will come.

  5. This is still a topic of discussion in 2017? I would have thought that my now most CS PC meetings would be virtual.

    Which ones are physical and which ones are virtual? I ask nonrhetoically as always.

    I have never seen a good argument in favor of physical meetings.

    A few arguments FOR virtiual that I don’t think I’ve seen in the comments above:

    a) People spend MORE time reading papers and LESS time booking travel, travelling, etc.

    b) Global Warming. Might also apply to the broader issue of conferences in general.

    bill g.

    1. Thanks for the comment! I agree, and in particular (b) was also on my mind. But I didn’t want to bring it up for various reasons (e.g. as you say it applies more broadly to conferences). (But if you use the wordweb program like me this issue is on your mind.)

      Would be great to have a list of PC meetings that are virtual vs. not, across science.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s