Simon Price and Peter Flach discuss "Computational Support for Academic Peer Review" (cacm.acm.org/magazines/2017/3/213825), a Review Article in the March 2017 CACM.
00:00 That stack of papers. Every academic knows it. Especially conference organizers and journal editors.
00:10 They have to assign hundreds of submissions among dozens of reviewers to select the handful of papers that they hope will best advance their field.
00:19 It's a dizzying task that's mostly been guided by intuition and experience.
00:25 Join us as we talk with two researchers who believe that machine learning and other recent advances can improve the process, in Computational Support for Academic Peer Review.
00:38 [Intro graphics/music]
00:47 As a Program Co-chair responsible for an ACM datamining conference in 2009, University of Bristol professor Peter Flach had a puzzle to solve.
00:58 DR. FLACH: And so if we have 500 papers times three reviews it's 1,500 reviews, and that needs to distributed over let's say 300 program committee members.
01:08 He could have used traditional methods to allocate the papers, relying on his personal contacts and knowledge of the field. Instead, he turned to Simon Price, the university's Academic Research IT Manager.
01:21 DR. PRICE: Traditionally, the way that papers and reviewers have been characterized has been through keywords. But ... manual processes like this are known to be problematic.
01:35 As an expert in machine learning, Dr. Flach jumped at the chance to automate the process.
01:40 DR. FLACH: We develop all these clever techniques to do document classification, to do text mining, to do all sorts of machine learning. But we forget about applying those in our own daily work.
01:55 Earlier researchers had laid the groundwork by developing techniques that Drs. Simon and Flach used, like the so-called "bag of words".
02:03 DR. SIMON: So all the words in this paper, how similar are they to all the words in the publications of this potential reviewer?
02:15 They further automated the process by mining public information sources.
02:20 DR. SIMON: So, rather than asking people to provide lists of their publications, for instance, for the program committee, we take that information, built a harvester to take that information from a very-good quality online bibliography -- DBLP.
02:40 They developed the system named SubSift, and Dr. Flach uses it every week as editor of the Journal, Machine Learning.
02:48 DR. FLACH: For each paper I get a ranked list of my associate editors and then I decide who is the most appropriate editor to handle the paper. And then the same system is also available to the associated editors, so they can then rank the editorial board members.
03:09 They believe that such an automated approach could provide the power needed to consider papers for multiple conferences at once.
03:07 DR. FLACH: If I'm concentrating on editing the journal, then I'm taking essentially a published/not published decision. If I'm taking a bigger view as a computer scientist, then I could ask a different question. Here is a paper: Which venue is this paper best suited for?
03:37 Get all the details in the March 2017 issue of Communications of the ACM, in the Review article, "Computational Support for Academic Peer Review: A Perspective from Artificial Intelligence".
03:51 [Outro and credits]