Wisdom-of-the-Crowd Content Assessment: Human-Mediated Text Representation
An influential cognitive theory of human text comprehension developed in the 70s and 80s by Kintsh and van Dijk still guides research on student learning of reading and writing skills. It posits a conceptual model that differs from the kinds of text representation prevalent in AI and NLP research at that time, or used now, in that it depends on what is in the text, what is inferrable from the text, and what the reader knows. This talk will present an approach to content assessment of text that was originally developed to evaluate machine-generated summaries. It is distinctive in its reliance on model summaries written by a crowd (proficient individuals working independently). The crowd supplies an emergent property of observed content: relative importance. The talk will present similarities of this content assessment approach to coding rubrics used in research on reading and writing skills; two automated approaches to the content assessment; results of a recent experiment comparing the automated approaches to a content coding rubric applied to summaries written by community college students. The conclusion will touch on questions of the feasibility and utility of deriving such a content model without the intermediary wise crowd.
Rebecca J. Passonneau is the Director of the Center for Computational Learning Systems (CCLS) at Columbia University, and a Senior Research Scientist. Her research addresses how the same combination of words has different meanings in different contexts, for both written and spoken language. This question is a particularly challenging object of study, because context has many dimensions. Her recent work investigates content assessment for summarization and reading and writing skills, meaning and action in human-machine dialogue, data mining that links textual and non-textual sources, and word sense annotation and representation. She received her Ph.D. from the Department of Linguistics at the University of Chicago in 1985, and worked at several industry and academic research labs before joining CCLS in 2006. She has over 100 publications in journals and refereed conference proceedings, and has been Principal Investigator or co-Principal Investigator on 17 sponsored projects with funding from 11 government and corporate sources.