Authors: Mosab Khayat, Morteza Karimzadeh, David Ebert, Arif Ghafoor
Abstract: Many evaluation methods have been applied to assess the usefulness of Visual Analytics (VA) solutions. These methods branch from a variety of origins with different assumptions and goals which cause confusion about their proofing capabilities. Moreover, the seldom discussion about the evaluation processes may limit our potentials to develop new evaluation methods specialized for VA. In this paper, we present an analysis of summative evaluation methods that have been used to assess VA solutions. We propose a new metric called summative quality and use it during the analysis to rank evaluation methods according to their quality of proving usefulness. First, we provide a survey of the evaluation methods that appear in VAST literature in the past two years. Then, we show our analysis of these methods in terms of validity and generalizability of their findings, as well as the feasibility of using them.