Advancing the Assessment of Automated Deception Detection Systems: Incorporating Base Rate and Cost into System Evaluation
Document Type
Article
Publication Date
5-2019
Abstract
In the last two decades, there has been an increased interest in automated deception detection systems (ADDs) for use in screening, although little attention has been paid to the usefulness of these systems. ADDs use various means, both invasive and non-invasive, to ascertain individual intent to deceive or engage in malicious behaviour. Many papers introducing ADDs use signal detection theory to compare a technique's ability to detect malicious intent with other techniques, but in doing so, they do not include contextual information such as base rate and cost. In this paper, we aim to improve future research by showing how the inclusion of contextual information provides a more realistic picture of the research. Through both theoretical arguments and a real-data example, we show that especially for those contexts where malicious intent is infrequent (ie, with low base rates of deception) that not factoring in the base rate overestimates the accuracy and therefore usefulness. We conclude with recommendations for how future research should provide a fuller picture of the accuracy and usefulness of ADDs.
Publication Information
Twitchell, Douglas P. and Fuller, Christie M.. (2019). "Advancing the Assessment of Automated Deception Detection Systems: Incorporating Base Rate and Cost into System Evaluation". Information Systems Journal, 29(3), 738-761. https://doi.org/10.1111/isj.12231