City Research Online

Abstract

Objective: To investigate whether performance (number of correct decisions) of humans supported by a computer alerting tool can be improved by tailoring the tool’s alerting threshold (sensitivity/specificity combination) according to user ability and task difficulty.

Background: Many researchers implicitly assume that for each tool there exists a single ideal threshold. But research shows the effects of alerting tools on decision errors to vary depending on variables such as user ability and task difficulty. These findings motivated our investigation.

Method: Forty-seven participants edited text passages, aided by a simulated spell-checker tool. We experimentally manipulated passage difficulty and tool alerting threshold, measured participants’ editing and dictation ability, and counted participants’ decision errors (false positives + false negatives). We investigated whether alerting threshold, user ability, task difficulty and their interactions affected error count.

Results: Which alerting threshold better helped a user depended on an interaction between user ability and passage difficulty. Some effects were large: for higher ability users, a more sensitive threshold reduced errors by 30%, on the easier passages. Participants were not significantly more likely to prefer the alerting threshold with which they performed better.

Conclusion: Adjusting alerting thresholds for individual users’ ability and task difficulty could substantially improve effects of automated alerts on user decisions. This potential deserves further investigation. Improvement size and rules for adjustment will be application-specific.

Application: Computer alerting tools have critical roles in many domains. Designers should assess potential benefits of adjustable alerting thresholds for their specific CAT application. Guidance for choosing thresholds will be essential for realizing these benefits in practice.

Précis: Computer alerting tools (CATs) are useful and ubiquitous. A CAT’s effects depend on its alerting threshold (sensitivity/specificity combination). This study suggests that adjusting CAT thresholds, according to the skills of the user and the difficulty of decisions required, has the potential to substantially improve user decisions.

Publication Type:	Monograph (Technical Report)
Publisher Keywords:	human error analysis, adaptive automation, human-automation interaction, expert-novice differences, warning systems
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Departments:	School of Science & Technology > Computer Science School of Science & Technology > Computer Science > Software Reliability

Export

Downloads