City Research Online

A framework for evaluating automatic indexing or classification in the context of retrieval

Golub, K., Soergel, D., Buchanan, G. , Tudhope, D., Lykke, M. & Hiom, D. (2016). A framework for evaluating automatic indexing or classification in the context of retrieval. Journal of the Association for Information Science and Technology, 67(1), pp. 3-16. doi: 10.1002/asi.23600

Abstract

Tools for automatic subject assignment help deal with scale and sustainability in creating and enriching metadata, establishing more connections across and between resources and enhancing consistency. While some software vendors and experimental researchers claim the tools can replace manual subject indexing, hard scientific evidence of their performance in operating information environments is scarce. A major reason for this is that research is usually conducted in laboratory conditions, excluding the complexities of real-life systems and situations. The paper reviews and discusses issues with existing evaluation approaches such as problems of aboutness and relevance assessments, implying the need to use more than a single “gold standard” method when evaluating indexing and retrieval and proposes a comprehensive evaluation framework. The framework is informed by a systematic review of the literature on indexing, classification and approaches: evaluating indexing quality directly through assessment by an evaluator or through comparison with a gold standard; evaluating the quality of computer-assisted indexing directly in the context of an indexing workflow, and evaluating indexing quality indirectly through analyzing retrieval performance.

Publication Type: Article
Additional Information: This is the peer reviewed version of the following article:Golub, K., Soergel, D., Buchanan, G., Tudhope, D., Lykke, M. & Hiom, D. (2016). A framework for evaluating automatic indexing or classification in the context of retrieval. Journal of the Association for Information Science and Technology, 67(1), pp. 3-16., which is published in final form at http://dx.doi.org/10.1002/asi.23600. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Self-Archiving.
Subjects: Z Bibliography. Library Science. Information Resources > Z665 Library Science. Information Science
Departments: School of Science & Technology > Computer Science > Human Computer Interaction Design
SWORD Depositor:
[thumbnail of EvaluationFramework_v19_notable.pdf]
Preview
Text - Accepted Version
Download (390kB) | Preview

Export

Add to AnyAdd to TwitterAdd to FacebookAdd to LinkedinAdd to PinterestAdd to Email

Downloads

Downloads per month over past year

View more statistics

Actions (login required)

Admin Login Admin Login