Some factors affecting relevance judgements
Pope, J. M. (1976). Some factors affecting relevance judgements. (Unpublished Doctoral thesis, The City University)
Abstract
Relevance judgements were obtained from 31 research workers in Universities and Government Institutions, on titles and abstracts. The titles were obtained using SDI profiles for each user. A questionnaire was given to each user to discover their information gathering habits, attitudes towards information and various demographic details.
Signal detection theory was used to obtain a single measure of a user's sensitivity to relevance and the strictness of his criterion of relevance. Four different detection models were examined. It was found that an exponential model was most suitable for analysing relevance data.
Several variables were tested to see whether they affected relevance judgements. Several statistical tests were used to test the effects of the variables on the conditional probabilities of hits and false alarms and also on the sensitivity and criterion measures. It was found that 15 of the 37 variables examined significantly influenced relevance judgements. These variables related to the user's work and to his use of different information sources,
The distributions of the hit and false alarm probabilities were lognormal in all except one case. The variables examined were highly inter-related. Factor analysis and principal component analysis were used to reduce the complexity of these variables and to discover the underlying factors. Ten factors were extracted from a total of 36 variables.
The research students in the sample formed a distinct group in terms of their information gathering habits and attitudes towards information.
An attempt was made to understand the types of relevance judgements made by users by placing each judgement in one of five categories. This arbitrary classification was helpful in examining the reasons why various relevance
decisions were made.
A model of information seeking and acquisition was proposed which takes into account the storage of information in human memory and the way in which information is organized. This model was used to describe the relationships between information need, question asking and the subsequent relevance judgements.
Download (36MB) | Preview
Export
Downloads
Downloads per month over past year
Metadata
Metadata