City Research Online

Parallel methods for the generation of partitioned inverted files

MacFarlane, A., McCann, J. A. and Robertson, S. E. (2005). Parallel methods for the generation of partitioned inverted files. Aslib Proceedings; New Information Perspectives, 57(5), pp. 434-459. doi: 10.1108/00012530510621888

Abstract

Purpose
– The generation of inverted indexes is one of the most computationally intensive activities for information retrieval systems: indexing large multi‐gigabyte text databases can take many hours or even days to complete. We examine the generation of partitioned inverted files in order to speed up the process of indexing. Two types of index partitions are investigated: TermId and DocId.

Design/methodology/approach
– We use standard measures used in parallel computing such as speedup and efficiency to examine the computing results and also the space costs of our trial indexing experiments.

Findings
– The results from runs on both partitioning methods are compared and contrasted, concluding that DocId is the more efficient method.

Practical implications
– The practical implications are that the DocId partitioning method would in most circumstances be used for distributing inverted file data in a parallel computer, particularly if indexing speed is the primary consideration.

Originality/value
– The paper is of value to database administrators who manage large‐scale text collections, and who need to use parallel computing to implement their text retrieval services.

Publication Type: Article
Additional Information: This article is (c) Emerald Group Publishing and permission has been granted for this version to appear here http://openaccess.city.ac.uk/. Emerald does not grant permission for this article to be further copied/distributed or hosted elsewhere without the express permission from Emerald Group Publishing Limited.
Subjects: Z Bibliography. Library Science. Information Resources > Z665 Library Science. Information Science
Departments: School of Mathematics, Computer Science & Engineering > Computer Science > Human Computer Interaction Design
Related URLs:
URI: http://openaccess.city.ac.uk/id/eprint/4457
[img]
Preview
PDF - Accepted Version
Download (218kB) | Preview

Export

Downloads

Downloads per month over past year

View more statistics

Actions (login required)

Admin Login Admin Login