City Research Online

Abstract

Instructional Multimedia (IMM) applications use a variety of representation media with which the user interacts in order to achieve certain learning goals. IMM software has special needs for usability evaluation techniques beyond the typical desktop, office work applications that the majority of the traditional evaluation methods have been developed for. There have been a limited number of evaluation methods developed specifically for the evaluation of IMM applications. Their effectiveness, however, has not been studied empirically neither in a laboratory nor the field. Thus, no definitive conclusions can be made regarding how effective they are in uncovering valid usability and learning problems, what is the nature of the problems they identify, or how they compare in cost effectiveness.

The first part of this thesis investigates empirically the effectiveness of three expert evaluation methods for IMM. The performance of one cognitive walkthrough, one checklist, and one taxonomy-based approaches was studied against a set of eleven effectiveness criteria, such as method’s validity, thoroughness, reliability, and cost effectiveness. The empirical study found that the three methods are not as effective as practitioners and researchers would like them to be, particularly in identifying valid usability problems. The empirical study also highlighted certain characteristics of the expert evaluation methods which limit their effectiveness.

The second part of the thesis deals with improving the effectiveness of expert evaluation methods for IMM. A set of hypotheses for improving the performance of evaluation methods are formulated, particularly for improving their validity. The hypotheses are informed by the results of the empirical study, as well as theoretical work regarding cognitive and pedagogical implications of multimedia design. The hypotheses were tested empirically by developing a new evaluation method addressing the main limitations of the existing methods, which was then applied by expert evaluators. Improvement in the prediction of valid usability and learning problems, particularly those regarding learner comprehension was observed, together with a decrease in the number of false alarms predicted using the developed expert evaluation method.

The thesis fulfils its goal of providing an improved understanding into the effectiveness of existing expert evaluation methods for IMM and defining the characteristics that constitute an effective expert evaluation method for IMM. The results of the research can be used to inform further effort for developing and assessing the effectiveness of evaluation methods for IMM.

Publication Type:	Thesis (Doctoral)
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Departments:	School of Science & Technology > Computer Science > Human Computer Interaction Design School of Science & Technology > School of Science & Technology Doctoral Theses Doctoral Theses

Export

Downloads