Section: New Results
Comparison of Usability Methods: Inspection vs. User Testing
Participant : Dominique Scapin.
In the domain of user interface usability, a very active research field aims at defining new evaluation methods and at improving existing ones. A study has been conducted to compare threes usability evaluation methods: User Testing (UT), Document-based Inspection (DI), and Expert Inspection (EI). In an experiment [18] based in the context of Virtual Environments (VEs) evaluation, twenty-nine individuals (10 end-users and 19 junior usability experts) participated during 1 hour each in the evaluation² of two VEs (a training VE and a 3D map). Quantitative results of the comparison show that:
-
the effectiveness of UT and DI is significantly better than the effectiveness of EI.
-
DI- and UT-based diagnoses lead to more problem diversity than EI and thus have a better problem coverage.
-
The identification impact of the whole set of usability problems is 60% for DI, 57% for UT, and only 36% for EI for both virtual environments.
-
Reliability of UT and DI is significantly better than reliability of EI.
In addition, a qualitative analysis shows that:
-
UT seems particularly efficient for the diagnosis of problems that require a particular state of interaction to be detectable.
-
DI supports the identification of problems directly observable, often related to learnability and basic usability.
Also, the data obtained from these experiments have been analyzed further with a different perspective, through collaboration with M. Schmettow (University of Twente) and C. Bach (IRIT). The purpose here is to use a new statistical model to detect and measure heterogeneity bias in data during evaluation processes, what will improve efficiency of the process. This was published in a national conference [36] ..
