Abstract
IBM was contracted to provide a new Air Defence Command
and Control (ADCC) system for the Royal Air Force. The IBM Human Factors
(HF) team was responsible for the design of the operations room, workstations
and the graphical user interfaces. Because the project was safety-related,
IBM had to produce a safety case. One aspect of the safety case was a demonstration
of the operational effectiveness of the new system.
This paper is an in-depth case study of the user testing
that was carried out to demonstrate the effectiveness of the system. Due
to time constraints the HF team had to observe five participants working
simultaneously. Further, to provide a realistic operational environment,
up to twenty-eight operators were required for each test. The total effort
for this activity was four person-years. The paper will detail the considerations,
challenges and lessons learned in the creation and execution of these multi-user
user tests.
Practitioner’s Take Away
- Although the client may specify or mandate a UCD approach (for example,
ISO 13407 [3] or Defence Standard 00-25 [2]) to be followed, they may
not realise how expensive it is to support this approach. The reduction
in whole life cycle costs of using a UCD approach are well documented,
but the costs of supporting the development from the customer’s
perspective are sometimes less explicit. Therefore, it is worthwhile discussing
this with the client early in the programme, and if necessary, updating
the risk register accordingly. - The summative user tests are an excellent mechanism for collecting data
and evidence for a safety case. However, to ensure that appropriate safety-related
data can be collected on a safety-related system the participants must
be placed in an appropriate working environment under a realistic amount
of workload. - Do not rely solely on a summative user test to produce all the necessary
operational safety-related data for a safety case. A good design process
that includes activities such as formative user tests and heuristic evaluations,
as well as the demonstration of effective training materials can also
produce an enormous amount of safety-related data [4]. - The production of the training materials must be aligned with the user
tests. The Training team must be notified as early as possible on the
required roles and tasks for each user test. This will ensure that the
appropriate training materials are available at the right time. Further,
Train The Trainer materials are not usually appropriate for a user test.
They are often lacking in context and standard operating procedures as
these are usually added later by the client’s Trainers once they
have been trained. On this project, it took the HF team 60 days to convert
the materials into the correct format. Therefore, design the training
materials so they can be used effectively for both purposes (i.e. TTT
and user tests). - Limit the number of observers to only those that are absolutely necessary.
When these tests were carried out, the project was nearing completion
and everybody wanted to see the system in action. This was a cause of
concern for the HF team as the more people that are around the more chance
of distraction and skewing of results, albeit accidentally. Therefore,
create and distribute observation etiquette instructions to everybody
entering the user test area. Even attach copies of the instructions to
the test room entrance, in the canteen and in congregation areas.