Abstract
This paper provides a brief review of the Hawthorne effect, a discussion of how this effect relates to usability studies and help for practitioners in defending their studies against criticisms made on the basis of this effect.
Practitioner’s Take Away
- The Hawthorne effect can be (mis)used as a basis on which to criticize the validity of human-centered studies, including usability studies. Therefore, it is important that practitioners are able to defend themselves against such criticism. A wide variety of defenses are possible; depending on which interpretation of the Hawthorne effect is adopted. To make an informed decision as to which interpretation to adopt, practitioners should be aware of the whole story regarding this effect.
- A precursor to any defense should be pointing out that there are many significant differences between the studies carried out at the Hawthorne works and typical usability studies. Therefore, care must be taken when relating any interpretation of this effect to our discipline.
- Most criticisms will be founded on the interpretation of the effect proposed by Mayo (1933). However, despite its popularity, this interpretation has been largely debunked over the last few decades. At worst, it can be considered as nothing more than a popular myth that has no place in any serious research thinking. At best, it can be considered as a controversial idea that has highly questionable relatability to our discipline. Therefore, a defense against this interpretation is not likely to be required once all the stakeholders in a study understand this fact. However, should such a defense be required, the study should maximize the use of (blind) controls and gather evidence of the causation mechanisms that resulted in any significant findings (benefits). This is probably best achieved through the use of qualitative techniques such as verbal protocols and pre- and post-test semi-structured interviewing.
- If the interpretation of the Hawthorne effect proposed by Parsons (1974) is adopted, then it should be insured that extrinsic performance feedback to participants in a study is eliminated, or minimized as far as is reasonably possible.
- Since some defenses affect the study design and its execution, it is clearly important that the study team agree with the client what position will be taken in relation to the Hawthorne effect in advance of the study. This position should be then be published in an appendix to the study report.