Article Information


Anna Paukova (

Independent researcher, Moscow, Russia

Citation: Paukova A. (2019) Ecologies of Usability Testing: (Un)Taming the Chaos. Zhurnal sotsiologii i sotsialnoy antropologii [The Journal of Sociology and Social Anthropology], 22(6): 86–102.

Abstract. Basic standards of usability are articulated in ISO 9241–11 but the means to evaluate the usability metrics, such as effectiveness, efficiency, and satisfaction, are mostly up to researchers. In this paper, based on autoethnography notes and reflections of a UX-researcher, some practical realizations of usability testing are considered. The author employs the points of personal confusion about hybrid nature of usability testing method as means to reflect on how digital products’ features and contexts of their development and use become entangled within a usability testing research design. The aim is to discuss how user feedback is framed, captured and translated within the usability research setup, to document how certain norms and conventions of this research are practiced, what marks product’s success or failure and how signals produced by users are read and interpreted. The author proposes the terms ‘ecologies’ to describe the points of agreement between different stakeholders, which serve as means to tame the multifaceted nature of user experiences in the wild and as general guidelines on how to bring it into the lab. Three cases, stemming from practical experience, when such ecologies are manifested, are considered: material ecologies which have to do with technical equipment in the lab; intersubjective ecologies, describing the voices of different stakeholders; metrics ecologies which reflect the complicated nature of measurements in UX. The production of knowledge in usability testing might be seen as entanglement of multiple ecologies in temporary assemblages, allowing to achieve the goals and test the hypotheses in a given research. Because of their temporary nature, such ecologies must stay flexible and to be constantly reconsidered and redefined ad hoc, thus embracing the practical messiness of digital product use and development.

Keywords: usability testing, design research, autoethnography, mess in research, research ecologies


Certeau M. (1988) The practice of everyday life. Berkeley: University of California Press.

Clandinin, D.J., Connelly F.M. (1996) Teacher’s Professional Knowledge Landscapes. Educational Researcher, 25(3): 24–30.

Friess E. (2012) Do usability evaluators do what we think usability evaluators do? Communication Design Quarterly Review, 13(1): 9–13.

Graham S., Thrift N. (2007) Out of Order: Understanding Repair and Maintenance. Theory, culture & society, 24(3): 1–25.

Hertzum M. (2016) A usability test is not an interview. Interactions. 23(2): 82–84.

Kieffer S., Sangiorgi U.B., Vanderdonckt J. (2015) ECOVAL: A Framework for Increasing the Ecological Validity in Usability Testing. In: 48th Hawaii International Conference on System Sciences: 452–461.

Lallemand C., Koenig V. (2017) Lab Testing Beyond Usability: Challenges and Recommendations for Assessing User Experiences. Journal of Usability Studies, 12(3): 133–154.

Latour B. (2005) Reassembling the Social — An Introduction to Actor-Network-Theory. Oxford: Oxford University Press.

Law J. (2004) After Method: Mess in Social Science Research. London: Routledge.

Lindgaard G. (2006) Notions of thoroughness, efficiency, and validity: Are they valid in HCI practice? International Journal of Industrial Ergonomics, 36(12): 1069–1074.

Mol A. (2002) The body multiple: Ontology in medical practice. Durham: Duke University Press.

Nansen B., Wilken R. (2018) Techniques of the tactile body: A cultural phenomenology of toddlers and mobile touchscreens. Convergence: The International Journal of Research into New Media Technologies, 25(1): 60–76.

Nielsen J. & Landauer T.K. (1993) A mathematical model of the finding of usability problems. In: Proceedings of ACM INTERCHI'93 Conference (Amsterdam, The Netherlands, 24–29 April 1993): 206–213.

Nørgaard M. & Hornbæk, K. (2006) What Do Usability Evaluators Do in Practice? An Explorative Study of Think-Aloud Testing. ACM Symposium on Designing Interactive Systems (DIS 2006): 209–218.

Reeves S. (2019) How UX Practitioners Produce Findings in Usability Testing. ACM Transactions on Computer-Human Interaction (TOCHI), 26(1).

Star S.L., Ruhleder K. (1996) Steps toward an Ecology of Infrastructure: Design and Access for Large Information Spaces. Information Systems Research, 7(1): 111–134.

Trivedi M.C., Khanum M.A. (2012) Role of context in usability evaluations: A review. CoRR, abs/1204.2138. arXiv:1204.2138.

Wenger M.J., Spyridakis J.H. (1989) The relevance of reliability and validity to usability testing. IEEE Transactions on Professional Communication, 32(4): 265–271.

Woolgar S. (1990) Configuring the user: the case of usability trials. Sociological Review, 38(S1): 58–99.