Since the introduction of application stores for mobile devices there has been an increasing interest to use this distribution platform to collect user feedback. Mobile application stores can make research prototypes widely available and enable to conduct user studies “in the wild” with participants from all over the world. Using apps as an apparatus goes beyond just distributing research prototypes. Consider apps as a tool for research means distributing specifically designed prototypes in order to extend our understanding of mobile HCI. In this tutorial we will provide an overview about recent research in this domain. It will be shown that stringent tasks and users’ motivation are crucial aspects. We will discuss how to design app-based experiments, what kind of users one can expect, and how to avoid ethical and legal issues.
Mobile application stores such as Apple's App Store and Google's Android Market revolutionized the distribution of applications for mobile devices. This distribution channel lowered the gateway hurdle dramatically and opened the market for small companies and engaged hobbyists. Mobile application stores – for the first time – enable virtually any developer to reach hundred thousands of mobile users. Recently researchers also discovered this opportunity and began to publish prototypes via mobile application stores.
It has been argued that the “easy access to such a potentially wide audience could radically alter the nature of many UbiComp trials”. In the tradition of UbiComp research first attempts to distribute prototypes via mobile application stores focus on the evaluation of prototypes. Proof-of- concept prototypes are developed and the large number of users is used to demonstrate the successfulness of the respective application. Feedback is mainly gathered to understand the nature of the respective prototype. In the tradition of psychology and social sciences, Human Factors and Human-Computer Interaction research, in contrast, focus on understanding the human. Commonly, controlled experiments, quasi-experiments and observations are used to derive general findings. As in psychology, prototypes are often just the apparatus to investigate a research question.
This tutorial will be based on our group’s experience in conducting user studies by publishing apps in mobile application stores. So far we conducted more than a dozen studies using different apps with varying success. Some studies never attracted a sufficient number of participants and others provided dubious results due to a flawed design. We, however, also conducted studies that attracted a hundred thousand participants that each contributed meaningful data.
The tutorial is structured as a one hour course that covers the important aspects of user studies in the app store era.
The tutorial will discuss the difference between sharing a research prototype with the public and conducting user studies where the app is just an apparatus. We will provide a systematic overview about the emerging research in this field and discuss why the studies have been successful. We will also present failures we and others experienced that usually remains unpublished but provides important insights for others.
We will discuss what kind of audience one can expect when conducting user studies in the app store and it will be shown which characteristics can easily be collected from participants. Based on case studies we will demonstrate typical results that can be collected.
We will explain why stringent tasks but also the users’ motivation are the most crucial aspects for the success of the study. We will discuss the importance of finding a balance between collecting valid data and developing a successful app. With a focus on using games to investigate research questions we will provide hints how researcher without a background in game design (like us) can increase players’ motivation. Advantages and disadvantages of repeated-measures and independent-measures that are specific to this kind of study will be outlined.
Attendees will learn about ethical and legal issues they might face when collecting data. We will show different approaches to inform users that they are about to participate in a study and compare their success.
Niels Henze is an associate researcher in the Human- Computer Interaction group at the University of Stuttgart. Beforehand he worked at the OFFIS Institute for Information Technology and later worked as a researcher and doctoral student in the Media Informatics and Multimedia Systems Group at the University of Oldenburg, Germany. His research interest is generally focused on mobile human- computer interaction. Particularly, he is interested in large-scale studies using mobile application stores as a research tool, interlinking physical objects and digital information, and multimodal interfaces. Niels published over 20 peer-reviewed papers in scientific journals and conferences and received best paper awards from CHI and MobileHCI. He organized several scientific workshops, served as a guest editor for the International Journal on Mobile Human Computer Interaction, and is a reviewer for a number of conferences and journals. Niels lectures human- computer interaction in practical and theoretical courses for several years. He developed and supervised the development of mobile applications to conduct large-scale user studies that have been installed more than one million times in total.
Martin Pielot is working with the OFFIS Institute for Information Technology in Oldenburg, Germany. He is a member of the Intelligent User Interfaces group, which is part of the R&D division Health. His research focuses on non-visual interaction with mobile and ubiquitous devices, including tactile, auditory, touch, and ambient UIs, as well as the methods needed to evaluate them. As a PhD candidate he investigates ways of conveying spatial information in mobile usage scenarios via vibro- tactile displays. In particular, he studies non-visual UIs for navigation systems published through the Android Market since 2010. Further, he works on tools to measure and evaluate these applications “in the large” from a usability and accessibility point of view.