In the context of research, experiments are designed to confirm or refute a hypothesis formulated by the investigators. The setup of the experiment is often tightly controlled to focus on a small number of independent variables that are key to the hypothesis. For research in human-computer interaction (HCI) and similar areas, such a setup typically consists of a set of tasks to be performed by users in a usability laboratory. While this enables reasonably controlled conditions, it often involves an environment that varies to some extent from the context in which participants would normally perform such tasks. For conventional computer use with desktops or laptops, this difference may be marginal and can sometimes be minimized by modifications in the experimental setup. Due to practical limitations, the number of participants in such experiments is also frequently quite low, and subjects are often drawn from easily accessible participant pools, such as students at a university, that may not reflect the actual user population.

These difficulties are magnified for investigations involving mobile devices, since their usage contexts are typically much wider and impractical to replicate in a lab setting. For example, a map-based navigation app with location support is intended for use while walking around a city block. To achieve a believable context, experiments involving the use of this app would require a virtual reality environment that immerses the user in a simulated city block, a feed of fake but realistic data, a way for users to walk and change directions, the incorporation of other traffic participants, and several other factors.

In this brief article, which appeared in IEEE Computer’s “Invisible Computing” column, the authors describe an alternative approach to conducting experiments involving mobile devices. To make up for the lack of control, they propose large-scale studies of tasks by distributing mobile apps or widgets among participants. These apps function as data collection devices, and are set up to allow for the analysis of collected data to examine independent variables. In one example, a text entry game (Type It! for Android devices) incorporates variations of touch position, key labels, and visualizations of touched positions. The large number of participants using different devices in different realistic contexts (in their example, the game collected data from 85,000 installations) led to a very high external validity. Such an approach clearly also has disadvantages, such as the high effort to develop suitable apps that motivate participants to engage in relevant activities, the self-selection of participants, and considerations of privacy and security.

For me, the most useful part of the article is its “Ten Steps To Conduct a Large-Scale Study” list. As with many such lists, the individual entries are fairly self-evident, but the compiled list is very handy to make sure that all critical aspects have been considered. Especially if the experiment includes the development of a game-based mobile app that uses fun as the incentive mechanism, the main effort may lie in the actual development of the app, and it could be easy to neglect such issues as obtaining informed consent from participants or implementing a well-planned systematic data collection method.