I suppose I should have guessed that this paper was about evaluating a tool for evaluating usability, rather than a usability test itself, but I became more and more confused as I read on. “Where is this heading?” I asked myself.

When I finally looked at the title again, I discovered my mistake. This paper is about a system the authors used to create a proxy mobile environment. As they point out, it is difficult to collect mobile phone or device data in the field, but working only in a laboratory restricts the amount of data available and may lead to poor interpretations.

So instead of letting their subjects loose in the great outdoors, they set up a trail in a building. The subjects, after being attached to galvanic skin response and heart rate variability sensors, were given a mobile wayfinding device called the NaviTerier, designed for visually impaired people, and were sent off on a journey through the building. All 22 participants were blind.

Also included were subjective post-test evaluations and cameras attached to backpacks carried by the participants. Additionally, observers recorded the participants’ actions for possible later analysis.

Along part of the route, the researchers created a fake construction area with buckets and plastic wrap scattered on the floor. This was designed to create stress for the participants, although the researchers later discovered that random social encounters and distant noises created stress as well.

The researchers pulled all of this collected data together in a software tool called the integrated interactive information visualization environment (IVE). The tool organizes the various types of data collected (galvanic skin response, video, participant comments, and so on) into a dashboard-like synchronized timeline. The program also allows drill-downs into individual data collections and points.

The entire system is very impressive and could be adapted by other researchers for similar experiments. But did the authors come up with any usability data?

They did, and they describe these results in a paragraph toward the beginning of the paper. They found that participants forgot certain features of their route, in particular, doors and the shape of the corridor, while correctly reporting the rest of the route. They theorize that this forgetfulness is caused by “a stress reaction, which negatively influences the cognitive processes, especially attention and memory performance in its acquisition phase [1].”