When I read this paper, it reminded me of a conference poster I’d seen many years ago. The poster authors had come up with a method for spell-checking using Google search. They reasoned that, if you searched for a word and Google didn’t provide the correct spelling immediately (“Did you mean...?”), and if you found that, in the majority of hits, it was spelled one way rather than another, then the majority spelling was probably right.

Walking up to the poster and reading the headline, I scoffed. After reading the rest of the poster and talking to the authors, however, I was completely won over. In fact, I use the technique myself all the time.

The authors of this paper did something similar in their experiment. Their goal was to see if, from online reviews, an automatic classifier could figure out which products were more usable or offered better user experiences.

This would be especially useful for finding out how the user experience might change over time. They point out that usability test participants interact with a product for at most an hour, which isn’t much time to get a feel for long-term use. So they looked at Epinions.com reviews for long-term usability and user experience (UUX) issues; Epinions gave them end-user rather than professional points of view.

The authors first had experts isolate the right dimensions (hedonic, learnability, errors and effectiveness, satisfaction, and detailed usability) and words or phrases to look for, and then enlisted graduate students to find these words and phrases in 520 software reviews and 2972 video game reviews. The final phase was to see if machine learning software could reliably classify sentences for UUX content.

In terms of coming up with a useful classification system, the results were mixed. The automatic classifier could pick out many of the words associated with the chosen dimensions, but not all, partly because reviewers use metaphor and irony.

For example, the authors quoted this sentence from a review: “There is nothing that can ruin a good [role-playing game (RPG) like] a partner that is supposed to be helping you but instead makes you feel like [you’re] babysitting a 5-year-old with mental problems” (page 2095). It’s easy for a human to guess that the reviewer didn’t like the game, but what would an automated classifier make of the word “babysitting”? Is that a negative or a positive?

The authors also mention a few caveats that affect the entire experiment:

• Do users typical of the user base write the online reviews? There is typically no way to identify the reviewers’ demographics.
• Some reviews may be fake (although there are algorithms that can flag fake reviews [1]).
• Users with “average” satisfaction don’t bother to write reviews. Only people with very poor or very good experiences are likely to write, meaning that the average user is probably underrepresented among reviewers.

However, although automated classifiers cannot reliably come up with UUX measurements--as the authors point out, no one writes: “The number of mouse clicks to navigate from the start screen to the functionality I want is 7, and this is annoying” (page 2096)--it is possible to extract how users express their feelings and experiences related to UUX.

Perhaps what makes this experiment interesting isn’t whether automated classifiers can do a good job (and the classifiers may get better over time as they learn more phrases and exceptions), but whether UUX or marketing practitioners can steal the idea easily. To get a sense of the universe of kudos or complaints for a particular product, I can see copying a few pages of Amazon reviews into a spreadsheet and then using filters to isolate various phrases (see page 2097 for the list of word stems they used). Sometimes the simple answer is just perfect.