POKÉMON GO
USABILITY STUDY
Conduct a usability test for Pokémon GO to illuminate usability issues with an emphasis on key dimensions including learnability, discoverability, and navigation.
MY ROLE
UX Researcher
THE TEAM
SJSU Human Factors & Ergonomics students
TOOLS & METHODS
User research, in-person usability testing, hidden camera, script development, insight-generation
BACKGROUND
But first… why a usability test?
Our motivations behind this usability test were to evaluate Pokémon GO’s user interface in manner that aligns with Niantic’s mission statement of “inspiring movement, exploration, and face-to-face social interaction.”
In addition, we also aimed to find opportunities to increase the user's level of investment in the game, particularly for beginning players.
To this end, we also wanted to validate whether game features are navigable, discoverable, appealing, and comprehensible for all users, even as they are in the early stages of learning to play.
Why a Non-verbal Protocol?
We felt strongly about incorporating a non-verbal protocol (as opposed to the typical think aloud protocol) as we feel that users are more likely to interact with a product in their most natural setting when they use it as they normally would on their own. Like this we were able to see participants react more organically and minimizing the sterile lab environment as much as we can, hopefully leading to more true reactions.
How participants feel with verbal protocols
How participants feel with non-verbal protocols
Usability Dimensions of Interest
The main dimensions of interest in this study were as follows:
Learnability
Discoverability
Navigation & Flow
Investment in Game
Entertainment & Satisfaction
PROCESS
Participants
We recruited Beginner players and Experienced players in order to a range of experience and be able to evaluate the game across different user types as both groups have differing experiences.
Beginner players – these participants were people who had little to no experience playing Pokémon GO and had little background information on the app (note: both participants had some knowledge from pop culture).
Experienced players – These participants were relatively active current users of the game and
knew the ins and outs regarding the majority of the use cases for the app.
Method
Member Roles
Two Moderators (one being me!) : Guided participants through the study, introduced tasks, and asked relevant follow-up questions to help gather as much data as possible.
Two independent data-loggers: Coded performance, behavioral, and subjective metrics as the test was completed.
One AV support / Operations lead: Monitored the video recordings and audio to ensure that all the data was accurately captured for later review + dealt with participants checking in + compensation
Testing Room Layout
Testing Room:
Included a hovering-phone stand to record the audio and video of the participant
A phone was provided to participants in the “beginner” group while participants in the “experienced” groups played with their own phones to have access to their accounts.
All phones were ensured to have been equipped with screen recording software.
The top-down view from the hovering phone-stand was directly live streamed via ZOOM which the data loggers used to take note of the steps that the participants were taking when performing tasks.
Moderating the Usability Test
The moderators sat adjacent to the participant, close enough to speak with them directly while also being able to view their interactions.
We were very familiar with the Pokémon Go app so they could easily follow along and communicate with the participants; this was especially crucial when it came to the participants in the experienced group.
Our data loggers sat in the testing room so they could easily view and track the participant completing tasks using the livestream from the laptop, and monitor their performance and behaviors by looking at the participants directly.
A laptop also sat across from the participant which recorded the participant’s body language and facial expressions. This footage was recorded so it could be reviewed at a later time.
Scripts
To execute our study design, we translated our study objectives and tasks into two unique testing scripts that guided our team and the participant through the usability test, depending on the participant’s experience level.
Each script was printed out for our usability test moderator to reference during the study.
Measures
In order to determine success and failure criteria for the tasks in the usability test, we defined a standard set of usability metrics that we used during usability test data collection
Performance-Based (Objective) Metrics: primarily used to measure the usability dimensions of
navigation, flow, learnability, and discoverability.
Task success
Task time
Knowledge probes during or post-task
Behavioral Metrics & Markers: used across all goals and tasks, as they provided insight into all
dimensions of interest, from navigation and flow to satisfaction and entertainment value.
Expressions of frustration
Expressions of confusion
Expressions of delight or other positive emotion
Subjective Metrics: primarily used to measure the usability dimensions of satisfaction and
entertainment.
Rating of how intuitive the game is
Rating of task difficulty and learnability
Satisfaction
User retention
System Usability Scale (SUS)
Findings
We uncovered a number of different findings but for the sake of brevity, I'll only illustrate a few below:
Beginner Players Findings per Scenario
Experienced Players Findings per Scenario
System Usability Scores
The average SUS score is 68 out of 100 possible points, and any score 68 or above can
be considered a usable system, or a system with favorable usability. In this case, all four
participants rated Pokémon GO below 68, and each score converted to a letter grade was an F.
Based on these scores, Pokémon GO fails considerably at providing users with good system
usability and thus compromises user satisfaction, confidence using the product, and likelihood of
using the product again or recommending it to a friend. The results of the SUS validate the observations we had during our usability test: that Pokémon GO struggles significantly in terms of Learnability and Discoverability
Conclusion
With these overarching study objectives in mind, we evaluated Pokémon GO against our
previously defined usability goals and dimensions of interest. We then assigned Pokémon
GO a letter grade based on how well they met these guidelines, per our usability test findings.
Based on the assigned grades above, we believe that the biggest issues with this game
are in learnability, discoverability, and navigation. This is because they are the biggest barriers
to users finding satisfaction and entertainment value in the game, which then impacts user
investment and retention. Our list of recommendations for addressing these issues, along with
their estimated priority, can be found below:
