An official website of the United States government
Here’s how you know
The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before
sharing sensitive information, make sure you’re on a federal
government site.
The site is secure.
The https:// ensures that you are connecting to
the official website and that any information you provide is
encrypted and transmitted securely.
The following indicators can help determine if a usability test will produce useful results.
This is not an exhaustive list, but it should be helpful in planning and assessing usability tests. Additions are welcome!
Study design
Indicator
Good sign
Bad sign
Study purpose
Researcher clearly articulates the purpose of the usability test (for example, as a specific question to answer, an area of inquiry, etc.).
Researcher does not specify a purpose for the test, or the purpose specified is very broad, like “testing the app” or “finding problems.”
Number of participants
Tests include a sufficient number of participants necessary to see patterns, and there’s only one participant per session.
Study includes only one participant, or includes more that two participants testing simultaneously (like a focus group).
Sampling bias
Participants are a diverse set of people who actually use the application (or do the task the application is supposed to support).
Participants are experts (vs. average users), friends, family, or close colleagues of the product team.
Moderator style
Indicator
Good sign
Bad sign
Question style
Researcher asks follow up questions that are open-ended and non-leading.
Questions are leading or subtly suggest potential solutions or hypotheses.
Task variability
Researcher asks participants to complete similar tasks related to the study’s purpose.
Researcher asks participants to complete different tasks, or tasks unrelated to the study’s purpose.
Framing
Researcher asks participants to complete tasks with the product or service that align with their work-related goals.
Researcher asks participants to complete tasks unrelated to their work-related goals. For example, asking a participant how they might send a fax when their job doesn’t call for that.
Priming
Moderator asks participants to complete tasks without indicating how that might be done. For example, “How would you view the status of your application?”
Researcher guides participants in completing tasks. For example, “Which of the links in the header would you click to login?”
Team participation
Indicator
Good sign
Bad sign
Number of teammates
The team designates a single moderator for the test, and at least one member of the product team observes the usability test.
A single person from the product team participates in and leads the test.
Observer role
Observers do not moderate. They are generally quiet, and ask open-ended questions after the test has concluded.
Observers interrupt the participant, or attempt to sell or explain the product. Observers debate the participant’s actions in their presence.
Sensemaking
Indicator
Good sign
Bad sign
Notetaking
Tests are recorded or notes are taken for sharing with absent stakeholders.
Tests are not recorded, or test results are not documented.
Synthesis
Moderator debriefs with teammates immediately after each interview. Researcher looks for patterns across multiple participants and surfaces problems that affected several people.
Moderator reports the most memorable problems without conducting affinity mapping or some other analysis technique.
Presentation of findings
Researcher reports findings to team and stakeholders in an easy to follow, well prioritized way.
Researcher presents the team a “basket of issues” or an unprioritized laundry list of potential changes.
Incorporation of findings
Product team translates findings into future user stories or product refinements.
Researcher reports do not affect the product backlog or ongoing development work.
User Experience Design Guide