User testing should be a standard part of your user experience practice. Getting user feedback on new or existing products will help ensure that you meet user needs, increasing the success of your digital product and reducing the likelihood of costly mistakes. This resource provides an overview of the user testing process and practical tips on how to implement it within your organization.
Phase 1: Pre-Planning
Before conducting user testing, it’s crucial to engage in some pre-planning activities to ensure that the testing process runs smoothly and that you achieve your desired outcomes. This phase involves several key activities:
Identify Test Goals
One of the first things you should do is identify your test goals. Document your learning objectives and describe the digital websites, mobile apps, or channels that will be a part of the usability test. Your test goals should be specific and measurable. By setting clear goals, you can ensure that your user testing activities align with your business objectives. Some common goals for user testing might include improving the user experience, identifying usability issues, and understanding user preferences and behavior.
Prepare Product or Design for Testing
Once you’ve identified your test goals, you’ll need to prepare your product or design for testing. You’ll need to determine how to represent your designs, whether it’s through paper sketches, whiteboards, wireframes, or a prototype. If you are testing on a live site, you’ll need to determine which sample pages or screens will be a part of your test. Preparing your product or design is an essential step in user testing, as it helps you to ensure that you’re testing the right elements and that you’re not wasting participants’ (or your teams) time.
Identify Ideal Participants
Identifying the ideal participants for your user testing is critical to its success. You’ll need to determine which user group you want to represent and identify which of your personas or customer segments are in scope. Once you’ve identified your target users, develop screening questions to ensure that you’re targeting the right participants. When creating your screening questions, be sure to ask questions that are relevant to your test goals and are not leading. It’s also important to determine how many participants you need based on how statistically relevant your user testing results need to be.
Once you’ve identified your target participants, you’ll need to recruit them. Determine where you’ll get your participants and offer an incentive (or a strong reason) to participate. It’s common to offer some sort of incentive, such as cash or a gift card, for participants’ time. When recruiting participants, be sure to communicate clearly about what you’re testing and why, how long the test will take, and what the incentive is for participating. You should also ensure that you’re recruiting a diverse group of participants that are representative of your target user group. By following these pre-planning activities, you can set yourself up for success when conducting user testing. Once you’ve completed these activities, you’ll be ready to move on to the next phase of user testing: conducting the test.
Phase 2: Test Design
After completing the pre-planning phase, it’s time to move on to test design. This phase involves crafting the details of your user testing process. Here are some activities to help you design your test:
Write Test Plan
The first thing you should do is develop a test plan that lists all the details of the usability study you’re preparing. Work with stakeholders and design team members to refine the test plan. By creating a comprehensive test plan, you can ensure that all necessary details are accounted for and that your team is aligned on the testing process.
Learn more about creating a test plan here
Once you have your test plan, you can begin to document the profile questions you want to ask your target participants. These questions are used for categorizing results and understanding participant characteristics. You should also document additional qualitative questions you want to ask participants. Consider your learning objectives when crafting these questions. It’s important to avoid leading questions and to ask questions that are specific and relevant to the scenarios and tasks being tested.
Craft Scenarios and Tasks
Next, create specific scenarios that address your learning objectives. Scenarios provide context for the user about why they might be arriving at the experience being tested. For each scenario, list the tasks that you would like the user to attempt to complete. Be sure to write clear, specific instructions for each task. Crafting scenarios and tasks can help you to better understand how participants interact with your product or design and can provide valuable insights into usability issues.
Create a Measurement Plan
Document what data you want to capture during the usability test. For example, if understanding “task completion time” or “number of clicks” is a goal of the test, then this is a data point you’ll need to capture. Develop a plan for capturing each data point, whether it be through a question or by having the facilitator and notetakers observe and record the data. Creating a measurement plan can help you to ensure that you capture all necessary data points and can help you to more easily analyze the data post-testing.
Create a Discussion Guide
Develop a discussion guide that outlines how each individual session should be structured. The discussion guide should include an introduction to the study, a description of the scenarios and tasks, and questions to ask participants after completing each task. By developing a discussion guide, you can ensure that each testing session is structured in the same way and that you’re able to gather consistent data across all participants. Work with members of the research team to revise the discussion guide as needed. By following these test design activities, you can create a structured and effective user testing process that will help you to achieve your desired outcomes.
Learn more about creating an effective discussion guide here
Phase 3: User Testing
Now it’s time to actually conduct your user test. This is where the real work happens! Here’s how to make the most of this phase:
Conduct a Pilot Test
Before you begin the actual study, it’s a good idea to conduct a pilot test. This will help you avoid issues during the real test and ensure that everything is running smoothly. Here’s what you need to do:
- Test run your session 1-2 days before the actual study so that you can avoid issues during the real test. Run through all questions, tasks, and scenarios. Make sure that all materials are present and equipment is running.
- After the pilot test, make adjustments to the test and correct any technical issues. You’ll also want to review the data that was collected during the pilot test to see if any changes need to be made to the test plan.
- Make sure that all materials, consents, and documentation have been prepared. This includes consent forms, task sheets, and any other materials that you’ll need for the actual study.
Conduct Usability Tests
Once you’ve conducted your pilot test, it’s time to move on to the actual study. Here’s what you need to do:
- Conduct your user test with each participant. To prevent bias or skewed results, follow the same script and process each time. Counterbalance task order from participant to participant.
- Immediately after the test, allow the research team to document any notes about the participant tests. You’ll want to do this while it is fresh in their head and before they hear thoughts from others.
- If there is time available, get the research team together to debrief. The team will be eager to share their observations and thoughts with each other. New insights might develop from these conversations. Capture them. They will be useful in the next phase.
During this phase, it’s important to remain flexible and open to feedback. If something isn’t working, be willing to make adjustments to the test plan. Remember that the goal of user testing is to learn as much as possible from your participants. By staying open and flexible, you’ll be able to get the most out of this phase.
Phase 4: Analysis & Actions
The final step of the user testing process is to analyze the results and develop actionable findings. This stage is critical in ensuring that the product is optimized for user experience.
The first step in analyzing user testing results is to document observations. Each member of the research team should provide their final observations about each user participant. These observations should be compiled into one place and organized for easier evaluation. Additionally, a quick report with any quantitative data points or metrics should be developed. This report will help the team understand the data collected and make data-driven decisions.
Once the observations have been documented, the team should review and discuss them. This includes analyzing metrics such as completion rates, task success rates, and time on task. The team should identify themes and insights across all users and tests. Where possible, isolate results by customer segments/attributes to get deeper insights. This process helps the team understand how users are interacting with the product and identify any issues or areas for improvement.
Identify Issues and Determine Severity
As the team analyzes results, they should identify any issues and determine the severity of each usability issue encountered. The severity of an issue is determined by its impact on the user experience. High-severity issues may include anything that prevents users from completing tasks or using the product. Medium-severity issues may include anything that makes the product less efficient or less intuitive to use. Low-severity issues may include minor usability issues that do not significantly impact the user experience.
For key observations and issues, the team should brainstorm what actions should be taken next. In some cases, these may be improvements to the experience design. In other cases, this might involve collecting more research. The goal is to identify actionable recommendations that will improve the user experience.
Prepare Findings Document
The final step in the user testing process is to prepare a findings document. This document should summarize the results, issues, and recommendations. It should be clear and concise, making it easy for stakeholders to understand the key takeaways. The findings document will serve as a guide for the next steps in product development.
Here are answers to some frequently asked questions that I receive about user testing:
What is SUS and why should I integrate it into my user test?
SUS or System Usability Scale is a questionnaire that measures user perception of satisfaction and usability of a product or service. The SUS questionnaire consists of standardized questions about how easy-to-use a system is. The SUS number can be benchmarked against other companies and industries. Integrating SUS into your user test can help measure the effectiveness of your product and improve its overall usability.
How do I find and recruit participants for user testing?
To find participants for user testing, you can start by simply asking existing customers or recruiting from representative users if you have access to them. Another way is to post a request on your social media channels and target social media channels that your potential users are likely to be on. You can also outsource the recruiting process to a recruiter, who can screen, schedule, and remind participants about their test appointment, and handle additional administrative duties such as administering incentives for participants (i.e. gifts or money), and in some cases travel/parking expenses.
How many tasks should I include in my user test?
You should limit your user test to 6 to 12 tasks due to time constraints and users’ mental exhaustion. This varies based on the device being used (desktop, mobile, etc.).
How many participants are needed in a user test?
As a rule of thumb, test with at least five users. Always recruit at least one more participant than you need, just in case one person cancels. According to usability expert Jakob Nielsen, testing with five users will catch 85% of the usability issues. To catch the remaining 15% of issues, you can test with 15 users.
At what points in my project should I conduct user tests?
Testing should be done throughout every stage of the product’s design, and continue to be done after the product launches. Products do not have to be complete to be tested. Testing can be done on the entire product or on just a few select screens. Testing can be done on a live site or a test site. Products being tested can be in draft form such as paper sketches, wireframes, or HTML prototypes.
What is the standard structure for scenarios and tasks?
During the user test, you will have several tasks that you want to target to attempt to complete. Each task should follow a standard structure, including setting context, providing the user with a specific task, and following up by asking a question to gauge the user’s experience. For example, to learn how well a person was able to navigate to a type of product, you could have them start on the home page and say, “You are here to buy a fire alarm. Where would you go to do that?”
How do I prepare for things that might go wrong?
Prepare for things to go wrong (and something always does). Consider the following:
- Some participants will be a few minutes late. If they are, but you still want to use them, what are the lowest priority tasks or questions that you will cut out?
- The prototype software could stop working or have a bug. Try to have a backup – such as paper screenshots – if you think this is a possibility.
- In a remote study, some participants will have difficulty using the video conferencing tool. Know in advance how the screen looks to them, what they should do, and common things that can go wrong so you can guide them through the
Can I use internal stakeholders as users?
It is not recommended to use internal stakeholders as test participants. It is important to select users who represent your actual user base to ensure that the feedback received is relevant and accurate. While internal stakeholders may provide valuable insights into what your users might value, they are not actual users and may miss important insights. However, if business stakeholders want to be involved in the testing process, pilot tests can be a great opportunity to include them.
What is the difference between user testing and usability testing?
In this resource, we use the terms usability testing and user testing interchangeably because the process is similar and they are often done at the same time. However, the goals of these two types of testing are different. Usability testing aims to identify critical problems that prevent users from completing tasks or achieving their goals with the product. On the other hand, user testing aims to understand whether the user needs the product and if it provides value to them. By conducting both types of testing, you can gain a comprehensive understanding of how users interact with your product and how you can improve their experience
How do I sell the value of user testing to my leaders?
This is a question worthy of it’s own article!