Remote Testing empowers researchers by providing reports with statistically valid quantitative data, as well as immediacy to usability issues with video, qualitative, and behavioral data. When conducted well, it is a research ally that affords incomparable and actionable insights into user behavior and attitudes.
However, it still remains a relatively new method for many UX Pros. Let´s have a look at twelve key tips that will help you make the most out of the Remote Testing methodology, be efficient, avoid pitfalls and gather as much quality data as possible. Here we go!
1. Define the study goals well
Determine tactical and strategic goals of the study before diving in. When researchers do not set up their study goals and requirements at the onset, they tend to shuffle from one Remote Unmoderated Usability Testing study type to another. Vague and unclear goals lead to study designs that are not well thought out and consequently lead to less than actionable data. While well-defined goals lead to a tighter study design. The researcher should have a hypothesis he/she would like to test. Each question should have a clear purpose and the researcher should have a vision of what the final report will look like.
Example: Instead of designing and creating a study well in advance to insert it into your remote usability testing solution, some researchers spent hours revising each task and re-thinking the purpose of each task.
Plan to spend several hours finalizing the goals and refining the tasks for your projects. You will need time to discuss each task, its goal and what it will help you accomplish well before you put anything down on paper.
It might seem a lot of work at the onset, but it reduces the overall time involved to execute a study. A well-defined questionnaire makes analysis easy, particularly when the researcher already has a vision of what the report will look like. Questionnaires that aren’t designed well lead to unnecessary questions for the participants, inflating the overall time required to take a study, and contributing to participant fatigue and lower quality data.
2. Select the right research approach
Some remote testing tools offer a variety of research approaches including live sites, prototypes, mobile sites and mobile apps. These research approaches support a wide range of study goals. Your testing software should enable you to run both Customer Experience studies (e.g., VOC – voice of the customer, website feedback surveys) and User Experience studies (remote usability testing, card sorting, tree testing, screenshot click testing and timeout testing). Knowing the goal of the study helps researchers choose the right study type.
If the goal is to restructure the site’s information architecture, then card sorting and tree testing along with some survey questions might be the best bet. If you would like to compare your site’s performance with competitors, then task-based usability testing is the appropriate study type to select.Example:
Researchers testing live prototypes with limited functionality often wrongly choose task-based usability testing when the study goal is to test the site nomenclature, particularly the navigation menus. This results in participants being directed to the live prototype but not having the ability to deeply interact with it. Sending users to achieve a task on the prototype (e.g. use navigation menu to find some information) will provide researchers task success rates and the amount of time on task, but yet the overall labeling of the navigation menus may still remain unaddressed.
Instead, the researcher should select screenshot click testing and timeout testing question types when testing navigation labels on prototypes to directly understand how well users understand the site nomenclature.
Selecting the right research approach is essential to obtaining data that will meet their research objectives.
3. Choose between behavior tracking options:
The goal is to gather behavioral data such as clickstreams, heatmaps, screen recordings, and user videos.
1. Browser add-on:
Choose a tracking technology which only collects user data during an active study.
Some participants may hesitate to install the add-on. To overcome this hesitation and increase the study completion rate, set participant expectations up front and budget for an incentive to compensate both for installing the add-on and completing the study.
The benefit of tracking with JS code method is that users will not have to download any add-on, typically resulting in an increase in completion rate by 10-15%.
No method is better than the other, and there are trade-offs with both. With JS code, researchers will have a higher completion rate. But researchers rarely have access to the website’s code. Browser add-on is simple to execute, but it may reduce the completion rate.
4. Choose the right recruitment method
There are 3 methods to recruit participants to complete a study:Panel: A panel provider is a third party vendor that specializes in screening and recruitment of participants.
When the participant profile is heterogeneous or a general online population and the desired number of completes is over 100, then recruitment through a panel vendor may be the best method.
Quality panel providers realistically estimate about the recruitment feasibility and offer optimal rates depending on the type of study. Researchers benefit from this as it helps them set realistic timelines on project completion.
Customer list: If the researcher has a list of customers who have opted in to participate in research or receive communications, these customers can be recruited by email invitations. A link to the study would be included within the email and ideally clear communication about who is conducting the study, the purpose of the study, the study length, and how customers will benefit (e.g., a better web experience, an incentive) and if there is an add-on required. Any device or system requirements should also be clearly conveyed.
When the participant profile is too homogenous or narrowly defined (e.g., users of a company’s internal product) then using the email list to send participant invitations is best. With more extensive participant criteria, there is a lower incidence rate of those types of users in the general population; therefore panel recruitment may be more difficult.
Intercept from a live website: A percentage of visitors from your website will receive an intercept message and be invited to take part of the study.
Intercepting visitors, like other options, has its pros and cons. The biggest benefit is that researchers can invite actual visitors. Moreover, there are substantial savings since there are no recruitment costs and incentives may not be necessary. The biggest drawback, however, is to be mindful of disturbing the visitor experience. Intercept studies are best kept simple and short.
5. Set clear expectations
Clearly communicating to participants at the invitation level and at the beginning of the study helps properly set their expectations and prevents any surprises and frustration on their part.
Having a good participant experience will ensure better completion rates and higher quality data.
Participants should be informed of any device requirements, the time commitment to take the study., if a browser add-on installation is required, the number of tasks in the study, and also notified of their progress through the study.
For example, let participants know what task they’re on (e.g., task 4 of 5) or that they have completed the tasks and have a few final wrap-up questions before completing the study.
6. Create clear tasks
Directing participants to your website or application without much instruction is like leading them into the unknown. It is best that you send them on short missions with clear instructions on what they are required to accomplish.Example:
For a recent study, we were evaluating the “search for a garment and add to cart” behavior on an e-commerce website. On the face of it, it seemed like one fluid task.
However, selecting a garment involves several little actions. One has to look at the right garment type, color, price, fit, etc. So it actually can be broken down into several mini tasks. Breaking a larger task into smaller ones allows participants to focus carefully on each step of the process and provide specific feedback about each part of the larger experience.
Had we grouped the behavior of “filter & sort” and “add to cart” into one comprehensive task, we would not have been guaranteed that participants would indeed indulge in those behaviors in one task. Also participants may not remember certain aspects of a large comprehensive task to provide good feedback.
By breaking a large task down into smaller concrete ones, we not only ensured that they did each of the tasks well but also got accurate feedback on the follow-up questions.
Go through and test your tasks carefully, particularly when running a competitive study. The tasks should be realistic and achievable on all sites to ensure an apples-to-apples comparison. Asking participants to perform a task that is achievable on one site and not the other is setting the latter site up for failure. The parameters of the task should also be the same. For example, if participants are asked to find a phone number for online banking on two banking sites, but asked to find the address on the third site because the phone number is not available, the parameters are slightly different and not directly comparable.
7. Define the tasks and determine task success
The unmoderated methodology requires that researchers predetermine success and failure for each task through task validation. There are three ways to do this.
1) Validation Question: If the goal of the task is to test findability of information, such as total cost, discount percentage etc., then it is best to confirm by a validation question. When validating by question, participants are asked to find or remember a relevant piece of information from the website that they would only know if they had finished successfully.
2) Validation by URL: When tracking behavior and clicks on a site, Validation by URL may be the right option. It helps to accurately establish findability of pages on a website. When validation by URL is enabled, success is based on whether participants reach the designated URL(s).
3) Self-reported Validation: A third method is self-reported validation where users indicate whether they felt they succeeded in completing the task. This method of validation is more subjective, but is useful when participants are asked to gather information, review content etc. For example, did a pharmaceutical site adequately explain side effect information of a drug? Did participants feel like they were able to gather enough product information to make an informed purchase decisions?
We know from experience that users are likely to take the blame for not finding information on a site. Selecting the right validation method ensures that researchers do not have to spend time recalculating effectiveness (task success rates) ratios post study.
8. Keep participants engaged
The key to keep participants engaged is to be succinct and to keep the study as fun and interesting as possible. In the interest of sounding professional, studies can become sterile and boring in tone. Task descriptions can also become lengthy and verbose. Users may complete the study but the experience may not be an interesting one.
If the task description and study expectation are boring and long-winded, then users are more likely to abandon the task or study before completing it.
Ensure task descriptions and follow-up questions are concise and easy-to-read. Participants often times do not read everything carefully. Highlighting key parts of the task description or the question through the use of bolding, a different color, underlining, or italics enables participants to quickly see and understand the question. Bullet points or numbering is also useful in conveying important details in an easy-to-read manner.
The average completion rate for a URUT project is about 75%. However, using fun engaging language and making tasks and questions easy to read and easy to understand will increase the completion rate by at least another 10%.
Example: One nice way to keep users engaged is to communicate directly with them during the study and to acknowledge their effort. Let users know that their feedback will help thousands of future users have better experience on the site.
Notify them periodically they are troopers for sticking around. Inform them of the progress and how much longer they have in the study. This indicates that you respect and value their time.
Acknowledging effort and appreciating feedback often keeps users engaged and encourages wholehearted participation in the research.
9. Create a good mix of quant & qual questions
Because the URUT methodology enables the capture of effectiveness (successful/unsuccessful at task completion) and efficiency (time on task) metrics automatically, researchers may rely too heavily on it without sufficient qualitative feedback. A usability report that is all graphs and statistics without a human element of users’ comments may be Spartan in its tone and tenor.
Conversely, researchers with ethnographic and in-lab research background, are more likely to rely on qualitative question types. While these questions are insightful, it can be cumbersome to analyze 100+ user feedback comments to determine a pattern. It also fails to utilize the full spectrum of the URUT method.
Ensuring that you ask the right number of questions with a mix of quantitative and qualitative question-types will help you create a more compelling story in the report.
Avoid having too many matrix or rating scale questions in a row. Participants may not read each question / answer choice carefully when faced with a large block of matrix questions. Some participants who speed through a study may select the same numeric answer choice for the entire block of questions.
Consider having at least one positive and one negative open-ended question in each task to capture what is working well on the site for participants and what is not working well. A multiple response question with a list of potential issues encountered is also helpful in drilling down and understanding the problems participants are experiencing on the site and how pervasive the problems are. Coming up with a thorough list while designing the study will make analysis much easier.
Specific qualitative questions can also be asked to obtain more specific feedback. These questions may be asked of all participants or of a subset of participants who answered a quantitative question in a certain way, but try to avoid similar questions that may be perceived as redundant.
For example, there may be one general question asking, “What did you dislike about the site while performing the task?” A participant may answer, “I didn’t know where to click.” Another follow-up qualitative question to a quantitative ease of use question may be, “Why was it difficult to find information about xyz product?” From the participant’s perspective, his/her answer may be the same as their prior qualitative response, not knowing where to click.
10. Be mindful of participant’s time, and yours
Even though there’s no limitation on how long a study can last, it is recommended that a URUT study be no more than 20 minutes. A 20-minute study usually entails about 5 tasks that take users ~ 4 minutes each to complete. You may also then add a final questionnaire with ~ 10 questions that will take users 2-5 minutes to complete.
A 30 minute study is pushing the upper limit of how much time a user is likely to spend on a study, UNLESS of course, you keep them engaged with interesting tasks and fun and challenging questions, or you are providing participants with a reasonable incentive for the amount of time they’re asked to engage in the study. (See point 11.)
It’s sometimes difficult to keep studies at a reasonable length for participants because there is much a researcher would like to learn for the amount of time and money invested in the project. Keeping study times shorter in length increases the completion rates, thereby ensuring that the data is collected in time. Long studies can lead to participant fatigue compromising the quality and integrity of the data captured.
Splitting up a study into two or more separate projects might be one option to an ambitious research project. Another option is to have participants perform a subset of tasks rather than all of them.
11. Preview internally and soft-launch your study to a small sample
Preview the study to eliminate problems with the spelling, grammar and logic. Have another person, preferably one who is not familiar with the study, review the study. This person can not only help spot grammatical problems and logic issues, but also provide feedback on task difficulty and the length of time required to complete the study.
Once the study has been thoroughly QAed and tested, conduct a soft-launch or pilot. This means testing it with a small representative sample, usually with 10-20 users before blasting the study out to a large number of participants. The purpose of the soft-launch or pilot is to identify any problems that may have been missed during the QA and testing process.
Soft launching studies is essential because they help avert disasters before time and money is wasted on questions or tasks that are problematic and do not provide useful data.
Researchers can evaluate the kinds of responses that come in. Sometimes looking at the responses helps researchers evaluate whether they are optimally addressing the research questions. If problems are spotted in the soft launch and the study needs to be fixed and re-launched, then only a small number of participant data is lost.
12. Be fair in your incentives
For lab studies as well as URUT projects, it is ideal to find users for whom incentives are secondary to providing invaluable feedback. However, incentives are a vital part of usability testing projects and it is important to incentivize users fairly for their time and effort.
One important advantage of the URUT methodology is that it is cost-effective. While a lab-based study incents anywhere between $50-100 per user, a URUT project incents between $10 and $20.
There are many factors that may raise the incentive amount. The cost is dependent on the complexity of the participant profile, study and device type.
In our experience, a mobile study has higher completion rate when panel providers introduce higher incentives. For a 15-minute mobile study, you can expect to provide $12 – $25 as incentives.
Incentives do not necessarily have to be monetary in nature. Discount codes, access to exclusive content or information, donation to a charity, or a drawing to win an item are some other possibilities to consider.
The Successful Remote Unmoderated Usability Testing method is capable of capturing exciting research outcomes. First, it has the capability to bolster research results with statistically significant data such as time on task, success and efficiency rates. Second, it justifies the usability metrics with the “whys” by way of clickstreams, heatmaps and video session replay.
These twelve best practices will help you maximize the benefits out of your Successful Remote Unmoderated Usability Testing.