Moderating Usability Studies for Varied Audiences
By: Ariel Roy
Many usability studies are designed with adult, native English-speaking users in mind, overlooking key populations who are often legitimate “intended users” of many products such as epinephrine devices, inhalers, and insulin delivery devices, to name just a few. Evaluating products with younger participants—children (ages 2-11) and adolescents (ages 12-21)— as well as non-native English speakers is just as crucial for ensuring safety and effectiveness as testing with native-English speaking adult users. However, research teams must modify their approaches when working with these diverse groups to gather valid, useful data. Specialized moderation techniques are necessary to elicit meaningful insights from different user groups. In younger users, this can involve accommodating developmental differences in cognitive abilities, attention spans, and communication styles; among non-native English speakers of all ages, this can include considering language and cultural variations. These adaptations help ensure research findings translate to accessible and usable products for the full spectrum of intended users. The methods we’re about to describe are applicable in all user research contexts: whether evaluating the safety of operating a medical device or discussing a user’s ideal experience when brewing their coffee every morning.
The Importance of Expert Moderation
The success of usability testing fundamentally hinges on expert moderation. When working with diverse populations such as children, adolescents, or non-native English speakers, this expertise becomes even more critical. Without expert facilitation, findings from testing may not accurately reflect real-world user experiences. A skilled moderator knows when to step in, and when to let users work through challenges on their own – recognizing that though watching users struggle can be uncomfortable, it is often necessary, yielding important insights. They excel at listening to what users say, but also at probing into what is unsaid, without leading the participant.
Expert moderators bring several essential qualities to the research room, combining their technical knowledge of research methodologies with a deep understanding of human capabilities and limitations across physical, cognitive, and perceptual domains. Typically, these moderators have received comprehensive formal education in human factors engineering, cognitive psychology, or related disciplines, along with specialized training in evaluation methodologies and medical device testing. This foundation shapes the process in which studies are conducted, bolstering it with scientific and professional rigor. This allows the moderator to notice subtle user behaviors that may go unnoticed by less specialized moderators, and to ask probing questions that reveal deeper underlying cognitive processes.
While someone with good interpersonal skills might create a comfortable testing environment, expert moderators know exactly when to intervene and when to observe silently, how to phrase questions without introducing bias to the results, and which unexpected user actions might signal potential safety issues. They bring patience, awareness of potential pitfalls, genuine curiosity, and the ability to recognize and mitigate their own biases. Most importantly, they’re adaptable – quickly assessing and adjusting to participants’ varying proficiency levels.
One of a moderator’s key responsibilities is to accurately identify the source of user confusion, whether it’s because the task instructions are unclear or because the product itself has design flaws. For instance, when a child struggles with a task, is it because the instructions were unclear to them, or does the design itself need improvement? Similarly, if a non-native English speaker has difficulty following the instructions for use (IFU), does this point to a language barrier or an unclear interface?
This expertise allows them to respond fluidly in real-time when unanticipated scenarios emerge, maintaining study validity while capturing valuable insights. When a device leaves the hands of its creators, unexpected scenarios are virtually guaranteed. True, designers and engineers are remarkably creative, but even the most proficient user experiencing a design with fresh eyes may bring a very different perspective to a study. We’ve been met with a multitude of surprises in our work – a participant inserting a nasal spray device into a manikin’s ear, or another slamming a prototype against a table to test its “durability” come to mind. A well-practiced moderator thinks on their feet, maintaining their focus on understanding the root cause through post-task interviews while ensuring participants don’t feel they’ve made a mistake.
Essential Moderating Techniques for Diverse Audiences
Approachable Presentation
Something as simple as clothing matters more than you might expect. Being observed by a moderator in a suit can make participants feel like they’re being evaluated. Dressing casually helps create an atmosphere of equal standing and comfort, particularly when working with children and adolescents.
Similarly, using complex vocabulary can intimidate participants who may have different language skills, particularly in a user testing environment involving laypeople, making them feel judged and evaluated rather than relaxed and at ease. Technical jargon, complex vocabulary, or overly formal phrasing can intimidate participants and influence their responses. This can hinder their ability to provide honest feedback by redirecting their focus towards hiding their verbal insecurity rather than using the product as they normally would.
For example, rather than asking a participant:
“How could this design be improved?”
“Did you experience any pain points when completing that step?”
“What barriers did you encounter when attempting to accomplish this task?”
We may ask:
“What would make that step more obvious to you?”
“What changes would you make if you had a magic wand?”
“What could be done so that someone else doesn’t get confused by [issue observed by the moderator]?”
“I noticed you stopped to think while completing that task. What were you thinking about?”
The intention isn’t to oversimplify, but to know your audience and make them feel comfortable. Ultimately, the goal is to have them share valuable insights while not feeling judged or interrogated.
Active Listening
Active listening goes beyond simply hearing what participants say – it’s about creating an environment where they feel heard and valued. Participants, especially children, often doubt the value of their insights or hesitate to share concerns they may think are trivial. This self-doubt can be particularly pronounced in medical device testing, where participants might feel intimidated by the technical nature of the product.
A skilled moderator demonstrates active listening through both verbal and non-verbal cues: maintaining eye contact as appropriate (or avoiding it, to avoid putting pressure on a participant), offering occasional nods, and using thoughtful follow-up questions that show genuine interest in the participant’s experience. When a participant shares their thoughts, we acknowledge it meaningfully rather than just moving on to the next task. For example, if they mention struggling with a particular step, we might follow up with: “That’s an interesting observation about the wording in Step 3 of the instructions. Could you tell me more about what made you feel that step was challenging?”
By consistently reinforcing that their participation matters, and their perspective is genuinely valued, we help participants feel more confident in sharing detailed, honest feedback. This approach often reveals subtle usability issues that might otherwise go undetected in less supportive testing environments.
When participants dismiss their own feedback as “dumb” or attribute issues to their perceived shortcomings, we remind them why their input is valuable. Emphasize that their observations help identify issues that others will likely encounter, ultimately contributing to safer and more intuitive product designs.
Clear Expectations
Many participants, drawing from their academic experiences, may feel they’re being tested. Past experiences of being questioned by authority figures – whether in classrooms, job interviews, or performance evaluations – can create anxiety and affect natural behavior. This is particularly true in a research setting where participants are being observed, recorded, and asked to complete specific tasks.
It’s essential for us as researchers to explicitly clarify that we’re evaluating the device’s design and how well it communicates proper usage – not the participant’s abilities. We begin sessions casually by explaining that there are no “wrong answers,” that we did not personally design the product and our feelings cannot be hurt by their feedback, and that any difficulty encountered helps us identify opportunities to improve the design. This shift in perspective helps participants feel more comfortable sharing honest feedback, including those moments of confusion or frustration that are integral to our research.
The Impact of Expert Moderation
The art of moderating usability tests extends far beyond following scripts. It requires a careful balance of technical expertise, emotional intelligence, and adaptability. When done right, expert moderation transforms usability testing from a routine evaluative exercise into a rich source of insights that inform meaningful product improvements.
In medical device testing, where user understanding directly impacts patient safety, the stakes are particularly high. Expert moderation creates an environment where participants feel comfortable sharing their thoughts, helping to uncover crucial usability insights that might otherwise go undetected.
Every usability test presents an opportunity to make products safer and more effective. Through thoughtful moderation techniques and attention to participant needs, we can ensure these opportunities translate into meaningful improvements in product design and user safety.
Ariel is a human factors researcher and user experience (UX) specialist at Priority Designs, where she has worked since 2021. She holds a B.S. and M.S. in Human Factors Psychology from Embry-Riddle Aeronautical University in Daytona Beach, FL. She uses her expertise in human factors to improve product safety and effectiveness for PD’s consumer and medical clients. She especially enjoys interviewing and working with the end-user to understand their wants and needs, and translating those findings into informed design recommendations.
Human Factors Specialist