Why Remote Research Is (Or Isn’t) Right For Your Project

17 Apr 2020
Research
April 17, 2020 vbreon

Why Remote Research Is (Or Isn't) Right For Your Project

By Katie Mowery, Sr. Human Factors Specialist

Times have changed for user-centered design research.

With the realities of social distancing likely impacting society for months, if not years, those of us who do physical product evaluation with users have spent the last few weeks reconfiguring our study designs to keep moving forward with a user-centered product development process. At Priority Designs, we’ve spent the last few weeks getting innovative and being realistic about how we move forward with design research in a socially distanced world.

We’d like to share our experiences with remote research over the last few weeks, along with a perspective of 20 years of experience in design research. In this article, our intent is to help you understand when remote research supports good decision making, what to consider regarding your unique project, and when it might be best to wait for in-person research.

priority_designs_usability_human_factors_research_003
"In this article, our intent is to help you understand when remote research supports good decision making, what to consider regarding your unique project, and when it might be best to wait for in-person research."

Remote Research Has Many Benefits

1. In-Home Context

Many products today are intended for in-home use. Doing the interview remotely, while the participant is at home, can allow researchers to understand the context of use better without the time and planning required for in-home, in-person interviews. Additionally, some participants might be uncomfortable with allowing a team of researchers into their home (especially a team consisting only of the opposite gender), so remote interview techniques can lend a feeling of safety without sacrificing context.

2. Convenience

If ever there were a great use for ethnographic journaling tools, this is it. Allowing participants to journal their experiences with a product at any time of day creates a level of convenience that will increase your compliance. While our audience is home and somewhat ‘captive,’ they might have more freedom to document experiences ‘real time’ rather than waiting until the end of their shift.

3. Live interview and observation is still possible, and valuable

Once you get over any technology hurdles, there are so many options for using cameras and web conferencing software to watch, interview and record your participants’ interaction with the product. Whether you’re using screen capture or webcam, with a good camera and view, we can still see our users’ natural interaction with a product and probe into any difficulties we observe. Quality moderating is all that’s needed here, probing into what you’re seeing and getting down to the root cause of the experience (what attribute of the design is leading to that experience) is just good evaluative research, whether you’re in person or remote.

Concerns About Validity of Remote Research

Without a doubt, there are pros and cons to remote research. Some organizations may be wary of whether the data they get will truly provide the confidence they need to make million-dollar decisions, and for good reason. They recognize, as do we, that conducting certain types of research in-person, in a controlled environment, is the ideal. It is critical to observe all product interactions and understand the true root cause of any difficulties a participant might have.

There are several important elements of research that are more difficult remotely, but when you understand and plan for limitations, you can still gain valuable insights that keep you moving. Let’s go through some considerations to address any roadblocks and risk mitigation solutions to ensure that both researcher and client can feel confident about the validity of the remote research:

1. Building Rapport

Discomfort with technology can create a sizeable roadblock in putting participants at ease to share their experiences. Similarly, there’s no guaranteeing that challenges with Wi-fi bandwidth or video conference services won’t get things off to an awkward start – from choppy audio to freezing video and speaking over each other, video interviews can make for uncomfortable experiences. As seasoned researchers know, if you can’t establish good rapport at the beginning of an interview, it can be difficult to ever get the participant to truly open up.

      • Mitigation: Add time to talk to your participant before the session starts. Be transparent about how these methods of interaction can sometimes be awkward and that you’re in it together. Be human, be humble, and be humorous. Make small talk just like you would in a face-to-face session to help get them comfortable.

2. Observer Anonymity

Most web conferences aren’t set up for participant or observer anonymity. Clients often want to dial in and listen to the interview live, just as they’d be able to watch from behind the mirror in the lab. The drawback of this is that their presence is known. When the participant logs into the web conference, he or she can see exactly how many other people are listening to the call, which can lead to discomfort and anxiety, especially as those numbers increase. At times, this could even give away the client’s identity. Even if their names don’t appear, their location and company email address might. If that location is focused in a specific area (like the same city as a certain medical device headquarters), then it can be easy for a savvy participant to deduce who’s behind the research.

      • Mitigation: Research your web conference platforms on how much participant information they show, choosing one that allows for the most anonymity. Send a list of rules to your clients for participation:
        1. Use only first names, and last initial if required
        2. Do not enter company name or emails (these are usually not required fields in most web software)
        3. Dial in no less than 5 minutes early so the participant doesn’t hear them join the call
        4. Keep your phone on mute. Only the researcher(s) and participants should be heard
priority_designs_remote_research_home_medical

3. Context of Use

If the context of use for your product is in the home, independently, while sitting relatively still, then great! Or for example, many products designed for home or personal use are often experienced by the whole family, so testing them in-home can be a valuable way of understanding their true impact on all users. However, many products are not intended for these conditions. As a result, the objectives of the research need to be considered carefully when weighing how changing the context of use will have an impact on the validity of the data you collect. Let’s tackle each of the possible challenges separately.

  • Environment
    When you are conducting observational research, context is critical. For us, this is often the operating room, a dental operatory, the job site or even a mock setup in our lab. Many factors of those environments can play into how the product is interpreted and used and the validity of the resulting data.

    • Example: A medical device intended to be used during surgery that communicates primarily with audio tones would need to be tested under a representative sound simulation at a minimum.
    • Mitigation: in the case of audio, consider whether a recording of representative sounds can be played while the user is interacting with the device.
  • Group-use
    Products might be intended to be used by more than one person during a task. This is especially true in the medical environment, where it’s common to see devices passed between clinicians. Another common use case involves communication between a nurse who’s manning the interface while a doctor operates the associated handheld device. When testing in-home, those other key roles are not represented which makes task completion, well, incomplete.

    • Example: If a surgeon needs to keep his eyes on the patient while operating a handheld tool, and the nurse verbally confirms changes in settings as she modifies them on a separate interface, the two roles would need to be represented in the study to ensure validity.
    • Mitigation: If it’s a matter of communication between two roles, consider having a member of the research team ‘role play’ as the other user to fill the gap.
"Many products are not intended for home conditions. As a result, the objectives of the research need to be considered carefully in weighing how changing the context of use will have an impact on the validity of the data you collect."

  • Intended Use
    If you require the user to sit at a desk or hold a phone to participate in a live interview, the intended use of the product may be impacted and your data might be affected. You’ll see frustrating experiences with your product because the conference call is an added physical inconvenience that wouldn’t normally be present, and your data will be negatively skewed.

    • Example: Use of a specific tool at a job site where the contractor is using his or her hands and moving around the entire space to get the job done. If you’re asking the user to hold and constantly reposition the aim of a phone or camera, you’re going to struggle to capture the authentic experience.
    • Mitigation: Consider sending a camera with a tripod with setup instructions. This could allow the user to mount the camera and capture a wide field of view without holding anything.

4. Experimental Control
Lab-based research is generally best for ensuring all participants experience your device under the same conditions. In some cases lighting, noise, and other distractions can influence how they experience your product and how focused they are on the subject at hand. Testing at home adds an unpredictable variable to the mix, which could introduce uncontrollable variables into your study.

    • Example
      One participant experiences your product instructions with small text in a dimly lit room and complains that the font is too small or lacks contrast, while another experiences it in a bright room and has no issues. How do we deal with that data? This can impact the repeatability of your findings and how you interpret certain issues that are out of your control.
    • Mitigation
      In this case, consider the extent to which these extraneous variables could actually impact your data. Good researchers seek experimental control because that’s how we’re trained, but there are times when it can be sacrificed. If your desire is to identify whether the user can understand how to use your product, then that data can still be obtained even if other variables aren’t controlled. If it happens that the user mentions that it’s difficult to see some aspect of your interface, and this seems like an outlier, use the interview to understand whether the lighting in their environment is too dim, or if they would normally wear glasses, to identify whether this is an exception or a rule.

5. Logistical Challenges

It’s to be expected that coordinating around tech challenges will add to your planning time. There’s no guarantee that a participant will have good wi-fi, or even know how to connect to it effectively. Additional time is required at the beginning of, or prior to, each interview to work through technical difficulties that may arise. Additionally, if your research involves the evaluation of a physical product, prepare for the complication to increase. Testing products remotely requires more calendar time and logistical planning to ship and rigorous disinfect between participants, and may require the development of backup test materials should the original break in transit, or to minimize fielding time by shipping more than one out to participants concurrently.

    • Mitigation
      Just because the logistics add time doesn’t mean you can’t become a pro at managing those hurdles. It’s always safe to ship with plenty of time, or just drop the package off at your participant’s home if that’s an option. Be sure to add an extra week for customs if you’re shipping abroad. Make extra prototypes if you need to swap out any broken parts, and do a 15 minute “tech check” call before your participant’s interview. This is your opportunity to make sure they’re all set up and know the steps to take when it’s time for the interview. This reduces anxiety when it comes time to do the session and hides the “mess” from your clients or other stakeholders who might be watching the interview. It also puts the participant at ease, so they aren’t starting out the session with troublesome technology.
"Just because the logistics add time, doesn’t mean you can’t become a pro at managing those hurdles."

6. Confidentiality Concerns

The confidentiality around your product design is less secure when you’re sending your one-of-a-kind design out into the world. While it’s encouraged to ask participants to sign confidentiality agreements, there’s more of a chance they can take photos or screen shots and share them outside of your conversation.

    • Mitigation
      At a minimum, send a confidentiality agreement and require the participants to sign and return prior to participation. If they refuse to sign, do not allow participation. Keep these records on file should legal action be needed, but usually the agreement is enough to scare most participants away from doing anything dishonest.

As you can see, when carefully considered and properly planned, remote research can provide the answers you need to confidently make decisions and keep your product development process moving. This will ensure that when the time comes to validate your decisions with in-person research, you’re not starting from scratch and you’re all-the-more confident that your product is right where it needs to be.

Deciding What's Right For You

Remote research can be a great solution during times like this and can yield valuable data. But truthfully, it just isn’t right for all research questions, and there are some situations where we would advise clients to hold off. Let’s go over a few example situations:

Situations when waiting for in-person research is recommended.

  1. Is the environment in which this project is to be used critical in understanding the user experience? If getting the context right is critical to understanding product use (and in-home is not the right environment), you may want to wait until you can head back out into the field or test in a controlled laboratory setting.
  2. Are there other users who might share or collaborate the use of the product that can’t be present or simulated for testing? If this product requires joint-use between more than 1 user to complete a task, and you can’t get those people together, it might be better to wait.
  3. Does my research require a distraction-free, controlled environment to generate confidence in the results? If this is your final step in product approval, and the design is not still in a state of evolution, you want strong experimental design backing your decisions. This is likely not the time to sacrifice experimental control.

Situations when remote research should be pursued.

  1. Is my product design still evolving? Good human-centered design requires multiple research and design iterations, using qualitative research to unearth usability and user experience issues and preferences. If this is where you are in your product development process, remote research can still get you the answers you need to keep the program moving! Using journaling tools (Indeemo, D-Scout, and others) can help the user to document their experiences and needs over time, and live interviews via web conference can get you a wealth of data regarding the user’s impressions about your product so you can make informed decisions with confidence.
  2. Will the video interview physically impede the user’s ability to use the product? If the typical use case involves activity or both hands. Explore a means of sending along a tripod or wearable camera mount that might still get you a good view. If it still seems like managing the camera will get in the way of the product use, re-evaluate what questions you want to ask for remote research. You may be able to get some product impressions or usability data without the full set of tasks on which you originally planned.

Key Takeaways

Few of us have ever experienced a pandemic of this magnitude. Its impacts will require fundamental changes in the way we operate. We are encouraged by the adaptation and problem solving we are witnessing across the globe and find ourselves motivated to lend our experience where we can. While in-person research is still often the ideal, remote research is a reality that needs to be embraced. We believe navigating the challenges of remote research means thinking through each variable including:

  • Building rapport
  • Observer Anonymity
  • Context of Use
  • Experimental Control
  • Logistical Challenges
  • Confidentiality Concerns

For some research questions, the needs of the study simply cannot be met with remote tools in a way that is necessary, and it may be best to wait until you can conduct in-person research. But in many cases, these concerns can be mitigated and overcome to ensure the study can continue on with remote research that will yield valid, valuable and actionable insights. Researchers can help navigate these waters to recommend a solution that is best for your study.

Want to receive new articles and updates? Join our email list.

[mc4wp_form id="9498"]

About The Author

Katie Mowery, Sr. Human Factors Specialist
With an expertise in human factors psychology, interface and experimental design, Katie is passionate about understanding the way people think and feel, helping clients understand end users’ needs and applying those insights to improve the product’s design. Her work in automotive, retail, defense and software brings a wide range of knowledge that spans across industries. Katie is curious, thoughtful and she’s even done research while riding in tanks! When she’s not watching Disney movies with her two little girls, she’s playing outside as much as possible.