Table of contents
1. What does Machine Learning mean for users?
1.1 Discovery and Ideation
1.2 How can research help during the ideation phase?
2. Modeling phase
3. Prototyping the algorithms
4. Continuous monitoring
1. What does Machine Learning mean for users?
Let’s make it clear from the get-go that machine learning is not necessarily the most meaningful term to use. To users, Machine Learning translates to automation.
For example, instead of spending time looking for new music or movies they like, they allow the system to give them suggestions based on their past choices and therefore the automation of actively searching.
That’s why when we invite people to use ML-based products, we are asking them to automate part of their operations or even their entire roles. And that’s exactly why experience research matters when designing ML-based features.
Without discovering whether users trust automation, find it useful or indeed have a need for it, your product is a wishful shot in the dark.
Involving users will result in a system that is not just smart but also reliable, trustworthy and, most importantly, used.
UX Research (UXR) can help throughout the entire development cycle but especially in three crucial moments in the ML workflow:
1. Before investing in a product (discovery and ideation)
2. While designing and training the machine learning models (modelling)
3. While evaluating the implementations for updates
1.1 Discovery and Ideation
Research directs ML-based technology to answer the core questions before investing in a product:
- Who are the users? Do they have different ways of overcoming their needs? Why?
- What do users need to achieve?
- What enables them to complete the steps at hand, as efficiently as possible? And what does efficiency mean to them? Is it doing tasks more easily and quickly or is it something else? Inviting your team to focus on these questions helps guide them away from what’s considered “innovation” or “cool” to what truly works best for the users and business goals.
Once you are all on the same page about the answers and users’ needs, you can move on to the ideation phase and brainstorm and select 2-3 ideas to iterate on till you have a winning concept.
1.2 How can research help during the ideation phase?
It’s already in the ideation phase when the team is considering different solutions, technical feasibility, and potential profitability.
This is when a researcher can assess if the product idea is reasonable to users and pinpoint any probable challenging scenarios before the ML team begins gathering data for building a model.
Researching users’ mental models throughout this stage is crucial to make sure that you conceptualise and understand their would-be system
Here are a few points that UXR can help with clarifying:
- Non-ML vs. ML-powered mental models: How users currently meet their needs without ML means that they will need to either adapt or completely form a new mental model when using an ML-powered product. Taking a closer look at the user’s journey through non-ML solutions can reveal insights for the new ML solution, making it easy for the user to adapt and understand.Researching this will also enable your team to design transparently so that users know how their data will be used, making the product more trustworthy.
- User needs: How do users currently overcome their needs with or without ML applications? Do they have any workarounds? What problem is the new product attempting to solve? To what extent do users’ needs, and workarounds, overlap with the solution at hand?
- How fast and accurate do users expect the model to be?
ML’s accuracy depends on the type of industry and user needs within that specific domain. Sometimes, the model is designed to predict user preferences for movies or predict a property’s price based on its features. A few irrelevant movie suggestions or accuracy within a few thousand dollars is probably satisfactory to most users. However, an ML model used for cancer prediction or healthcare should be extremely accurate. And should a model provide solutions within a few seconds or instantaneously?UXR can help figure out industry expectations for different domains by interviewing field experts and end-users.
2. Modeling phase
During the modelling phase — this is when training a machine learning algorithm happens — UXR brings in more data with the goal of closing gaps or holes in the data.
Before and throughout this stage, research can help gain a concrete insight into how the user is currently going through their step-by-step journey and if an ML model can make their tasks more efficient.
This is where already existing user journeys and product’s learnability are studied extensively through a combination of methods for the relevant industry through contextual inquiries and lab-based cognitive walkthroughs. By zooming into the existing mental models, UXR can provide insight into how to train the model, its interface design, and its learnability.
When studying the user’s journey, you will notice parts of the process that users enjoy doing because it’s rewarding to be creative and develop output. Such parts of a user’s journey make users feel accomplished because user effort is part of the creative process. Carefully studying every step of the user’s journey and their mental model enables you to make informed decisions on which processes to automate, how and why.
3. Prototyping the algorithms
This is basically creating an early low-to-no-code prototype that behaves similarly to the ML-based feature or product. For this to happen, you don’t need a fully-fledged prototype. In fact, you can use the Wizard of Oz technique to simulate what the system is doing.
In tests, I have been using a combination of an early prototype with Wizard of Oz to capture users’ verbal and behavioural cues on system interaction. When users see how the system reacts, they can form mental models based on their reactions.
You could help your team by asking the following questions while they are thinking about the earliest prototypes:
- How does the model describe what it is doing? [thinking about transparency and building trust with users]
- How will users provide feedback to the model? [ideation about possible user interactions to train the model continuously]
- How will the user be able to approve or intervene? [User control]
Answering these questions will help your team better understand how to model intelligent algorithms. That’s why you want to conduct user research by pinning these questions on top of your plan:
- What do users think about these suggestions for what to do next? (Helpfulness)
- Do users know how the system decided to …? [Action, understanding user’s mental model about system’s operation]
- Was there enough information for users to (take action)? [Confusion vs. clarity on how the system works]
- Do users know how the model would change over time? [user’s knowledge — this one especially matters because the user’s acknowledgement of the system’s transparency is linked to trust]
Besides qualitatively testing the prototype, you might consider other approaches in case your team has very large stakes in the product, this is especially for cases when ML is a visible part of the experience journey.
That’s why measuring users’ attitudes (trust), behaviour (task duration and usability performance) and mental models (how the system should operate) should all equally matter in ML-powered products.
4. Continuous monitoring
You have run your research, provided your team with insights and now they are more confident about the design direction. But your work is not done, yet!
Now’s the time for more research to shine up to ML modelling by reassessing how the trained model meets the users’ expectations.
Evaluating the ML-based system against user’s mental model:
You want your system to adapt and evolve with the user’s mental model. This way users won’t be forming their own theories about how an ML-powered system works.
The importance of frequent behavioral studies is to help your team accommodate the ML-model to the user’s mental model. Continuous research is needed because:
- Humans change over time, and so do their behaviour
- Technology is changing all the time: people influence technology and technology influences them
- Your team’s biases change over time
- Product value and perception of usefulness are non-linear.
Once the model is in use, it will be easy to draw conclusions based on analytics and other quantitative data, however continuously testing the model in the hands of the users will reveal feedback on how it feels and reveal answers to such questions:
- How easy is it for users to use the system? Easy, difficult? Do they even notice there’s ML embedded in their journey?
- How’s their life with it? Is it improved, do they like it or can they live without it — why and why not?
With insights collected through direct user feedback, product developers can make educated decisions together around what improvements to implement in the next upgrade.
5. Wrap up
The agile philosophy has conditioned us to go for lean user research and test an MVP by releasing it into the market as quickly as possible. While embracing the agile method is great, it cannot come at the cost of quality, and as we have demonstrated, this is critical in the ML space.
Research during discovery, ideation, before and post-launch is incredibly valuable. Make sure you help your team by asking the right questions to shape the direction and the design of your ML-powered products.
