Everything you should know about user testing

What is user testing?

User testing surveys, tests, and other tools are the means to bring end users’ views into the development process. This is especially true when working with agile principles which emphasize the importance of user input. The phrase “user testing” describes research methods designed to test the functional aspects of a digital product with end users. The broad goals of user testing are:

• to investigate the usability of the solution being developed,
• to establish whether the product addresses user needs,
• to explore any user concerns,
• to identify ways to improve the user experience.

Although there are many answers to the question, how to do user testing, the common feature is that all methods involve real users using the product to perform specific tasks in realistic conditions.

The benefits of user experience testing

The basic goal, and fundamental benefit of user testing is to decide whether the digital product is on track to be launched. Within this umbrella benefit, there are a number of specific advantages to user testing.

• Cost savings – Timely user testing means spotting design and functionality issues prior to deployment and release. The earlier such issues are identified, the less likely the development process goes down a blind alley. Unnecessary expense is therefore avoided, especially when user testing techniques are applied to early or partial versions of the product, such as prototypes or MVPs.
• Broader design input – Until we have user input, the product is based on the assumptions of the development team and stakeholders. Those assumptions may be accurate but until we test with users and observe real user behavior with the product, we cannot be sure which aspects and functionalities are really important to users, and which aren’t.
• Improved user satisfaction – By identifying and removing design bottlenecks (e.g. confusing or contradictory messages on an e-commerce platform prevent the user from completing sign-up or payment) the product is easier and more intuitive to use, improving the user experience.
• The final design is corroborated by facts – User testing ensures that the final product design is data-driven. When deciding any points of contention within the development team, data always wins.
• A source of innovation – Research and testing often uncovers new insights and in turn prompts fresh ideas for the product’s final design. This might be new features or functionality or even complementary new products.

Key factors when choosing user testing tools

The range and variety of user testing methods is significant, however, most tests can be defined according to a few criteria or factors.

Type of data gathered

Some tests gather information about user attitudes and beliefs. This is useful in terms of identifying and confirming user psychology. Examples include interviews and focus groups. Other tests seek to observe and classify user behavior, relying not on what users say but what they actually do. Examples include eye/mouse tracking and A/B testing.

Type of analysis conducted

Quantitative research gathers data which can be counted and measured. It produces objective, simple answers to such questions as, how many and how often. Quantitative testing is easy to measure, and repeat for comparison purposes. Examples include questionnaires and A/B testing.

Qualitative research is focused on gaining a deeper understanding of the issues by gathering subjective personal opinions and viewpoints. Examples include focus groups and ethnographic research.

Degree of formality

Formal testing utilizes standarized formats and usually takes great care to ensure that test subjects are closely representative of the product’s target users. Processes are easy to repeat (for comparison) and tests use prepared materials that are as close to the final product as possible, depending on the stage of development. Results tend to be accurate but the testing process is likely to be relatively costly.

Informal tests – sometimes called ‘guerrilla research’ – are less comprehensive, seek to test specific, limited aspects of the product, and are often used to give a quick indication or signposts on key elements of the design. Informal testing tends to be popular in startups and teams using a lean approach as they are agile, budget-friendly and quick.

Moderated vs unmoderated user testing

The difference here is whether or not a facilitator or moderator is present during the testing process. Moderated tests give you a high level of interaction with the user, allowing you to dig deeper into specific responses or opinions. According to Norman Nelsen research, for moderated user testing a group of five users is enough to ensure reliable results.

Unmoderated tests leave the subject to carry out the test solo. Without the need for a moderator’s presence, larger groups are needed in order to achieve accurate, generalized results. Unmoderated tests are usually much cheaper and quicker to administer, allowing a broader range of testing for the same budget.

Graph: User Experience Research division - based on Christian Rohrer | design concept Iga Mościchowska

Different types of user testing

Which user testing method you use depends on the product, the target users, and the research hypothesis you’re aiming to test. Having decided what needs to be tested, you can then decide how to test it. The main types of test in common use are as follows.

1. Individual interviews

   A moderator uses a prepared set of questions and/or tasks with individuals on a one-to-one basis. The goal is to collect qualitative data regarding user behavior, needs, and pain points for the further improvement of the product or to discover new development opportunities.

2. A/B testing

   In an A/B test, two different versions of a product or feature are tested with users. The purpose is to validate which solution provides a better user experience and/or better addresses user needs.

3. Card sorting

   A test for a group of users in which they prioritize a set of cards or place them in a logical order; cards may represent products, services, stages in a process, etc.

4. Heat maps, eye-tracking, etc.

   Tests like this use software to analyze product use. Eye-tracking follows the user’s line of sight, identifying which parts (the ‘hot zones’) of a webpage attract the most attention.

5. Focus groups

   Groups of users are gathered for a moderated discussion relating to the product. This kind of test is especially useful when the basic issues are well-known and the development team needs to dig deeper into user perspectives, feelings, opinions, and attitudes.

When to deploy user testing tools?

User testing should be considered as a continuous thread, conducted throughout the development process and addressing all key elements of the product.

In the context of the build-measure-learn cycle we use at Boldare, user testing forms a key part of the ‘measure’ part of our approach. Whenever a new feature or functionality is built, user testing is conducted to confirm the usability of the solution. This may be as simple as testing a low-fidelity prototype to validate the general approach, or an MVP to establish what remains to be done before launch. In the case of an already-launched product, user testing should continue periodically to observe users’ product use in the real-world environment. Such a strategy supports the longer term maintenance and scaling of the product.

The cost of implementing user testing tools

On the surface, user testing seems to be a potentially significant additional expense in digital product development. The reality is that product development is a gamble – the developed product is the best possible response to the identified user needs, based on the information and data available. User testing is an investment which results in more and better data, and consequently, a better product quality.

When the right tests are conducted, asking the right questions, the result is likely to be a saving on the budget compared to what it would have been if a lower quality product is released, requiring further development to correct issues as they come to light.

User testing at Boldare

At Boldare, we work in cross-functional teams based on the scrum framework. Teams consist of various specialist functions, including product designers, developers, product strategists, quality assurance experts, and always a scrum master. This means a variety of expertise with input into the user testing strategy. Our product designers take a leading role in preparing, conducting and analyzing user testing, although the rest of the team are involved from the perspective of their particular specialisms.

At Boldare, we find user testing to be an indispensable part of the product development process, supporting a research-based and data-driven approach to building the best possible product, addressing genuine user needs and meeting business goals.

The importance of user experience testing

Just to say it one more time, user testing is absolutely essential to creating successful digital products. Identifying user needs and concerns up front is a good start, but user testing tools should be deployed at various stages of the development process to ensure the product is on track. Then, post-launch, an ongoing program of user testing surveys and other user engagement will enable you to maintain the product’s market fit and point the way to future scaling for new markets and users.

Are you curious how it feels to become a member of Boldare team? Visit our career page and check the current vacancies!