5 Effective Ways for Usability Testing to Play Nice with Agile


Jeff Gothelf

June 15, 2011

Back to Resources

Usability testing has been a fundamental tool in the UX arsenal for decades now. The value of actually meeting your customers and letting them experience your product makes a significant impact to the shape of that product. In it’s most formal version, testing can be a multi-day, multi-thousand $/€ process that delivers final analysis days if not weeks later. With many organizations moving to an Agile philosophy and methodology, UX practitioners are finding it difficult to integrate formal usability testing into this faster-paced, iterative approach to software development.

See? Lions and zebras can get along. So, too, can Agile and Usability Testing.

See? Lions and zebras can get along. So, too, can Agile and Usability Testing.

The main challenge is time. The need to keep the teams moving forward, coupled with typical two-week development cycles between releases (aka sprints) doesn’t provide an obvious point to insert testing. Indeed, fitting a formal usability testing cycle into a series of two week sprints is challenging if not impossible. Instead, consider the following techniques to help get that critical customer feedback and still keep your team’s momentum.

1. Start small.

Jakob Nielsen has proven through scientific method that 5-8 is the optimal number of participants for qualitative tests. Consider testing only 3 people each time instead. The goal of agile usability testing is to clear the boulders from the road – not to provide pixel-level design direction. Three participants will, very quickly, validate (or not) your product’s core workflows and give your team a sense of where to focus next.

In addition, three participants take less time to test than 5+. Whereas a traditional rubric would have a team tied up for a day or two, a 3-person study can be completed in a morning. This is especially helpful if you don’t have a dedicated research staff. Your UX generalists can test, get feedback quickly and be back optimizing the design within one day.

Finally, three participants are cheaper. There is greater value in testing multiple iterations on more customers than one design on many customers. It allows you to move forward more quickly and keep your Agile team’s momentum.

This approach assumes you are performing moderated usability testing in the context of a lab, work site or some other location. As an additional data point, consider using remote usability testing as a way to provide additional perspective on your findings.

2. Test *every* week.

Pick a day (preferably not Monday or Friday) and make that your “testing day.” Every week, on that day, you’re in the lab, coffee shop, or customer site testing. Make it clear to your teams that this is testing day. Put recurring meeting invitations on your calendar and forward out to stakeholders. The night before testing day, send out a reminder with a participant schedule.

The goal is to make that day synonymous with testing. Teams should stretch to get work into that day’s test and use the cadence to drive productivity.

What ends up happening is the organization becomes accustomed to a steady stream of qualitative insight coming from the UX and product teams. Testing day becomes a target for quick-turnaround validation and the team becomes almost reliant on that stream to ensure the quick decisions made at iteration planning line up with business and user goals.

In weeks where testing day gets de-prioritized (hey, it happens), a concurrent micro-study using an automated online tool can maintain the flow of insight until you get back on track.

3. Show them whatever you have ready.

Customers will react to whatever you put in front of them. Yes, the reaction will differ based on what that item is or it’s fidelity but you’ll get *some* feedback — and that’s much better than nothing.

If you have code in production or a qa environment, show it. If you have wireframes or mockups, show them. If you can string together a series of mockups in fireworks, visio or even PowerPoint, do that and show it to real customers. A napkin sketch is also viable. Anything that will get your ideas in front of users will provide value. Even if you have nothing, it’s still worth sitting down with a customer and talking about their pain points.

4. Invite everyone (or bring the testing to them).

*Note: this tip assumes you have some kind of lab onsite. If you don’t, and typically do remote usability testing, guerrilla testing or site visits, point #5 below is most crucial to your success.

Key to getting the findings from these weekly tests into the iteration is buy-in from the rest of your scrum team and the organization as a whole. The most effective way to do this is to get the engineers, product managers and stakeholders to the testing sessions.

Most testing software broadcasts over your local network and if yours doesn’t you can always use Webex or GoToMeeting to broadcast. Setup multiple viewing areas convenient to your team. Ensure they know where to go and how to turn viewing on.

Another option is to enable viewing at each person’s desk. Again, most usability and general screensharing software allows this. This is most convenient as it doesn’t require the team to go anywhere to participate.

After viewing one or two customers the team gets a sense of where the problem areas are leaving you free of the responsibility of convincing them later. In addition, most team mates start to like the immediacy of the feedback on their work and start attending regularly without much reminding.

5. Report back quickly.

This one is critical. Turn feedback around to your team and to the organization as quickly as possible. No need for a fancy report or designed layouts. Type up a quick executive summary followed by recommendations for the pain points discovered and send out via email. Many tools offer some kind of findings visualization that allows you to, very quickly, generate a scatter plot or heat map of user activity. Ideally you should keep a copy of the findings in a commonly accessible place like a wiki or shared folder.

The goal here is twofold. One, you are getting customer feedback to the team so they can adjust their work accordingly. The other goal is to tell the broader organization that, despite moving quickly, you’re validating your approach with real people.

*5.5 – Use an outside recruiter.

Getting the right customers to your studies is critical to the validity of your findings  Keeping up the pace of recruiting participants to your studies on a weekly cadence, however, is a time consuming endeavor. It is highly recommended that you find a resource external to your team to perform this task. If you can afford it, hire a recruiter who specializes in this task. If not, minimize time on task for this by spreading the work across the team using a shared calendar.

Agile development practices pose some challenges to traditional usability testing methods. However, with a bit of modification and a focus on outcomes as opposed to output, getting to validated learnings can be successful and rewarding.

What’s been your experience? Share your feedback in the comments.

This is the first guest post on our blog, and we plan to do more. Interested in writing a post for us? Please contact loucas@usabilla.com

Subscribe for the lastest CX content

Copyright © 1999-2021 Momentive|888-684-8821|Privacy notice|California privacy notice|Terms of use|Cookie policy

*Net Promoter, Net Promoter System, Net Promoter Score, NPS and the NPS-related emoticons are registered trademarks of Bain & Company, Inc., Fred Reichheld and Satmetrix Systems, Inc.