I came across this article back in 2011 and it was written in 2010, and I saved it on a little forum site I have so that if webpages ‘disappear’ (as they do) then I would still have a copy. I’m moving this forum site now so I am writing up the posts that I still find useful, and this one has been around so long and I still refer to it – I should memorise it really, so I am writing it up briefly here.
The following block of text highlighted from the webpage is one of the important passages,
“As a strategy, pick some percentage of problem occurrence, say 20%, and likelihood of discovery, say 85%, which would mean you’d need to plan on testing 9 users. After testing 9 users, you’d know you’ve seen most of the problems that affect 20% or more of the users. If you need to be surer of the findings, then increase the likelihood of discovery, for example, to 95%. Doing so would increase your required sample size to 13.
The best strategy is to bring in some set of users, find the problems they have, fix those problems, then bring in another set of users as part of an iterative design and test strategy. In the end, although you’re never testing more than 5 users at a time, in total you might test 15 or 20 users. In fact, this is what Nielsen recommends in his article, not just testing 5 users in total.
So if you plan on testing with five users, know that you’re not likely to see most problems, you are just likely to see most problems that affect 31%-100% of users for this population and set of tasks. You will also pick up some of the problems that affect less than 31% of users– just not 85% of them. For example, a sample size of 5 should pick up about 50% of the problems with likelihoods of occurrence of 15%, 75% of the problems with likelihoods of 25%, and so on. Change the tasks or type of users and you’ll need a new sample of users. “
I would love to read any comments if you have used this five user testing method, I have tested with 3 and already so many issues and thoughts have cropped up, so I can easily see why this is a great process for eliminating initial problems in your designs / prototypes.