Tuesday, July 8, 2014

Set usability study observer expectations

In a recent usability study that my team and I were conducting, an event occurred that prompted me to write this article. Often times when you are facilitating a usability study on a design in an effort to identify usability focal points, you may decide that it would be good to have additional observers in the room with you while you are sitting with a user. During this most recent usability testing session, my user was working through a task that was somewhat difficult to complete. Yes, they probably would have needed an assist from me, but instead, what happened is that one of my observers spoke up and said "Oh, this point again? Yah, this is where everyone's been having trouble. You'll need to click the ...".

My jaw dropped.

What happened at that very point was that the 20-30 minutes we'd already invested in the study became nullified. We could no longer ethically use this user as part of our study due to this "contamination in our specimen". What a pain.

When you ask observers to come in and help take notes and observe a user test session, it's very important that they have a clear understanding of what their expectations are. On the above mentioned test, I did verbally state that they must be silent, but for whatever reason simply forgot.

Was it that I wasn't clear in the way I spoke to them? Could they have been thinking about something I just said when I told them? Maybe that understood, but then forgot. Who knows. The problem to solve at this point is to help the observer become more aware, or to rank the information about their expectations at a higher severity level. To assist with this, I recommend that you establish some sort of "Observer Expectations" document that you can hand them and then walk through with them. This will help this more clearly understand what they are bringing to the table.

Here is an example of one such set of guidelines that may assist in this process...


Observer Expectation Guidelines

Show Up Early!
When you are invited to be an observer at a usability study, it's important that you show up early so that you can have an opportunity to get settled. Your attention to this detail communicates to the user that you are part of this cohesive and organized usability study team. It also communicates to the user that you respect their time.

Stick around!
If you feel that you have a conflicting meeting or that you must leave to go to some other prior engagement, it is better to decline an invite to be an observer. When you are in a usability session, you should count your time as belonging to the user. If you get up to leave early, you will likely become a distraction. It is also possible that leaving early will create an undue sense of urgency for the user. They may feel like they must hurry up so that they don't take up the usability team's time.

Be completely quiet!
You may not realize it, but you are normally really noisy. Grunting, laughing, body shifting, cell phone sounds, scratches, throat clearing, and other noises can be very distracting or can clue users. It's important to the sanctity of the test that you avoid contaminating the data captured for the study by being completely aware of these noises. 

During the session, you must not talk unless you are asked, otherwise you are likely to cause any test findings to be terminated. The facilitator of the study will be looking for the user to struggle, therefore if you help the user then the purpose of the study has been nullified and everyone's time is wasted.

Don't ask leading questions
If the facilitator asks you if you have questions, be sure not to lead the user in answering the way you want them to. For example, "You probably really liked that feature didn't you?" is a leading question that should not be asked. A better way to find out what the person thought is to ask in this way, "What did you think of that feature? Did you like or dislike it?".

No comments:

Post a Comment

Yes, please comment.