 |
|
Observation methods and tips for
usability testing
Sim D'Hertefelt,
12 December 1999
There are two observation methods
for usability testing. "Unobtrusive observation" means
you observe what test users do and refrain from interacting
with them. With unobtrusive observation you learn whether the
system is easy to use. "Obtrusive observation" means
you interact with test users, e.g. by asking questions. With
obtrusive observation you learn more about the usefulness and
acceptance of the system. Besides explaining these observation
methods, this article provides practical observation tips for
each method.
Introduction: what is usability testing?
Usability testing is a technique to
evaluate the ease of use or ease of learning of an interactive
system. During a usability test a real future user uses (a prototype
of) the system while one or more observers look at how this
is done. Usability tests are often task-oriented, i.e. the test
user receives a set of predefined tasks to perform. Usability
tests often use the think aloud technique, i.e. the test user
is asked to say aloud what he thinks while using the system.
With usability testing, you learn whether the design is adapted
to future users and their tasks.
Two observation methods
There are essentially two observation
methods you can use during usability testing. The first method
is "unobtrusive observation": you concentrate on observing
what the test user does and refrain as much as possible from
influencing her/him by explaining the design or asking questions.
This is harder than it sounds and it actually takes some tricks
to do it well.
The second method is "obtrusive
observation", which means you are allowed to explain design
decisions, ask questions, or engage the test user in a discussion.
Neither method is right or wrong.
Each uncovers different things and you will want to use both
methods in one usability test to learn as much as possible about
the usefulness and usability of your design. One way of doing
that is to start each usability test with unobtrusive observation:
you observe how users execute the tasks you give them. After
that, you reserve some time to ask questions, explain design
decisions, and answer the test user's questions.
Unobtrusive observation
With unobtrusive observation you learn
whether people can use your design in an easy and efficient
way, and where this is not the case. How people behave and how
people explain their behavior are two different things. If you
want to learn about behavior, you have to study behavior. If
you want to learn whether users can use your design, you have
to observe how they use it, not ask them what they think of
it.
Opinions are bad predictors of behavior.
During one usability tests we observed that test users were
able to complete only 1 out of 5 basic tasks with the system.
But when we asked them their opinion afterwards, they told us
the system was great.
| |
Tips for unobtrusive
observation
1. Observe: be quiet,
watch, understand.
- Don't explain
- Don't ask the test user's opinion
- Don't defend the design
- Don't apologize
- Don't suggest
- Don't contradict the test user nor agree with him/her:
stay neutral
This is actually very hard.
It seems to be in our nature to help others when they
are in trouble. It is even more difficult to see test
users struggle with the design if you have worked on the
design yourself. Just keep in mind that you're there to
learn about the test user's behavior in relation to your
design, not to convince or teach the user.
2. Only help to overcome
the limitations of the prototype. Explain briefly
and in a neutral way what would happen in the future system.
3. First observe, then
take notes. Don't let your note taking get in
the way of observing what the test user is doing. You
don't have to write down everything you notice during
observation. Instead, take 15' to clean up and complete
your notes after the observation session. But don't postpone
it because you will quickly forget important details.
4. Stimulate users to
think aloud. But use neutral prompts, e.g. “What
do you see, what are you thinking, what do you want to
do, what are you looking for?”
5. Limit the time test
users have to execute a task. Don't prolong the
test user's suffering longer than necessary. If a test
user is really stuck on a task and you have learned why
this is the case, thank the user for trying and ask him/her
to continue. Usability testing can be a frustrating experience
for test users because people seem to be naturally inclined
to do the best they can.
6. Elicit detailed information.
Test User: “I see a lot
of information.”
Observer: “Could you tell me what information you
see?”
7. Answer test user
questions with questions. Dealing with test user
questions is probably the most difficult aspect of unobtrusive
observation. It seems natural and polite to answer questions.
Before each usability test, you should explain the test
user how you will deal with questions. You should encourage
the test user to ask questions because that allows you
to learn what is not clear in the system. Explain that
you will not answer these questions right away but that
you will write them down and answer them at the end of
the test.
A productive way of dealing
with test user questions is to answer them with questions.
If you simply leave the question unanswered, the test
user might feel ignored. His/her motivation to think aloud
and to continue asking questions will diminish or even
vanish. By bouncing the question back you acknowledge
the question and you encourage the test user to elaborate
his/her question, enriching your observation. To minimize
your influence on the test user, use the same words as
the test user.
Test User: “What does this
text mean?”
Observer: “What do you think it means?”
Test User: “Do I have
to click here?”
Observer: “What do you think will happen if you click
there?” |
Obtrusive observation
What unobtrusive methods will not
tell you, is what test users think of your design. Do they like
to use it? Does it answer their needs? You can't observe opinions
by just watching people. Even extreme emotions are difficult
to observe accurately: does my test user get angry because s/he
doesn't understand my design or because s/he thinks it is so
simple it insults her/his intelligence.
If you want to learn more about the
usefulness and the acceptance of your design, you will have
to ask test users.
| |
Tips for obtrusive
observation
1. Think about what
you want to ask before the test. When
you have just finished a test session with unobtrusive
observation, you will have a lot to talk about with the
test user. You can ask the user to clarify or explain
actions you have observed, you can explain and discuss
design decisions, etc.
But you will also need a checklist
of things you want to know from all test users. This checklist
will contain items related to functionality (what is useful
- useless - missing?) and items related to user acceptance
(what do they like - don't like).
2. Ask open questions.
Avoid closed questions,
i.e. questions that can be answered with yes or no. You
will get more detailed and accurate information with open
questions.
Bad: “Do you understand
what this means?”
Good: “What do you think when you see this?”
Bad: "Did you know you
can click here to achieve that?"
Good: "What would you do if you would want to achieve
that?"
3. Don't blame the test
user. Remember:
you're testing the usability of the design, not the computer
literacy of the test user. If the test user does not understand
something, this something will have to be improved in
the design.
Bad: “Why don't you understand
this?”
Good: “Could you tell me what this means for you?”
4. Don't ask the test
user for design solutions. Test users are not
interaction designers. That is why test users will almost
never provide you with good design solutions. Don't bother
asking, but take note of suggestions they make spontaneously.
It is your task to design new interaction solutions on
the basis of your better understanding of user needs.
Bad: “Do you need a News
button here?”
Good: “Which information do you need at this point?”
Instead of asking for design
solutions, it is sometimes useful to ask the test user
to compare the system with related products s/he has used
before. By using interactive systems everybody gains tacit
knowledge of interaction design. But because test users
are not interaction designers, they lack the vocabulary
to make this tacit knowledge explicit. It is much easier
to compare the system with actual experience with a related
product.
Bad: "How can we redesign
this page to make it easier for you?"
Good: "Compared to the product you have used before,
what do you think is better or worse in this system?"
5. You can do obtrusive
observation with groups of test users. People
often find it easier to formulate their opinions when
they are confronted with the opinions of others. That
is why you learn a lot about the usefulness and acceptance
of your design when you ask a group of test users to discuss
your design. Your role is to facilitate and focus the
discussion.
Group discussions are harder
to organize than individual sessions because you need
several test users at the same place at the same time.
They are also harder to manage because you have to control
not only individual reactions but also group processes.
You will want to stimulate some group processes (e.g.
opinion disclosure) but at the same time avoid unwanted
group effects (e.g. group intimidation).
|
Read more about Sim's current activities: Copywriting by Kwintessens | Information architecture by Kwintessens (in Dutch)
|
