Top Ten Off-Shore Usability Testing Tips

One of the user experience community’s goals is to share lessons learned; tips to make our professional lives easier.  Recently we were asked to conduct some usability testing for a company that has offices and offers services throughout the world.  This is the sort of business that is increasingly becoming the norm as markets in countries outside the United States continue to mature and rival those in the United States.  For example an estimate published by the Economist last year predicted China to surpass the United States as the world’s largest economy by 2019 (http://www.economist.com/blogs/dailychart/2010/12/save_date). So when it became time for your client to test their new corporate website, we did phone interviews with participants from Hong Kong, Singapore, Shanghai, Philippines, Columbia, Australia, Mexico, U.K and the U.S.

Today, I want to share some of the top lessons learned for that testing.

Top ten lessons learned from off shore testing.

  1. Today is Actually Tomorrow
  2. Gotta Get Down on Friday
  3. There are different depths to communicating in English
  4. Make a Phone Call, Confirm the level of Communication Clarity
  5. Protect Yourself, Leave time between sessions and overnight
  6. Record. Record. Record.
  7. Create a Checklist
  8. The US is Eurocentric, users struggle to locate countries by map view
  9. Always bring a co-pilot
  10. Country of Origin, include countries in your findings

1. Today is Actually Tomorrow.
It is important to keep in the mind the time difference when attending interviews or scheduling them for other people on our team.  The biggest difference that we witnessed was with Asia Pacific which is about 13 hours ahead of us here in Chicago, IL.  We would perform some calls late in the evening, which turned out to be the following morning for the participants on the call.

2. Gotta Get Down on Friday.
Watch out for Fridays.  Friday afternoon meetings in the US are Saturday morning meetings in Beijing, Hong Kong, Singapore, Tokyo, etc.

So, if you are not using a meeting scheduling application (Outlook), hence tracking your interviews in Excel, you will want to make sure you are aware of the date change. After all, no one appreciates a Saturday morning meeting.

3. There are different depths to communicating in English
Just because you asked the participants in a questionnaire if they understand a language, doesn’t mean they speak it fluently.  In the United States, we take for granted that English is spoken and taught throughout the world.  Trying to understand a participant who speaks English as a second or third language, over a speakerphone is difficult for both people involved in the conversation.  It is difficult for the participant trying to understand your instructions and it is difficult for you, as the receiver, trying to understand their answers.

4. Make a Phone Call, Confirm the level of Communication Clarity
This is an easy problem to alleviate early in the recruiting process. Insist on whomever is setting up the interviews that they make a phone call prior to scheduling a participant.  Ask them to rate participants’ English skills on a 5-point scale and don’t accept anyone who scores lower than a 4.

Even a proper Englishman can be difficult to understand.  Just try to understand a Geordie accent straight from a Manchester United football match where multiple beers were consumed. (http://www.videojug.com/film/how-to-do-a-geordie-accent)

5. Protect Yourself, Leave time between sessions and overnight.
It is important to take an active role in the planning time of your sessions.  Do not blindly leave the planning to another teammate or the client.  You will want at minimum 1 hour between session, in case the previous sessions runs long or to take follow-up notes.

As well as between days, do not schedule a late sessions one night and then an early sessions the following day.  What happened to us is that we had a 9:00 pm call with a participant, only to be followed up with a 5:00 am call the next morning.  That is an 8-hour difference between sessions; once you include commuting and morning shower time it only leaves about 5 hours for sleep.

6. Record. Record. Record.
Recording your sessions is your friend. Every time you conduct an interview record the session.  If the session is in person, ask permission to record the screen with a digital camera recorder.  If it is a call, ask permission to record the conversation.  If you have access to a usability tool such as Morae use it to record the user’s screen.  If you are worried about not getting permission, ask for forgiveness after the fact.

No matter how good a note taker you or a colleague is, you will find it invaluable to go back and listen to the interview a second time.  You will either learn new issues that you missed the first time or you can use the recording to get clarification on a note you previous took.  You can also use the recording to pull direct quotes from your participants and for a highlight reels of your findings.

7. Create a Checklist.
It is eventually going to happen.  You walk through a great sessions with a participant only to discover that you did not start the screen recording.  As Steve Krug recommends in Rocket Surgery Made Easy, a checklist is a great way to assure that your environment is set-up and ready.  Be sure to run through the checklist prior to the start of every session.

8. The US is Eurocentric, users struggle to locate countries by map view.
Our North American history gives us a close tie to Europe.  Our assumption is that everyone knows where Italy is; after all it looks like a boot.  During testing when asked to locate Italy using a map interface, users in the Asia Pacific region struggled.

The lesson learned? Participants tend to know their regional geography and not world geography.  Ask yourself could you locate “Cape Verde” Or “Sri Lanka” On a map?  When then shown a list of countries in an A-Z index, users could quickly find Italy.

9. Always bring a co-pilot.
This is an old standby of usability testing, but when possible have a second colleague with you to scribe notes.  This accomplishes two things. One it helps to capture feedback and notes.  Second it provides a second set of ears and eyes, which will undoubtedly notice flaws in the design that the primary interviewer overlooks.

10. Country of Origin, include countries in your findings.
Your final usability report should include the countries that your different participants resided from.  While it is important both to protect your participants’ identities and that listing their country of origin will help shine some light on any geographic bias that may come out of testing.

The link below provides a power point presentation with notes if you want to use some of this content and brief your team.
Off-Shore Usability Testing Tips

Advertisements
Tagged , , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: