I am starting a wonky mobile behaviours series. This is the first post.

Check out the thumbnails in landscape mode in this Ice Cream Sandwich phone: sore neck guaranteed, courtesy of HTC.

I just came across Vellamo, and Android app from Qualcomm that

evaluates browser performance on Android devices. Vellamo provides a holistic view into browser performance and stability, including networking, JavaScript, rendering, and user experience.

When they say user experience, what they really mean is scrolling performance.

After running the tests, Vellamo gives you the results and compares your phone to others. This is how my HTC Desire HD compares to a whole bunch of other Android devices.

Cool, but...

Vellamo allows you to customise which websites to use when running the test, but not the browser. Since I have more than one browser installed on my Android phone, I guess the app is testing the Android one. It would be pretty cool if you could choose which browser to test.

It also gave me different results for the exact same test.

The first test puts my phone below the Samsung Nexus S.

The second test puts my phone above the Samsung Nexus S.

I have no idea why the difference, since I was testing using the same websites, on the same phone and the same network connection. Still: Vellamo is pretty cool.

Thanks to Antoine RJ Wright for choosing this as the post of the week in the 260th Carnival of the Mobilists. Head there for some of the best in mobile writing on the web.

---

The conference season is upon us

We are going to South by South West and UX Sofia this year, and we have started to prepare our sessions:

• We'll be doing (again) the old DIY Mobile Usability Testing in Austin. We promise it will be the last time, although... didn't we promise that before?
• We'll be turning the session into a workshop for UX Sofia (very exciting and very challenging)
• And there will be a new presentation in Sofia aswell, about design and mobile fragmentation

I will be posting some of the materials that come out of the preparation process. Here comes the first one: some thoughts on applying usability testing to mobile software.

Usability testing of mobile software

Applying usability testing to the study of mobile applications and websites brings considerable challenges. Which phone should we use for testing? Can we use an emulator? How do we prototype for mobile? Can we just recycle the tasks we use for testing desktop software? Do we test in the lab or in the field? How do we record the mobile phone screen, user’s input and facial expressions?

In spite of all these tribulations, test we must. When we are designing desktop software, we can afford to skip usability testing sometimes. We have done it before, we have a good idea of what works and what doesn’t, and we have a set of well established design patterns that have been proven to work across all platforms to assist our design effort.

Mobile could not be more different. Mobile is new, mobile changes fast, mobile is very fragmented, mobile involves a complex and still badly understood context of use, and mobile does not benefit from well established design patterns that work across all platforms and form factors.

Testing with users and the resulting design iteration is the only way to develop useful, usable and beautiful mobile software.

What is usability testing

A process that employs people as testing participants who are representative of the target audience to evaluate the degree to which a product meets specific usability criteria

This is the definition offered by J. Rubin and D. Chisnell. We all now pretty much how it goes: you get a few people who fit the profile of your target audience, you ask them to do some tasks with your software - or with a simulation of that software i.e. a prototype, and then you watch as they attempt to complete the tasks.

We record usability testing sessions. Recordings become a very useful memory aid that complements our notes when required. However, the most important reason why we record tests is related to video as a communication tool. Even the teams most reticent to believe that anybody could have difficulties using their beloved software will cave when watching real people having real trouble. Video constitutes unequivocal evidence of the existence of usability issues.

Useful as this is, I believe the real power of video comes from its ability to generate empathy. For design and development teams, the end user is an abstract entity. By watching usability testing videos, that abstract entity gains a face: it becomes real people, trying to do real things with software, and having real problems when doing so.

The team making the software will remember these people for months. They become a point of reference when making design and development decisions. Team members start saying things like: “Do you see that button, remember that girl that couldn’t find the button, do you think she would find it now? Is it big enough for her? Would it look clickable to her? Is it in the place where she would expect it to be?”

By this “concretisation”, users are brought into the centre of the software-making process. This is something we normally attempt to achieve using personas, although I must confess that where my personas have failed miserably, I have succeeded using videos from usability testing sessions.

What do we record?

When recording usability testing sessions, we aim to capture actions and reactions:

• The user’s actions, mainly in the form of mouse clicks.
• The software or prototype reactions, which happen on the screen
• The user’s reactions. We humans react with our face, and that’s why we record facial expressions

Desktop usability testing vs. mobile usability testing

At first sight, doing usability testing of mobile software is pretty much the same as doing usability testing of desktop software. Only that it really isn’t. As always when you throw mobile into the equation, you are adding a few extra challenges.

Simplifying the whole thing a little bit, those challenges can be reduced to 3 big questions you must answer while planning usability testing with mobile devices:

1. Which phone will you use?
2. Which context will you choose?
3. And which connection?

It’s important to understand that there are not right or wrong answers to these 3 questions. Your response will depend on the nature of the software you are testing, and on what you are trying to achieve with your test.

1. Which phone will you use?

On his Alertbox column of 20 July 2009 Jakob Nielsen shares the results of a usability testing experiment they ran with 48 people who were asked to complete some tasks using several websites on their mobile phones. Nielsen determined the average success rate at 59%. When they broke down that average figure by the type of phone participants used during the test, they found the following:

• Participants using feature phones (something like a Nokia 6300, with a smallish colour screen and a numeric keypad where you cannot install native applications) were able to complete 38% of the tasks.
• Participants using smartphones (something like an oldish Blackberry Bold, with a bigger colour screen and a qwerty keyboard where you can install native applications) were able to complete 55% of the tasks.
• Participants using what Nielsen calls “touch-screen” phones (something like an iPhone, with an even bigger screen than a smartphone incorporating touch technology) were able to complete 75% of the tasks.

The moral to this story is that handset usability affects test results. A wonderfully designed website will feel difficult and cumbersome when used with a phone plagued by usability issues. Not that feature phones are badly designed (some are, some aren’t), but they are probably not optimised for web browsing or application usage. Similarly, not all touch-screen phones are built equal, and some of them will perform better than others.

In any case, we must find a way to minimise the effect of handset (and browser) usability on test results, and here is how:

• Whenever possible, test with participants’ own phones. They might be terrible as handsets go, but participants are probably accustomed to their flaws and have developed tricks and workarounds.
• If this is not possible, and sometimes it won’t be, schedule training time on your test plan to explain participants how to use the phone, and include some warm-up tasks participants can attempt to become familiar with the handset.

2. Which context will you choose?

The question of where to run the tests, in a usability laboratory or in the field, is important in the mobile world. The busy, noisy, distracting mobile context of use couldn’t be more different from the calm, quiet and focused laboratory environment.

Comparative studies about the importance of field testing in mobile software have reached contradictory conclusions. A. Kaikkonen et al. (pdf) tell us:

There was no difference in the number of problems that occurred in the two test settings. Our hypothesis that more problems would be found in the field was not supported.

C.M. Nielsen et al., however, found that

evaluations conducted in field settings can reveal problems not otherwise identified in laboratory evaluations

What both papers agree on is the fact that testing in the field is a messy affair. A. Kaikkonen et al. observe it takes

double the time in comparison to the laboratory

C.M. Nielsen et al. describe it as complex and time-consuming.

The truth is that most industry projects lack the time, budget, personnel and expertise to run tests in the field. For most of us, field testing is not even an option, and we must find consolation in the fact that testing in the lab is better than no testing at all.

If field testing must be done (and for certain types of software, it must: think of data collection applications, geo-location related software or mobile payments), it should be done late in the design cycle as a validation mechanism. It should be carefully planned and rehearsed with pilot runs, and the team should be ready to handle unexpected problems on test day.

3. And which connection?

With extended 3GSM availability in Europe (Ofcom’s 2011 Communications Market Report - pdf - indicates that 95% of the United Kingdom population now lives within 3G coverage), and some operators already testing and rolling out 4th generation networks, it is easy to forget that fast data wireless connections are not accessible to significant portions of the rural population. Only 54% of the people living in Northern Ireland enjoy 3G coverage, according to the same Ofcom report. The conclusion is that we still must test our mobile software over 2G connection speeds as part of our quality assurance process.

When it comes to usability testing, we must remember that the bulk of our software usage will take place over the mobile phone network. In most cases, using wi-fi during usability tests will give participants an unrealistic picture of speed, performance and software responsiveness. Remember: test over the available mobile phone network (unless of course your software is like the BBC iPlayer and only works over wi-fi).

And don’t forget the small detail of covering the data costs your participants might incur as a result of the test.

The final challenge: recording your tests

We have found 4 main approaches to recording usability tests involving mobile devices:

1. Wearable equipment
2. Screen capture applications
3. Document cameras
4. Mounted cameras

1. Wearable equipment

Wearable equipment involves

• cameras recording the mobile phone screen and participant's face
• microphones and headsets to provide instructions or recording comments
• and, unfortunately, belts and backpacks to carry batteries, since the equipment has to be powered somehow.

Wearable equipment allows you to test in the field, but:

• it’s not easy to set up
• it’s intrusive, uncomfortable and a bit heavy

For more details on wearable equipment check this paper from the Helsinki Institute for Information Technology.

2. Screen capture applications

Screen capture applications involve installing software on a computer and on the phone. A connection can then be established between them so that it’s possible to follow on the computer screen what’s happening on the phone screen.

Screen capture applications provide high quality screen recordings, but:

• No application will work with all mobile platforms.
  • Mobiola is available for some versions of Symbian S60 and Blackberry 4.2 or later
  • Display Recorder works on most iOS devices (not on iPhone 3G though), but you must jailbreak them first
  • Ovo Studios offers screen recording software for iOS devices without jailbreaking them
  As far as I know, no screen recording applications exist for Android yet.
• Participants will not appreciate you installing “stuff” on their phones. You might be constrained to run tests on lab phones, instead of using your participants’ own devices.
• As observed in this paper, screen capture applications will not show you the user interaction with the physical device (i.e. the fingers). This means you might miss some errors, like false negatives (undetected touches in touch screen devices).

3. Document cameras

Document cameras stand on a desk and are normally employed to record documents (hence their name).

This approach seems to be popular and has been documented by several people, including Google and Scott Weiss in his book Handheld Usability. However:

• Document cameras are not cheap. An Elmo TT-02RX camera at B&H, a professional audiovisual equipment store in the USA, is priced at about USD 549. Shipping to the UK would bring the total price to USD 842.65 (prices from www.bhphotovideo.com accessed on January 2010). For a more affordable alternative, check Ipevo's cameras.
• Participants must keep within the camera range, normally marked with a square of tape on the desk where the camera stands. This adds cognitive load to the tasks, and deviates participants’ attention from the object of the test.
• The phone must lay on the desk or being hold at a flat angle, far from our natural posture when using mobile phones, which we held one or 2-handed at a 45% degree angle.

4. Mounted devices

Mounted cameras on mobile phones come in 2 flavours: you can buy them ready made, but you can also build them yourself.

Mounted devices allow natural interaction with the phone, but:

• They are not cheap to buy. For example, in his blog 90 percent of everything Harry Brignull mentions a quote for £750 from a company in the UK.
• They are messy to build. For example, in the Little Springs Design blog, Barbara Ballard explains how they must swap out lenses and use a quad-processor that “companies are no longer interested in making”
• If bulky, they can prevent single-handed use of the phone.
• If heavy, they can become tiring for participants.

The perfect recording set up

It seemed to us none of the approaches above ticked all the boxes. In our opinion, these are the characteristics of the perfect recording set up, and how the above approaches comply with them:

Of the 4 approaches, mounted devices, if well implemented, seemed the most promising. Unfortunately, there was nothing we could do about the price of the ready made solutions, so we got to work on a DIY mounted device that was easy to put together and repeatable (i.e. can be easily rebuilt if damaged or lost).

comScore has published some numbers on the evolution of the smartphone market in the UK over the past year.

They include market share percentages for a few smartphone platforms:

• 27.1% of UK smartphones are iPhones
• 26.7% of UK smartphones are Android phones
• 22.5% of UK smartphones are Symbian phones

According to Engadget citing comScore

42 percent of all mobile users in the UK used a smartphone in May of this year (2011)

What does this mean for all phones?

Although I love my smartphone to pieces, if I was trying to choose a platform / channel for my mobile service I would be interested on the mobile market as a whole, and not only on the smartphone chunk.

So I decided to calculate what the comScore figures actually mean for the overall UK mobile market:

• 11.382% of all UK mobile phones are iPhones
• 11.214% of all UK mobile phones are Android smartphones
• 9.45% of all UK mobile phones are Symbian smartphones

So, if you choose a native application as the channel to deliver your mobile service, and you release it for iPhone and Android phones, you are targeting 22.596% of the overall UK mobile phone market.

PS: I am assuming the comScore numbers are smartphone only, and therefore do not include tablets.

Huge disclaimer: I am no statistician, so I sincerely hope I made my calculations right. If you spot a mistake, please let me know!

On February 28th I sent an email to 3 mobile-related discussion groups (Mobile Web, WURFL and Mobile Monday London) about user research activities applied to mobile applications and websites. This is what I wrote:

I am doing some work around user research for mobile apps and websites, and it would be really interesting to know how many of you do some kind of user research as part of your development cycle (usability testing, interviews, etc).

Do you validate designs, prototypes or early builds with prospective users in any way? Do you do user research before starting development to explore different design options? Or maybe you don't do it at all. If you don't, why not? Is it lack of time and money, or are there other reasons? You can answer on or off list. I'll compile the answers (if any) and distribute the outcome.

Thanks!!

7 wonderful people replied, and what follows is a compilation of their responses.

The wonderful respondents

Replies came from the UK, US, India, Australia and New Zealand.

• 1 came from a software company with successful desktop products who have developed mobile versions
• 1 came from a software company with a successful mobile product
• 2 came from software development agencies
• 1 came from a mobile design agency
• 2 came from independent user centered design consultants

The user research techniques

All 7 respondents carry out user research activities during design and development of their mobile software. The most popular activity seems to be usability testing, which was mentioned by 6 out of 7 respondents. Other user engagement activities that came up are:

• focus groups
• field studies
• A/B testing
• surveys
• and interviews, although not everybody is a fan

We tend not to do interviews though, as we have found it to be largely counter productive. As mobile is so new as a medium customers either do not 'get' what we are trying to do, or just recycle experiences that they like from other apps.

Respondents also mentioned internal brainstorming, and peer reviews as a back-up activity whenever project constraints prevented them from involving end users.

I cannot say it happens all the time though. When time is tight, we may 'just' do peer reviews instead.

Usability testing

Assessment, not exploration

Usability testing is mainly used to evaluate possible design solutions:

We don't hold-off development if we have a strong use-case but we bring in users/customers very early-on and make adjustments to our design as needed.

Once we had designs that we were happy with (based on our experience of what does/doesn't work for mobile), it's time to take it to users for validation (...) All in all, this was a fairly tactical exercise, the main priority being to validate the proposed designs.

We test (…) normally part way through the development process when we've got some kind of prototype we can use.

Methodologies widely used for testing desktop software seem to translate well to mobile. Respondents mentioned:

• task-based assessment
• the think aloud protocol
• the System Usability Scale (pdf),
• Rapid Iterative Testing and Evaluation (RITE)
• and Rapid Desirability Testing with the Microsoft product reaction cards.

Prototyping

Only one respondent mentioned paper prototypes: most seem to use interactive ones.

One respondent reported using storyboards for exploration:

For newer ideas, we do storyboards and get feedback.

One respondent mentioned that no-code-required prototyping tools frequently used for desktop software do not work well for mobile.

a quick prototype was built. Well, it wasn't that quick… since couldn't use Axure etc. since not suitable for mobile, so it was necessary to code up the pages (…) it's hard to build a prototype (because you can't use the standard toolkit like Axure/Balsamiq or whatever).

An additional factor impacting prototype build is the importance of animations and transitions in touch interfaces:

Something you didn't mention was around animations and transitions, which can massively influence the experience. We do a lot of work around these too, especially looking at how haptic elements can be included.

Prototypes built by linking static mockups are obviously unsuitable for testing animation frameworks.

Devices

One respondent observed how the introduction of device types makes participant profiling and recruitment more complicated than in desktop software:

Recruitment was really important (…) we wanted to see a range of different devices… some iPhones, some non-iPhone smartphones and some feature phones. (…) It costs more money to recruit.

The other issue related to mobile devices is which one to use during the test itself. Respondents who mentioned the matter agreed on using participants' own phones:

For these interviews, it was important to have participants use their own devices - which removes any bias/difficulty that might be caused by unfamiliarity with the device, or of course, the loss of context if you were to test with an entirely different form-factor (like an emulator on a laptop or PC).

Using participants' own phones is easy to do with web-based functionality, but more complicated when testing resident applications (native or not), which require installation:

If it's something we have public we ask them to bring their own phones and run the test on their own handset. In the case of the iPhone development it's more difficult as we don't want to get into sideloading dev versions of apps onto their phones, so in that case we just use our own handsets for the test, but first ask them to show us how they use [our application] on their handset (to check which version they are on etc - and make sure they really are an iPhone user).

Recording

Capturing video of usability testing sessions is a bit of an issue, so much so that one of the respondents has decided not to do it at all:

We have used Morae in the past also for web site testing, but could not easily use the system for mobile (…) Obviously it's difficult to video something like that so we just sit next to them and observe.

Another respondent seems to have built his own infrastructure for video recording:

You need some equipment to record (…) We got by with a webcam attached to a piece of curved metal which the participants phone can be strapped into and held in-hand.

If you are stuck about how to record your usability tests of mobile software, our DIY kit might help.

Surveys

They deserve their own section, because they are carried out in quite innovative and unorthodox ways. Online and mobile surveys were of course mentioned, but one of the respondents does them on the street too:

One other thing we've found helpful recently is impromptu street surveys - not so much for usability, but for early design and colour feedback, including for icons and logo work. A really quick way to get some input and an impartial 'outside view'. We just look for people in our target age group, ask them if they can help briefly and double check their age, then show them some designs and ask them what they think. Using an iPad is great for this, partly because people want to stop to take a look at the iPad itself.

It is also remarkable the use of remote services initially created for software testing, not for user testing:

We utilize services of Mob4hire users for testing across geography, network condition, etc. I have used Deviceanywhere too.

Is it worth it?

Despite the time, effort and investment required, respondents agreed that involving users in the design process is as worthwhile in mobile software as it is in desktop software:

(Research)'s been very important in new product/platform launches especially. Every time we've done it we've learnt something and caught 2-3 interaction issues that we would not have realised were a problem without testing.

In mobile we are learning that a lot of people have strong opinions on how something should work based on the app or platform (iOS/Android) they are most familiar with. Also, we are surprised (and delighted) at the solutions they suggest based on leveraging the smartphone.

PS: my apologies to all list members for the delay in posting this summary, and my gratitude to the seven people who took the time to share their experiences with us all.

Mobile Monday London came back yesterday with a jam-packed session about tablets. Rumor has it there were people from LG, Blackberry and Google showing non-iPad versions of the thing.

This might sound a bit daft, but the main point of the night was the very bold statement: a tablet is a tablet. It's not a phone, it's not a netbook, it's not a laptop. We therefore must try and figure out what tablets are for, since they must be for something different, right?

The books

David Roth-Ey, Group Digital Director at Harper Collins, had a few interesting things to say about how book publishers are dealing with the digitalisation of their content.

• Book lovers feel guilty about abandoning the physical book. He thinks they'll need a powerful reason, a truly wondrous digital book-reading way if they are to leave the paper incarnation. I have to agree with him: I will not migrate easily.
• There is life beyond the book reader. Harper Collins are experimenting with new ways of delivering content, and apps is one of those ways. Check out the SAS Survival Guide for iOS
• All-you-can-it subscriptions for books are a no-go. When asked about a monthly subscription model for books (pay a monthly fee, read as much as you want), Mr. Roth-Ey replied that's one of his worst nightmares. He explained digital business models must compensate for the value of the content book publishers provide.
• Buy once - read anywhere. For Harper Collins, it's all about accessing the content you buy from any device. Once you pay for the book, it's yours to read from your iPhone, your iPad, your Kindle, your IdeaPad, your Inspiron Duo and whatever next. This poses a few interesting content adaptation and multiple user interface questions: what's the digital equivalent of folding the page?
• The ubiquitous promise of the web. HTML5 vs. native came up of course. Mr. Roth-Ey was quite bold about the whole thing. He explained how fragmentation is not a big problem for publishers, since they can easily piggyback on existing readers for multiple platforms. But their moto is maximum reach, and the web is, from all platforms, the best positioned to deliver it.

The telly

David Gibbs, Director of Mobile Applications and Services at BSkyB, always strikes me as an unusually focused person. These Sky guys know what they are about:

• Consumption. They don't care about people producing their own content. They only care about people consuming Sky's content.
• and TV. They are years ahead of the other TV providers in the UK when it comes to incorporating portable devices into their offering, but their strategy keeps television as the centre of gravity.

For Sky, portable devices have limitations: they cannot offer access to all TV content through phones and tablets due to rights issues. They have however discovered their strengths in 2 areas:

1. Self-service, with the web as the preferred platform to deliver it.
2. Remote control capabilities, with native applications as the preferred platform to deliver them.

Sky's clarity is incredibly rare in the cross-device world, and it's probably paying off handsomely.

The promise

But for the rest of us earthlings, what are tablets for? Someone from the audience confessed himself underwhelmed by what has been offered through tablets so far: clones of smartphone and desktop applications, mere scans of print magazines and newspapers. Sure we can do better, but how?

Harper Collins replied: developers should be working more closely with our writers. Add your readers to the mix, Mr. Roth-Ey, and you might be onto something.

To commemorate the upcoming Global Service Jam a few people at LBi London have organised a series of lunch sessions with external speakers.

I finally managed to catch one yesterday.

The happy marriage of video and ethnographic research

Geke Van Dijk is the soul behind a company called STBY. I think they pronounce it stand by. They specialise on ethnographic research for product and service innovation.

Geke focused on how they are using video for research activities, armed with just a few Flip cameras.

In my head, all the examples Geke shared fit neatly into 2 groups:

1. Video as a research tool
2. Video as a communication tool

1. Video as a research tool

• Self-interviews: participants record themselves answering questions set by the researches. Geke explained how they handed an envelope with questions and a Flip camera to the participants, brought them to a park nearby and set them loose to record their answers. The main advantage of this method is that answers are free from group bias. To me this also seems a good way of running interviews in very little time.
• Video safari: during contextual enquiries, record a journey and the stories that surface. For example, if you are researching water usage in participants' homes, ask them to guide you through the different water dispensing points in the house. Water-usage related stories are likely to surface at each point, and they'll be captured in your recording.
• Video-diaries: use video as the main logging mechanism for diary studies. Geke advised that it's important to provide clear instructions on how to create video diary entries. Otherwise you'll end up with widely different formats, which will make analysis much harder and time-consuming. She gave us an example of such instructions:
  • Step 1: point the video camera outwards and rotate 360 degrees to show us your surroundings.
  • Step 2: point the video camera to yourself and respond to the question we sent you via text message.
  • Step 3: point to a specific thing in your location you'd like to show us.

2. Video as a communication tool

• Video to document personas: each persona is illustrated by video clips from the research that exemplify goals and behaviours.
• Video to immerse design workshop participants: use video fragments from the research to set the tone and drive inspiration during ideation and co-design activities.
• Videos as customer icons: Geke explained that once they release these video clips from research to their clients, they acquire a life of their own. They go around the organisation, they are shared across disciplines and departments, they become "customer icons", a kind of shortcut to end users. I really liked this one.

Managing hand off

I asked Geke how to make sure research insights make it into design. She gave me 2 tricks:

1. Design workshops using research outcomes: they help reinforce research results in designers' heads, and also provide an example of how research insights can be used in design activities.
2. Inverted engagement: I just made up the name, by the way. Basically, designers are lightly involved from the beginning of the research process. Sometimes they attend research activities, but they also help prepare them. Apparently, designers are very good at producing research props. Over time, design involvement increases and research involvement decreases. Once in the design phase, the research team stays lightly involved. Normally they analyse the research material to look for answers to new questions that invariably rise during the design process. I've made a little drawing to illustrate this trick.

This was a truly fascinating talk. Well done to the LBi guys who organised it, and thanks to Geke for sharing her research experience. I am off to buy a Flip camera!

These are the slides for my 10-minute presentation at the UK UPA February event. Well done to the speakers, attendees and specially to the organisers: I found the 10-minute presentation format really pleasant.

This is what my Gmail account told me this morning.

Look at what happens when you click the green window control in the Calculator application (Mac OS X 10.6.5).