User testing
Useless. Had no concept of what an audience segment was.
User testing is commonly referred to as a method for validating our designs. A colleague of mine would always challenge this, preferring to refer to it as a way to invalidate them instead, which I agree with.
My experience with user testing goes back to when I moverd from client services into the design team team. Our London offices were based in Spitalfields, just back from the market. At that time the newly formed design team wasn't doing any user testing as they were: 1. new and 2. based in Belgium on the outskirts of Brussels.
Fortunately I was able to leverage my relationships with clients such as Tesco, Hearst publishing, TI Media, Games Workshop, and their CSMs from my days in services and would start doing user testing for the features I was working on whenever these clients were in the office. The results were not only beneficial for invalidating design concepts but it became clear that they were an important relationships tool. Not only were clients exposeed to something new (user testing) but they enjoyed it and were happy to be taking part in the shaping of new features in the platform.
During this period I was able to do lots of moderated testing using interactive prorotypes with our clients not just in our office but sometimes on-site.
Then COVID happened.
Usability weeks
Transitioning to fully remote user testing wasn't actually so bad. There were limtiations for sure and looking at someone's head and shoulders just isn't the same as being in person. Yet we persevered and during this time even managed to get Usability weeks up and running. Working with CSMs and the Client Success team we established a database of actual customers who were happy to take part in user tests.
Once a week every month would be dedicated to user testing. Calendars were cleared, meetings a kept to a minimum. Prior to each week we would nominate a topic for testing that we were working on as we would be recruiting client candidates. Based on their profile or persona wew would then match testing topics to them. An hour session would be split into 3 tests. Typically this would be:
- Research questions to derive personas
- 2x project tests using interactive prototypes
On a good week we would be doing up to six user tests per week each between the three design team members. I think my personal record was five back-to-back in one day. I would not recommend this.
Testing paper prototypes over a Zoom call
You've tested with paper prototypes right? It's not exactly something new. But have you tested paper protptypes remotely via the medium of a Zoom call?
I have.
‘How do you test paper prototypes remotely?’ was the question I was asking myself after my boss had tasked me with testing the segment builder project using paper prototypes. At first I thought she was joking, then that I was being punished. I told her as much.
Why? Well, the segment builder project had gotten quite complex at this point using a lot of conditions with a wide variety of data sources. The interface had to be able to handle nested grouping, drag and drop etc. Making all this work in Axure was a nightmare by this point.
Now I love a challenge and I love coding and problem solving. Axure is a great tool and insanely powerful for creating complex interactions that supports variables, data sets and maths functions, the problem is that at a certain level of complexity you would be better off actually coding the damn thing.
Setting-up and testing the actual user tests was taking far too long. This is where paper prototypes comes in.
I had no idea where to start so I grabbed some pens, gridded paper, a ruler and some scissors and started recreating all the necessary interface elements that would be needed for testing.
Source sketches and interface elements. Yeah check those Air Max 90s.
Cut-out interface elements
First pass
OK great, now I had a lovingly reconstructed UI recreated on bits of paper. Now what?
At this point I only had my Macbook, external monitor and a kitchen table and a bag of paper UI elements. I still had no idea how to test it.
I figured I could digitise the paper elements (I know, bear with me) and re-import them back into Axure (stop laughing at the back) and simply dump them into Axure and give the user screen control so they could move everything around. You know, like they were actually there.
I figured this could work, I could retain overall control of the test and add/remove elements etc. and they could then move items around according to how they interpereted that they could carry-out the task. I did a couple of test runs and it all worked OK.
When conducting the first user test we immediately hit problems. What I had failed to realise whilst testing locally was that the Axure file generated by importing lots of unoptimised graphic elements was huge. A bastard. It was over 1GB in size. This wasn't a problem locally running off of a Macbook Pro's flash storage but over a cable DSL network connection the performance was awful, taking forever to load or refresh. It was more-or less unworkable. We skipped that particular task and moved-on.
I was in trouble. The first test was more-or less a write-off and we had at least five more tests to with this project. I had to sort myself before the next test.
But then, a moment of inspiration! Remember the kitchen table? It was glass. The UX equivalent of the spirit of McGyver took me. I had the Zoom app on my phone, this could be used as an external viewing device. By connecting to the Zoom meeting I could place it on the table and aiming it's camera at the floor where I could arrange the UI elements. I improvised using a sheet of brown greaseproof baking sheet taped to the floor as a canvas and would place UI elements on that according to the subject's instructions. Holy shit! I thought, this might just work.
Take two
This was the result below. Masking tape marked-off where the phone should be placed and UI elements were placed on a chair just out of shot. I would join the call on the device and crouch by the table to be the computer, adding and removing elements as neccesary. I did a test-run over Zoom with my colleague to check it would work prior to the actual user test. It did. I was simultaneously happy and also shitting myself.
Set-up for the user test
The first test went like a charm. Everything work as planned from a technical point of view and the test subject clearly enjoyed taking part in this somewhat unusual test. Arming myself with a sharpie enabled my to draw directly onto the baking sheet "canvas" giving an extra degree of flexibility. If the user expected something that wasn't there and that I hadn't prepared I could simply draw it.
Pictured below is the final result of one of the tests.
End result of a user test
I was over the moon with the success of this set-up. All the remaining tests were a success and we got great results I don't think would have been possible using an interactive prototype. Apart from anything else I thing that the stripped-back nature of the set-up and not looking at something digital forced users to use their imagination a bit more and gave us more creative responses. I think it also enable the test to go in directions that wouldn't have been explored by having a prototype with limited functionality.
It was also a lot quicker to set-up but most of all, IMHO, it was fun. It took me back to my child-hood cutting things out for art projects at school. Also spending hours drawing and cutting-out paper UI elements was very theraputic.
I was again to repeat the paper prototyping set-up in a slightly different manner for a subsequent usability week in an even more basic format, using layout paper this time.
So that's how you can do user testing with paper prototypes online.
Set-up for the user test second time