Wait. Mobile app User Acceptance Testing? Isn’t that a box we have to check so we can hurry up and launch? We’ve gotten this far, why the sudden holdup? We have a great mobile app, we made sure it aligns with our mobile strategy and solves a compelling problem for users. Now we need to run mobile user acceptance testing?
Ahem. The top 4 considerations for mobile app User Acceptance Testing are as follows: Goals, Setup, Execution and Launch.
Define the Goals:
First, define the Goals of your UAT. The standard goal is: does the app meet the requirements? This should be further contextualized: does the app still meet the requirements when used by actual people outside the confines of your development team? In addition, it’s also important to define what UAT is not. It’s not a time to add new features or make changes to the app. Other methods to gain user input during the design phase already exist. These includes focus groups, usability studies, day in the life studies and later, in post-development, user input may be gained through feedback and analytics.
Have Setup in Place:
Second, it’s important to have the Setup in place. Do you have a dedicated UAT environment setup? Does it mirror exactly your Production environment? What about content? Often overlooked when transitioning from a development/staging environment to UAT, business and technical teams should work together to ensure that close to production data (or a copy of production data) is available in UAT so that the testing experience is as realistic as possible.
Next comes user setup. If you’re planning for 20 testers, be sure to ask yourself: are all the credentials setup and associated correctly with content? Also make sure your user list maps each test account ID to the person doing the testing so that when issues are found it’s easy to identify who reported them.
Do your selected mobile devices reflect a representative sample of your real-world audience? Be sure you possess a complete set of the most appropriate iPhones, iPads, Android phones, etc. for your project. How about the participants themselves? Is the room a good configuration? It’s always important to recruit users outside your main team who are unfamiliar with the app. Such users will provide the most realistic feedback and may even uncover bugs you never thought of.
Third is Execution. Both in terms of the logistics of running the UAT itself as well as the process of executing a small development cycle to fix any defects found. Depending on the app’s complexity, we typically recommend UAT occurs over a minimum of two weeks. This allows time for feedback as well as for making any fixes over 1 or 2 short development cycles.
At the start of your initial UAT session, familiarize your testers with the app by walking them through the major paths of the app. Best practice is to project your phone’s screen or share it via a screen-share app. Alternately, connect your device to a laptop and use QuickTime to make the device screen visible on the laptop screen. Be sure to pause and make note of any feedback or defects as they are reported. Otherwise, you risk losing the information (or losing the situation that resulted in the defect). Once all the app’s major paths have been tested, provide another 30 to 60 minutes for additional exploratory testing. This lets testers really dive in and try to use the app in unconventional ways. They may even try to break it, which should be encouraged. Scenarios involving multitasking and switching between apps can often have an undesired affect on the app being tested. This is great UAT feedback because in the real world, users will deal with distractions and the work of their day and your app must be able to accommodate any level of choppy usage. Defects (or areas needing improvement) may be discovered during app switching, including undesired functional behavior. This should all be recorded for future enhancements. It’s also beneficial to test to see how the app performs on cellular versus WiFi (and to try switching back and forth between the two). Also, be sure to test in environments that offer weak connectivity (elevators, for instance) to determine well how the app has been designed to handle the offline (or sporadically offline) user.
Participants should write any defects found in exploratory testing on the board. Once all feedback has been gathered, try this prioritization exercise: First, distinguish functional software defects from feature requests. Then focus on prioritizing the defects themselves. Depending on timeline criticality, typically only the true blockers (major defects that prevent the business process) are addressed during UAT. However, the team can agree where to “draw the line” in terms of priority. Go through each reported item, assign it a High, Medium, or Low priority, then rank order the list. Doing so, a clear picture of what to focus on first should emerge. These must-fix items should be immediately conveyed to the development team to put the final polish on the app. This process should continue to be executed until no blocking defects remain.
The fourth and final consideration is the Launch! Now you’re your development team has resolved all the blocking bugs reported, the app is ready to launch! Now it’s important to determine the appropriate launch path: public App Store (i.e. Google Play) or private Enterprise Distribution method (i.e. Apperian)? Keep in mind, each public App Store has its own set of considerations. This includes the time it takes Apple to approve an app, and the time and effort required to generate marketing assets for the store listing, development certificates, etc. On the other hand, a private store listing should be coordinated with your technical team to enable a launch via your own internal Enterprise App Store. It’s important to fully plan out the transition from UAT to launch and to define your launch communications so that once the app has the green light and gets certified, you are good to go!
With these four considerations: Goals, Setup, Execution, and Launch, in mind — you will be well on your way to a successful and productive UAT and app launch. Feel free to drop me a note if you have any questions or comments about the mobile software development lifecycle, mobile user acceptance testing, or mobile strategy in general.
Mobile Strategist / Mobile Product Manager