Mobile App Testing with Robot Framework

Abstract

This document is a tutorial on mobile application testing using Robot Framework. The examples are written for Android, but most of them will be applicable to other platforms since Appium and Robot Framework are cross-platform.

Appium

Appium [5] is an mobile automation tool. It’s a user agent. It can be programmed to act on the device and applications on behalf of human users. For example, you could write a test that unlocks the phone, opens an app, taps on a button, scrolls down, reads a text on a label and makes sure that it has an expected value.

It is based on WebDriver, which is the same technology used by Selenium, the web application automation software. However, instead of controlling web browsers and web applications, Appium turns the input into actions on mobile devices and applications.

Appium supports native, mobile web, and hybrid applications on iOS and Android platforms. It’s cross-platform meaning that in theory, you would need to write the test just once and the test would run on all platforms. This is useful when the app UI is consistent across different platforms.

Robot Framework

Robot Framework [1] is a testing framework supporting ATDD. In Robot Framework, each test is written as a table. The format is readable by non-technical people and can serve as a documentation. Each row of the table is an action to take. The first column of the row is called keyword and the following cells are arguments of the keyword. For example,

Test Case Action Resource ID Argument
Login Input Text txt_username johndoe
Input Text txt_password supersecret123
Tap btn_submit

The keywords are provided by a keyword library which is responsible for processing and executing the keywords. Robot Framework has many different keyword libraries. We will use the Appium Library [2] in order to interact with mobile apps through Appium. The library provides keywords such as Tap, Swipe, Input Text, etc. [3] The library takes the keywords and translates them into WebDriver language that is understood by Appium.

Installation

For installation, roughly 3 things need to be installed:

  • Robot Framework
  • Appium Library (the Robot Framework keyword library of our choice)
  • Appium

Robot Framework is written in Python, and the Appium Library, our choice of keyword library, is a Python package. Therefore, Python needs to be installed beforehand. Note that the Appium Library only supports Python 2. Also, we recommend Virtualenv for setting up the environment.

Appium is a Node application, so Node.js needs to be installed beforehand as well.

Virtualenv

As mentioned above, we are forced to use Python 2 due to the Appium Library. Note that Virtualenv is a part of Python 3, but in Python 2, it is a separate package.
Install Virtualenv. The following is a command for installing Virtualenv on macOS with brew:

brew install pyenv-virtualenv

Once Virtualenv is installed, set up a new python environment for Robot Framework as follows.

virtualenv ~/robot

This will create a directory called “robot” in the home directory. Note that the name “robot” is arbitrary. You can pick any name you like for your environment.

One of the files in the created directory is bin/activate (bin/activate.bat for Windows platforms). To use the environment, this file needs to be sourced.

. ~/robot/bin/activate

Notice that your shell prompt changes. It will be prefixed by the name of the environment wrapped in a pair of parentheses. For example,

(robot) ~/Projects/RobotFrameworkTutorial $

Make sure you are inside the environment for the rest of this tutorial.

Robot Framework and Appium Library

Robot Framework is a dependency of the Appium Library, so installing Appium Library will automatically install Robot Framework.

pip install robotframework-appiumlibrary

As of writing, there is a compatibility issue with the latest version of Selenium. The version of Selenium is 3.3.3, and this is not compatible with the latest version of Appium Library. To fix this, we will downgrade Selenium.

pip uninstall selenium
pip install selenium==3.3.1

Appium

Assuming that you already have Node on your system, the following npm command will install Appium.

npm -g install appium

The Appium project also publishes installables. These come with GUI for those who prefer GUI to CLI.

Writing Test Cases

Organization of Test Cases

As previously mentioned, a test case is represented by a table. Each row of the table is an action consisting of a keyword followed by arguments. Keyword and its arguments are written in their own cell.

In terms of the physical format, Robot Framework supports HTML and a custom plain text format. It also supports reStructuredText, but it’s really the plain text format in rst code blocks. In this tutorial, we will use the plain text format.

A group of test cases is called a test suite. Naturally, a test suite is represented by a file. Test suites can also be grouped together in a directory. Groups of test suites can also be grouped into a higher level directory. This grouping can go on as many levels as desired.

Sample App

In the following sections, we will create a test suite with a few test cases. For this, we will use a simple app with two screens as shown in Figure 1. The source code for the app is available from our GitHub repository. [6]

Figure 1. Screens of the sample app. The dummy login screen is presented when the app starts. When the login button is tapped, the app moves to the second screen.

The app has the following requirements, for now.

  1. The login screen should be presented when the app launches.
  2. When the login button is clicked, if both username and password fields are NOT empty, app should move to the second screen.
  3. When the login button is clicked, if one of the username and password fields are empty, app should display a toast read “Login Failed”.
  4. The second screen should display text read “Hello World!”

First Test Case

We will start with the first requirement: the login screen should be presented when the app launches. The test case for this requirement is simple. We will just test that the login button exists on the screen. It would be written as follows more formally.

  • Test case 1: App presents the login page when app launches.
    • Check that the login button exists.

This can be translated into the plain text format of Robot Framework.

App Presents the Login Page When Launches
Page Should Contain Element id=btn_login

The plain text format shown above doesn’t look like a table, but it is actually parsed as a table. The first line is the name of the table. The following lines are rows of the table, and they are left-indented. In the above example, we have only one row, and it has 2 columns. The columns are separated by 2 or more spaces. The first column is the keyword and the following columns are arguments to the keyword. In the example, the keyword is “Page Should Contain Element”. This keyword is defined by the Appium Library that we installed earlier. Its argument is a locator, which locates an element in the UI. The Appium Library supports a few different locator strategies as shown in Table 1. In the example, we chose the id strategy. All in all, this row is an action that checks that there is a view in the current screen whose ID is “btn_login”.

Strategy Example Description
identifier Click Element | identifier=my_element Matches by @id attribute
id Click Element | id=my_element Matches by @resource-id attribute
accessibility_id Click Element | accessibility_id=button3 Accessibility options utilize.
xpath Click Element | xpath=//UIATableView/UIATableCell/UIAButton Matches with arbitrary XPath
class Click Element | class=UIAPickerWheel Matches by class
android Click Element | android=UiSelector().description('Apps') Matches by Android UI Automator
ios Click Element | ios=.buttons().withName('Apps') Matches by iOS UI Automation
css Click Element | css=.green_button Matches by css in webview

Table 1. Locator strategies supported by Appium Library. In the examples, ‘|’ is just a separator between columns allowed by the plain text format.

Test Setup

There is some configuration to set up before you can run test cases. First of all, the Appium Library needs to be loaded so that we can use its keywords. Also, we need to connect to the Appium server process and provide information about the device and the app that are being tested. The Robot Framework file is divided into sections. This kind of setup can go to the Settings section of the file, whereas the test cases go to the Test Cases section.

The following is an example of a settings section.

*** Settings ***
Library  AppiumLibrary  10
Suite Setup  Open Application  http://localhost:4723/wd/hub
             ...  platformName=Android
             ...  deviceName=My Device
             ...  udid=1eb3b4fb
             ...  app=/demo/HelloWorld/app/build/outputs/apk/app-debug.apk
Suite Teardown  Close Application

The section heading is surrounded by a pair of “***” to indicate the beginning of the section.
The first line of the section is for loading the Appium Library. The keyword Library is a built-in keyword of Robot Framework [4]. The first argument is the name of the library and the second argument is the timeout for loading the library.

The second and the third lines are a setup and teardown pair for the test suite. They will be executed before executing the test suite and after the suite is finished respectively. The Suite Setup and Suite Teardown keywords are also built-in [4].

The suite setup line does the most of the setup. As you can see, a long row can span multiple lines. If the second line is a continuation of the previous line, it must begin with an empty cell followed by a cell containing 3 dots (...). The cells following the Suite Setup keyword are an action, namely the Open Application action. Open Application is a keyword from the Appium Library, and it sets up an Appium session and opens the application on the specified device. The first argument to the Open Application keyword is a URL to the Appium server. The one shown in the example is the default URL of Appium when you run it locally. The other parameters are self-explanatory.

For tearing down the test suite, we just close the app and the Appium session using the Close Application keyword of the Appium Library.

Note that the device name and ID are hard-coded in the example above. They can be parameterized as shown below. This is useful if there are more than one devices you need to test. The values for the variables can be supplied from command line at run time.

Suite Setup  Open Application  http://localhost:4723/wd/hub
             ...  platformName=Android
             ...  deviceName=${deviceName}
             ...  udid=${deviceId}
             ...  app=${appBinary}

Running Tests

This section describes how to run the test we’ve developed so far. The source code for the app and the test can be found in our GitHub repository [6]. For this section, you can check out the first_test_case tag of the repository.

The following is a list of steps for running a test.

  1. Build the app. (Check that APK exists.)
  2. Make sure Appium is running.
  3. Connect the target device to the machine where Appium is running.
  4. Run the robot command with the test suite file and configuration variables as arguments.

Starting Up Appium

Running Appium is simple. Just run the appium command in a separate terminal without any arguments, and you can keep it running in the background. Appium needs to know where the Android SDK is installed. Ensure that the ANDROID_HOME environment variable is set. The following is a screenshot of a terminal with Appium has just started up.

Running the Robot Command

The robot command is part of the Robot Framework and it’s responsible for executing the input test files. In this tutorial, we installed Robot Framework using Virtualenv. Therefore, make sure to activate the environment in the current shell before running the robot command. Otherwise, the shell will not be able to find the robot command.

Our test suite has variables for device information and the app binary. The values for the variables can be supplied using the -v command line option of the robot command. For example:

robot -v deviceName:”Samsung Note 3” -v deviceId:1eb3b4fb ...

The name of the test suite file also has to be supplied as a command line argument. The following screenshot shows a full command line and its output.

This generates outputs as shown at the end of the screenshot. The output.xml file records the raw test results. The log.html and report.html files are HTML rendering of the XML, which are easier to read. The following screenshots are the log and report files open in a browser.

Finishing Up Test Cases

Returning to our requirements laid out earlier, we will finish test cases for the remaining requirements.

Requirement 2

The second requirement is repeated below.

  • When the login button is clicked, if both username and password fields are NOT empty, app should move to the second screen.

The test case for the second requirement shown above can be written as follows.

Move To Second Screen When Username And Password Are Not Empty
    Input Text  txt_username  user1
    Input Text  txt_password  secret123
    Tap  btn_login
    Wait Until Page Contains  Hello World!  timeout=2

It’s pretty simple. Populate the username and the password fields and click on the login button. Then, wait up to 2 seconds until the second page shows up.

From the source code git repository [6], check out the second_test_case tag which adds the new test case on top of the previous test suite file.

Requirement 3

The following is the third requirement.

  • When the login button is clicked, if one of the username and password fields are empty, app should display a toast reading “Login Failed”.

This one should be fairly simple too. The only thing to note is that the app will be on the second page after the second test case has run. Therefore, we need to move the app to the first page before continuing on the main test logic.

Show Login Failed If Username Or Password IS Empty
    Go Back
    Clear Text  txt_username
    Clear Text  txt_password
    Tap  btn_login
    Wait Until Page Contains Element  //*[@text=”Login Failed!”]

You can check out the third_test_case tag to see the new test case shown above. Note that our test is not exhaustive. That is, we are not testing the case where only one of the username and the password fields is empty. Try to add other test cases for exercise.

Check out the third_test_case tag and run the test. The test will fail because we didn’t add the functionality yet. This is a good example of how ATDD works. Your tests all fail initially, but gradually the code is added to satisfy the tests.

One thing to note is that we use “Wait Until Page Contains Element” instead of “Wait Until Page Contains” as in the second test case. Wait Until Page Contains Element can also detect toasts (on Android). However, detecting a toast requires UiAutomator2, which can be enabled by adding an additional parameter to the Open Application keyword:

Suite Setup  Open Application  http://localhost:4723/wd/hub
             ...  platformName=Android
             ...  deviceName=${deviceName}
             ...  udid=${deviceId}
             ...  app=${appBinary}
             ...  automationName=uiautomator2

Requirement 4

The last requirement we had was the following.

  • The second screen should display text reading “Hello World!”

Because this is already satisfied by the second test case, there is no need to add more test cases for this.

Conclusion

Robot Framework automates mobile application tests. It lets you write tests in a readable, table format that can also function as requirement documentation. It produces nice reports with screenshots to share among the team members. You can automate the QA process incrementally by adding Robot Framework test cases for existing application. If desired, you can follow the ATDD style to write requirements as Robot Framework test cases.

Internally, Robot Framework utilizes Appium as an automation agent. The Appium Library is an interface between Robot Framework and Appium. On the Robot Framework side, it provides keywords for controlling mobile apps and devices. It converts the keywords into commands understood by Appium.

Reference

[1] Robot Framework. http://robotframework.org/
[2] Appium Library, Robot Framework. https://github.com/serhatbolsu/robotframework-appiumlibrary
[3] Appium Library keyword documentation. http://serhatbolsu.github.io/robotframework-appiumlibrary/AppiumLibrary.html
[4] Robot Framework built-in keyword documentation. http://robotframework.org/robotframework/latest/libraries/BuiltIn.html
[5] Appium. http://appium.io/
[6] Hello World app. https://github.com/Anexinet/RobotFrameworkDemo-Android