Parallel Testing With Robot Framework

Did you ever encounter the problem that your regression test execution time has become so large, that you needed to wait for hours in order to get the results? This is where Pabot comes to the rescue! Pabot enables parallel test execution for your Robot Framework tests. Let’s see how this works!

1. Introduction

This post is the last in a series about Robot Framework, an automation test framework. In case you are not yet familiar with Robot Framework, you can take a look at the previous blogs about this topic:

It is fairly easy to set up a regression test with Robot Framework. You just run the robot command and indicate you want to run all the tests in a particular directory. Robot Framework will execute the tests sequentially one after another and the results will be gathered in one output log and report. What will happen when you have a lot of regression tests? The test execution time will take for hours and we do not want that to happen when we want to be able to deliver changes fast to production. What to do in this case? You can limit the number of tests you run in the regression test, but this is not something you want to do and is only a temporary (bad) solution. You can e.g. split the regression test and run it separately on different machines. That could be a solution, but you will need something to gather the results together. Another option is to run the tests in parallel by means of Pabot. Pabot has been developed by Mikko Korpela, a core contributor to Robot Framework. He mainly has been occupied with RIDE (the Robot Framework IDE) development. Documentation and sources of Pabot can be found at GitHub.

In this post, we are going to explore how Pabot can be used.

Sources are available at GitHub.

2. Application Under Test

We need an application to test. We will use a simple Python script wait.py containing three functions. Each function just waits for a certain amount of time.

import sys
import time

def wait_for_3s():
    time.sleep(3)

def wait_for_5s():
    time.sleep(5)

def wait_for_10s():
    time.sleep(10)

if __name__ == '__main__':
    actions = {'wait_for_3s': wait_for_3s,
               'wait_for_5s': wait_for_5s,
               'wait_for_10s': wait_for_10s}

    action = sys.argv[1]
    args = sys.argv[2:]
    actions[action](*args)

We can invoke the functions by means of the command line:

$ python3 wait.py wait_for_3s

We use a Python script as an application, but this can be any kind of application in any kind of programming language.

3. Our First Pabot Example

3.1 Create the Tests

We also need some tests. We create a separate Test Suite with one Test Case for each function. We create them in the test directory and name them test_wait_for_3s.robot, test_wait_for_5s.robot and test_wait_for_10s.robot. We tag them with the tag basic, we will create more Test Suites and do not want to execute all of them all of the time. By means of tags, we can limit the tests we want to execute.

The test_wait_for_3s.robot contains one Test Case which just invokes the Python command in order to execute the wait_for_3s function. The other *.robot files are similar to this one.

| *** Settings ***   |
| Documentation      | Test the function wait_for_3s of the wait.py Python script
| Library            | OperatingSystem

| *** Variables ***  |
| ${APPLICATION}     | python3 wait.py

| *** Test Cases ***            |                 |
| Wait For 3s                   | [Documentation] | Just call the function wait_for_3s
|                               | [Tags]          | basic
| | ${rc}                       | ${output} =     | Run and Return RC and Output | ${APPLICATION} wait_for_3s
| | Should Be Equal As Integers | ${rc}           | 0

Execute the tests from within the root of the project directory and execute all the tests containing tag basic in directory test:

$ robot --include basic test

All tests pass and when we take a look at the log.html file, we notice that the elapsed time equals 18 seconds. This is as expected.

3.2 Run the Tests Using Pabot

Let’s see what happens when we run the tests with Pabot. First of all, we need to install Pabot.

$ pip install -U robotframework-pabot

We execute the same tests as we did before. Notice that you can use the same options as for the robot command.

$ pabot --include basic test
Storing .pabotsuitenames file
2020-05-23 10:17:03.469249 [PID:10404] [0] [ID:1] EXECUTING Test.Test Wait For 3S
2020-05-23 10:17:03.470496 [PID:10407] [2] [ID:2] EXECUTING Test.Test Wait For 5S
2020-05-23 10:17:03.470529 [PID:10410] [1] [ID:0] EXECUTING Test.Test Wait For 10S
2020-05-23 10:17:06.780431 [PID:10404] [0] [ID:1] PASSED Test.Test Wait For 3S in 3.3 seconds
2020-05-23 10:17:08.790471 [PID:10407] [2] [ID:2] PASSED Test.Test Wait For 5S in 5.3 seconds
2020-05-23 10:17:13.805348 [PID:10410] [1] [ID:0] PASSED Test.Test Wait For 10S in 10.3 seconds
3 critical tests, 3 passed, 0 failed
3 tests total, 3 passed, 0 failed
===================================================
Output: .../MyPabotPlanet/output.xml
Log: .../MyPabotPlanet/log.html
Report: .../MyPabotPlanet/report.html
Total testing: 18.89 seconds
Elapsed time: 10.46 seconds

Some things to notice here:

  • Something is being stored in a file .pabotsuitenames. We will come back to that later on.
  • The tests are executed, each in their own process (unique PID’s).
  • One log and report file are generated.
  • We do not see the test output as we do when running by means of the robot command.
  • The elapsed time is now only 10 seconds instead of 18 seconds.

The log.html file also indicates these statistics. The tests are run in parallel and the elapsed time is determined by the longest running test, 10 seconds in our case.

Some other things to know:

  • We ran the Test Suites in parallel, which is the default behavior. It is also possible to run the Test Cases in parallel. Therefore, you can add the option --testlevelsplit to the command line options. Be aware that you first add the Pabot specific options to the command line and after these the standard robot command line options.
  • A directory pabot_results has been created which contains per test a directory with the output.xml, standard error output and standard output files.
  • When adding the Pabot option --verbose you can view the console output of the tests being executed just like running the robot command.

4. Influence Execution of Tests

When you start running tests in parallel, it might be that some tests need to run in a specific order. It might be that tests are dependent on each other (intended or not intended) or some tests just need to run before other tests for some reason. It is possible to influence the execution order of the tests. This brings us back to the .pabotsuitenames file we talked about earlier. Let’s take a look at the contents of this file:

datasources:3fd963ff61bab32d9113697e3e5a37fb84622e23
commandlineoptions:97d170e1550eee4afc0af065b78cda302a97674c
suitesfrom:no-suites-from-option
file:9314425a8c8b733b8b2316174da893e7ab2a8cee
--suite Test.Basic Example.Test Wait For 10S
--suite Test.Basic Example.Test Wait For 3S
--suite Test.Basic Example.Test Wait For 5S

The first four lines are not of interest for us, but the lines following contain the order of test execution.

The command line option --ordering allows us to influence the test execution. We only need a file following the syntax of the .pabotsuitenames file without the first four lines. We create a file .pabotsuitenames-ordering-wait where we ensure that the 10 seconds test is executed before running any other test. We use the #WAIT keyword for that:

--suite Test.Test Wait For 10S
#WAIT
--suite Test.Test Wait For 3S
--suite Test.Test Wait For 5S

Run the tests again with the predefined file:

$ pabot --ordering .pabotsuitenames-ordering-wait --include basic test
2020-05-24 16:11:30.239398 [PID:6353] [0] [ID:0] EXECUTING Test.Test Wait For 10S
2020-05-24 16:11:40.568166 [PID:6353] [0] [ID:0] PASSED Test.Test Wait For 10S in 10.3 seconds
2020-05-24 16:11:40.667778 [PID:6406] [0] [ID:1] EXECUTING Test.Test Wait For 3S
2020-05-24 16:11:40.668592 [PID:6408] [1] [ID:2] EXECUTING Test.Test Wait For 5S
2020-05-24 16:11:43.978060 [PID:6406] [0] [ID:1] PASSED Test.Test Wait For 3S in 3.3 seconds
2020-05-24 16:11:45.983518 [PID:6408] [1] [ID:2] PASSED Test.Test Wait For 5S in 5.3 seconds
3 critical tests, 3 passed, 0 failed
3 tests total, 3 passed, 0 failed
===================================================
Output:  .../MyPabotPlanet/output.xml
Log:     .../MyPabotPlanet/log.html
Report:  .../MyPabotPlanet/report.html
Total testing: 18.90 seconds
Elapsed time:  15.87 seconds

In the console log, we clearly notice that the 10 seconds test must be ended before running any other test. The elapsed time is about 16 seconds (10 seconds for the first test and approximately 6 seconds for the 3 seconds and 5 seconds test which ran in parallel).

Sometimes it is required that tests are not allowed to run in parallel. We therefore can add a directory to the ordering file. All tests inside that directory will run sequentially. In order to verify this behavior, we add a directory sequentially which contains copies of our three tests. We tag these tests with the sequentially tag. The directory tree of the test directory is the following:

.
├── sequentially
│   ├── test_sequentially_10s.robot
│   ├── test_sequentially_3s.robot
│   └── test_sequentially_5s.robot
├── test_wait_for_10s.robot
├── test_wait_for_3s.robot
└── test_wait_for_5s.robot

We create a file .pabotsuitenames-ordering-dir-sequentially and add this directory to the list of to be executed tests:

--suite Test.Sequentially
--suite Test.Test Wait For 10S
--suite Test.Test Wait For 3S
--suite Test.Test Wait For 5S

Execute the test:

$ pabot --ordering .pabotsuitenames-ordering-dir-sequentially --include basic --include sequentially test
2020-05-24 16:37:56.565453 [PID:7694] [0] [ID:1] EXECUTING Test.Test Wait For 10S
2020-05-24 16:37:56.566232 [PID:7696] [1] [ID:2] EXECUTING Test.Test Wait For 3S
2020-05-24 16:37:56.566397 [PID:7698] [2] [ID:3] EXECUTING Test.Test Wait For 5S
2020-05-24 16:37:56.566463 [PID:7703] [3] [ID:0] EXECUTING Test.Sequentially
2020-05-24 16:37:59.977723 [PID:7696] [1] [ID:2] PASSED Test.Test Wait For 3S in 3.4 seconds
2020-05-24 16:38:01.987965 [PID:7698] [2] [ID:3] PASSED Test.Test Wait For 5S in 5.4 seconds
2020-05-24 16:38:06.897472 [PID:7694] [0] [ID:1] PASSED Test.Test Wait For 10S in 10.3 seconds
2020-05-24 16:38:11.624207 [PID:7703] [3] [ID:0] still running Test.Sequentially after 15.0 seconds
2020-05-24 16:38:15.035548 [PID:7703] [3] [ID:0] PASSED Test.Sequentially in 18.4 seconds
6 critical tests, 6 passed, 0 failed
6 tests total, 6 passed, 0 failed
===================================================
Output: .../MyPabotPlanet/output.xml
Log: .../MyPabotPlanet/log.html
Report: .../MyPabotPlanet/report.html
Total testing: 37.50 seconds
Elapsed time: 18.58 seconds

We notice that all four tests (the three basic tests and the sequentially directory) are executed in parallel. The test in the sequentially directory lasts for 18 seconds and that is exactly the elapsed time of the complete test execution. This can also be seen in the log.html file:

Up till now, all available processes are being used, but this can also be limited. Therefore, use the --processes command line option.

5. The Pabot Library

Running tests in parallel can cause some problems when the same resources are being used by different tests. This can cause unwanted side effects and influence the tests results of the different tests. The Pabot Library contains keywords in order to be able to cope with these kind of problems. We will explore one use case where we will set a lock upon a certain resource.

We copy the contents of the test_wait_for_3s.robot file to files test_acquire_lock1.robot and test_acquire_lock2.robot. We tag them with the lock tag. Assume that the function wait_for_3s contains processing which causes problems when being executed in parallel (e.g. it writes or reads to the same file). We can solve this by using the Acquire Lock and Release Lock keywords from the Pabot library. With Acquire Lock we request a lock on this part of the test and no other test will be able to acquire the lock with an identical name until the lock has been released. Besides that, we also need to import the PabotLib library. We add the same locks to both test files.

| *** Settings ***   |
| Documentation      | Test the function wait_for_3s of the wait.py Python script in combination with a Lock
| Library            | OperatingSystem
| Library            | pabot.PabotLib

| *** Variables ***  |
| ${APPLICATION}     | python3 wait.py

| *** Test Cases ***            |                 |
| Wait For 3s                   | [Documentation] | Just call the function wait_for_3s
|                               | [Tags]          | lock
| | Acquire Lock                | Lock On 3s
| | ${rc}                       | ${output} =     | Run and Return RC and Output | ${APPLICATION} wait_for_3s
| | Should Be Equal As Integers | ${rc}           | 0
| | Release Lock                | Lock On 3s

Run the tests including the --pabotlib command line option.

$ pabot --pabotlib --include lock test
Robot Framework remote server at 127.0.0.1:8270 started.
Storing .pabotsuitenames file
2020-05-30 13:34:00.453909 [PID:6235] [1] [ID:0] EXECUTING Test.Test Acquire Lock1
2020-05-30 13:34:00.454809 [PID:6237] [0] [ID:1] EXECUTING Test.Test Acquire Lock2
2020-05-30 13:34:03.775381 [PID:6235] [1] [ID:0] PASSED Test.Test Acquire Lock1 in 3.3 seconds
2020-05-30 13:34:06.878935 [PID:6237] [0] [ID:1] PASSED Test.Test Acquire Lock2 in 6.4 seconds
2 critical tests, 2 passed, 0 failed
2 tests total, 2 passed, 0 failed
...
Stopping PabotLib process
Robot Framework remote server at 127.0.0.1:8270 stopped.
PabotLib process stopped
Total testing: 9.69 seconds
Elapsed time: 6.79 seconds

We notice that the elapsed time is almost 7 seconds, where we would have an elapsed time of about 3 seconds without any locking. The log.html shows this even more clearly: The Test Acquire Lock2 test lasts for 6 seconds whereas the Test Acquire Lock1 test lasts for 3 seconds.

6. Conclusion

Pabot is a great tool for running your tests in parallel and to speed up the overall test execution. Just like running code in parallel, it can cause some challenges when you want to run your tests in parallel. Luckily, some features are available in order to overcome these challenges. However, beware that you will need to spend some effort in changing your Test Suites in order to be able to make full use of parallel test execution.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.