Table of Contents
Robot Framework is a Python-based, extensible keyword-driven automation framework for acceptance testing, acceptance test driven development (ATDD), behavior driven development (BDD) and robotic process automation (RPA). It can be used in distributed, heterogeneous environments, where automation requires using different technologies and interfaces.
The framework has a rich ecosystem around it consisting of various generic libraries and tools that are developed as separate projects. For more information about Robot Framework and the ecosystem, see http://robotframework.org.
Robot Framework is open source software released under the Apache License 2.0. Its development is sponsored by the Robot Framework Foundation.
Note
The official RPA support was added in Robot Framework 3.1. This User Guide still talks mainly about creating tests, test data, and test libraries, but same concepts apply also when creating tasks.
Robot Framework is a generic, application and technology independent framework. It has a highly modular architecture illustrated in the diagram below.
The test data is in simple, easy-to-edit tabular format. When Robot Framework is started, it processes the data, executes test cases and generates logs and reports. The core framework does not know anything about the target under test, and the interaction with it is handled by libraries. Libraries can either use application interfaces directly or use lower level test tools as drivers.
Following screenshots show examples of the test data and created reports and logs.
The number one place to find more information about Robot Framework and the rich ecosystem around it is http://robotframework.org. Robot Framework itself is hosted on GitHub.
There are several Robot Framework mailing lists where to ask and search for more information. The mailing list archives are open for everyone (including the search engines) and everyone can also join these lists freely. Only list members can send mails, though, and to prevent spam new users are moderated which means that it might take a little time before your first message goes through. Do not be afraid to send question to mailing lists but remember How To Ask Questions The Smart Way.
Robot Framework is open source software provided under the Apache License 2.0. Robot Framework documentation such as this User Guide use the Creative Commons Attribution 3.0 Unported license. Most libraries and tools in the larger ecosystem around the framework are also open source, but they may use different licenses.
The full Robot Framework copyright notice is included below:
Copyright 2008-2015 Nokia Networks Copyright 2016- Robot Framework Foundation Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
These instructions cover installing Robot Framework and its preconditions on different operating systems. If you already have Python installed, you can install Robot Framework using the standard package manager pip:
pip install robotframework
Robot Framework is implemented using Python, and a precondition to install it is having Python or its alternative implementation PyPy installed. Another recommended precondition is having the pip package manager available.
Robot Framework requires Python 3.8 or newer. The latest version that supports Python 3.6 and 3.7 is Robot Framework 6.1.1. If you need to use Python 2, Jython or IronPython, you can use Robot Framework 4.1.3.
On Linux you should have suitable Python installation with pip available by default. If not, you need to consult your distributions documentation to learn how to install them. This is also true if you want to use some other Python version than the one provided by your distribution by default.
To check what Python version you have installed, you can run python --version
command in a terminal:
$ python --version Python 3.10.13
Notice that if your distribution provides also older Python 2, running python
may use that. To use Python 3, you can use python3
command or even more version
specific command like python3.8
. You need to use these version specific variants
also if you have multiple Python 3 versions installed and need to pinpoint which
one to use:
$ python3.11 --version Python 3.11.7 $ python3.12 --version Python 3.12.1
Installing Robot Framework directly under the system provided Python has a risk that possible problems can affect the whole Python installation used also by the operating system itself. Nowadays Linux distributions typically use user installs by default to avoid such problems, but users can also themselves decide to use virtual environments.
On Windows Python is not available by default, but it is easy to install. The recommended way to install it is using the official Windows installers available at http://python.org. For other alternatives, such as installing from the Microsoft Store, see the official Python documentation.
When installing Python on Windows, it is recommended to add Python to PATH
to make it and tools like pip and Robot Framework easier to execute from
the command line. When using the official installer, you just need
to select the Add Python 3.x to PATH
checkbox on the first dialog.
To make sure Python installation has been successful and Python has been
added to PATH
, you can open the command prompt and execute python --version
:
C:\>python --version Python 3.10.9
If you install multiple Python versions on Windows, the version that is used
when you execute python
is the one first in PATH
. If you need to use others,
the easiest way is using the py launcher:
C:\>py --version Python 3.10.9 C:\>py -3.12 --version Python 3.12.1
MacOS does not provide Python 3 compatible Python version by default, so it needs to be installed separately. The recommended approach is using the official macOS installers available at http://python.org. If you are using a package manager like Homebrew, installing Python via it is possible as well.
You can validate Python installation on macOS using python --version
like on
other operating systems.
PyPy is an alternative Python implementation. Its main advantage over the standard Python implementation is that it can be faster and use less memory, but this depends on the context where and how it is used. If execution speed is important, at least testing PyPy is probably a good idea.
Installing PyPy is a straightforward procedure and you can find both installers
and installation instructions at http://pypy.org. To validate that PyPy installation
was successful, run pypy --version
or pypy3 --version
.
Note
Using Robot Framework with PyPy is officially supported only on Linux.
PATH
The PATH environment variable lists directories where commands executed in
a system are searched from. To make using Python, pip and Robot Framework easier
from the command line, it is recommended to add the Python installation directory
as well as the directory where commands like pip
and robot
are installed
into PATH
.
When using Python on Linux or macOS, Python and tools installed with it should be
automatically in PATH
. If you nevertheless need to update PATH
, you
typically need to edit some system wide or user specific configuration file.
Which file to edit and how depends on the operating system and you need to
consult its documentation for more details.
On Windows the easiest way to make sure PATH
is configured correctly is
setting the Add Python 3.x to PATH
checkbox when running the installer.
To manually modify PATH
on Windows, follow these steps:
Environment Variables
under Settings
. There are variables affecting
the whole system and variables affecting only the current user. Modifying
the former will require admin rights, but modifying the latter is typically
enough.PATH
(often written like Path
) and click Edit
. If you are
editing user variables and PATH
does not exist, click New
instead.PATH
.Ok
to save the changes.These instructions cover installing Robot Framework using pip, the standard Python package manager. If you are using some other package manager like Conda, you can use it instead but need to study its documentation for instructions.
When installing Python, you typically get pip installed automatically. If that is not the case, you need to check the documentation of that Python installation for instructions how to install it separately.
pip
commandTypically you use pip by running the pip
command, but on Linux you may need
to use pip3
or even more Python version specific variant like pip3.8
instead. When running pip
or any of its variants, the pip version that is
found first in PATH will be used. If you have multiple Python versions
installed, you may need to pinpoint which exact version you want to use.
This is typically easiest done by running python -m pip
and substituting
python
with the Python version you want to use.
To make sure you have pip available, you can run pip --version
or equivalent.
Examples on Linux:
$ pip --version pip 23.2.1 from ... (python 3.10) $ python3.12 -m pip --version pip 23.3.1 from ... (python 3.12)
Examples on Windows:
C:\> pip --version pip 23.2.1 from ... (python 3.10) C:\> py -m 3.12 -m pip --version pip 23.3.2 from ... (python 3.12)
In the subsequent sections pip is always run using the pip
command. You may
need to use some of the other approaches explained above in your environment.
The easiest way to use pip is by letting it find and download packages it installs from the Python Package Index (PyPI), but it can also install packages downloaded from the PyPI separately. The most common usages are shown below and pip documentation has more information and examples.
# Install the latest version (does not upgrade) pip install robotframework # Upgrade to the latest stable version pip install --upgrade robotframework # Upgrade to the latest version even if it is a pre-release pip install --upgrade --pre robotframework # Install a specific version pip install robotframework==7.0 # Install separately downloaded package (no network connection needed) pip install robotframework-7.0-py3-none-any.whl # Install latest (possibly unreleased) code directly from GitHub pip install https://github.com/robotframework/robotframework/archive/master.zip # Uninstall pip uninstall robotframework
Another installation alternative is getting Robot Framework source code
and installing it using the provided setup.py
script. This approach is
recommended only if you do not have pip available for some reason.
You can get the source code by downloading a source distribution as a zip package from PyPI and extracting it. An alternative is cloning the GitHub repository and checking out the needed release tag.
Once you have the source code, you can install it with the following command:
python setup.py install
The setup.py
script accepts several arguments allowing, for example,
installation into a non-default location that does not require administrative
rights. It is also used for creating different distribution packages. Run
python setup.py --help
for more details.
To make sure that the correct Robot Framework version has been installed, run the following command:
$ robot --version Robot Framework 7.0 (Python 3.10.3 on linux)
If running these commands fails with a message saying that the command is not found or recognized, a good first step is double-checking the PATH configuration.
If you have installed Robot Framework under multiple Python versions,
running robot
will execute the one first in PATH. To select explicitly,
you can run python -m robot
and substitute python
with the right Python
version.
$ python3.12 -m robot --version Robot Framework 7.0 (Python 3.12.1 on linux) C:\>py -3.11 -m robot --version Robot Framework 7.0 (Python 3.11.7 on win32)
Python virtual environments allow Python packages to be installed in an isolated location for a particular system or application, rather than installing all packages into the same global location. They have two main use cases:
There are several demo projects that introduce Robot Framework and help getting started with it.
This section covers Robot Framework's overall test data syntax. The following sections will explain how to actually create test cases, test suites and so on. Although this section mostly uses term test, the same rules apply also when creating tasks.
The hierarchical structure for arranging test cases is built as follows:
In addition to this, there are:
Test case files, test suite initialization files and resource files are all created using Robot Framework test data syntax. Test libraries and variable files are created using "real" programming languages, most often Python.
Robot Framework data is defined in different sections, often also called tables, listed below:
Section | Used for |
---|---|
Settings | 2) Defining metadata for test suites
and test cases.
|
Variables | Defining variables that can be used elsewhere in the test data. |
Test Cases | Creating test cases from available keywords. |
Tasks | Creating tasks using available keywords. Single file can only contain either tests or tasks. |
Keywords | Creating user keywords from existing lower-level keywords |
Comments | Additional comments or data. Ignored by Robot Framework. |
Different sections are recognized by their header row. The recommended
header format is *** Settings ***
, but the header is case-insensitive,
surrounding spaces are optional, and the number of asterisk characters can
vary as long as there is at least one asterisk in the beginning. For example,
also *settings
would be recognized as a section header.
Robot Framework supports also singular headers like *** Setting ***,
but that
support was deprecated in Robot Framework 6.0. There is a visible deprecation
warning starting from Robot Framework 7.0 and singular headers will eventually
not be supported at all.
The header row can contain also other data than the actual section header. The extra data must be separated from the section header using the data format dependent separator, typically two or more spaces. These extra headers are ignored at parsing time, but they can be used for documenting purposes. This is especially useful when creating test cases using the data-driven style.
Possible data before the first section is ignored.
Note
Section headers can be localized. See the Translations appendix for supported translations.
The most common approach to create Robot Framework data is using the space separated format where pieces of the data, such as keywords and their arguments, are separated from each others with two or more spaces. An alternative is using the pipe separated format where the separator is the pipe character surrounded with spaces (|).
Suite files typically use the .robot extension, but what files are parsed can be configured. Resource files can use the .robot extension as well, but using the dedicated .resource extension is recommended and may be mandated in the future. Files containing non-ASCII characters must be saved using the UTF-8 encoding.
Robot Framework supports also reStructuredText files so that normal Robot Framework data is embedded into code blocks. Only files with the .robot.rst extension are parsed by default. If you would rather use just .rst or .rest extension, that needs to be configured separately.
Robot Framework data can also be created in the JSON format that is targeted more for tool developers than normal Robot Framework users. Only JSON files with the custom .rbt extension are parsed by default.
Earlier Robot Framework versions supported data also in HTML and TSV formats. The TSV format still works if the data is compatible with the space separated format, but the support for the HTML format has been removed altogether. If you encounter such data files, you need to convert them to the plain text format to be able to use them with Robot Framework 3.2 or newer. The easiest way to do that is using the Tidy tool, but you must use the version included with Robot Framework 3.1 because newer versions do not understand the HTML format at all.
When Robot Framework parses data, it first splits the data to lines and then lines to tokens such as keywords and arguments. When using the space separated format, the separator between tokens is two or more spaces or alternatively one or more tab characters. In addition to the normal ASCII space, any Unicode character considered to be a space (e.g. no-break space) works as a separator. The number of spaces used as separator can vary, as long as there are at least two, making it possible to align the data nicely in settings and elsewhere when it makes the data easier to understand.
*** Settings ***
Documentation Example using the space separated format.
Library OperatingSystem
*** Variables ***
${MESSAGE} Hello, world!
*** Test Cases ***
My Test
[Documentation] Example test.
Log ${MESSAGE}
My Keyword ${CURDIR}
Another Test
Should Be Equal ${MESSAGE} Hello, world!
*** Keywords ***
My Keyword
[Arguments] ${path}
Directory Should Exist ${path}
Because tabs and consecutive spaces are considered separators, they must
be escaped if they are needed in keyword arguments or elsewhere
in the actual data. It is possible to use special escape syntax like
\t
for tab and \xA0
for no-break space as well as built-in variables
${SPACE}
and ${EMPTY}
. See the Escaping section for details.
Tip
Although using two spaces as a separator is enough, it is recommended to use four spaces to make the separator easier to recognize.
Note
Prior to Robot Framework 3.2, non-ASCII spaces used in the data were converted to ASCII spaces during parsing. Nowadays all data is preserved as-is.
The biggest problem of the space delimited format is that visually separating keywords from arguments can be tricky. This is a problem especially if keywords take a lot of arguments and/or arguments contain spaces. In such cases the pipe delimited variant can work better because it makes the separator more visible.
One file can contain both space separated and pipe separated lines. Pipe separated lines are recognized by the mandatory leading pipe character, but the pipe at the end of the line is optional. There must always be at least one space or tab on both sides of the pipe except at the beginning and at the end of the line. There is no need to align the pipes, but that often makes the data easier to read.
| *** Settings *** |
| Documentation | Example using the pipe separated format.
| Library | OperatingSystem
| *** Variables *** |
| ${MESSAGE} | Hello, world!
| *** Test Cases *** | | |
| My Test | [Documentation] | Example test. |
| | Log | ${MESSAGE} |
| | My Keyword | ${CURDIR} |
| Another Test | Should Be Equal | ${MESSAGE} | Hello, world!
| *** Keywords *** | | |
| My Keyword | [Arguments] | ${path} |
| | Directory Should Exist | ${path} |
When using the pipe separated format, consecutive spaces or tabs inside arguments do not need to be escaped. Similarly empty columns do not need to be escaped except if they are at the end. Possible pipes surrounded by spaces in the actual test data must be escaped with a backslash, though:
| *** Test Cases *** | | | |
| Escaping Pipe | ${file count} = | Execute Command | ls -1 *.txt \| wc -l |
| | Should Be Equal | ${file count} | 42 |
Note
Preserving consecutive spaces and tabs in arguments is new in Robot Framework 3.2. Prior to it non-ASCII spaces used in the data were also converted to ASCII spaces.
reStructuredText (reST) is an easy-to-read plain text markup syntax that is commonly used for documentation of Python projects, including Python itself as well as this User Guide. reST documents are most often compiled to HTML, but also other output formats are supported. Using reST with Robot Framework allows you to mix richly formatted documents and test data in a concise text format that is easy to work with using simple text editors, diff tools, and source control systems.
Note
Using reStructuredText files with Robot Framework requires the Python docutils module to be installed.
When using Robot Framework with reStructuredText files, normal Robot Framework
data is embedded to so called code blocks. In standard reST code blocks are
marked using the code
directive, but Robot Framework supports also
code-block
or sourcecode
directives used by the Sphinx tool.
reStructuredText example
------------------------
This text is outside code blocks and thus ignored.
.. code:: robotframework
*** Settings ***
Documentation Example using the reStructuredText format.
Library OperatingSystem
*** Variables ***
${MESSAGE} Hello, world!
*** Test Cases ***
My Test
[Documentation] Example test.
Log ${MESSAGE}
My Keyword ${CURDIR}
Another Test
Should Be Equal ${MESSAGE} Hello, world!
Also this text is outside code blocks and ignored. Code blocks not
containing Robot Framework data are ignored as well.
.. code:: robotframework
# Both space and pipe separated formats are supported.
| *** Keywords *** | | |
| My Keyword | [Arguments] | ${path} |
| | Directory Should Exist | ${path} |
.. code:: python
# This code block is ignored.
def example():
print('Hello, world!')
Robot Framework supports reStructuredText files using .robot.rst, .rst and .rest extensions. To avoid parsing unrelated reStructuredText files, only files with the .robot.rst extension are parsed by default when executing a directory. Parsing files with other extensions can be enabled by using either --parseinclude or --extension option.
When Robot Framework parses reStructuredText files, errors below level
SEVERE
are ignored to avoid noise about possible non-standard directives
and other such markup. This may hide also real errors, but they can be seen
when processing files using reStructuredText tooling normally.
Note
Parsing .robot.rst files automatically is new in Robot Framework 6.1.
Robot Framework supports data also in the JSON format. This format is designed more for tool developers than for regular Robot Framework users and it is not meant to be edited manually. Its most important use cases are:
Note
The JSON data support is new in Robot Framework 6.1 and it can be
enhanced in future Robot Framework versions. If you have an enhancement
idea or believe you have encountered a bug, please submit an issue
or start a discussion thread on the #devel
channel on our Slack.
A suite structure can be serialized into JSON by using the TestSuite.to_json method. When used without arguments, it returns JSON data as a string, but it also accepts a path or an open file where to write JSON data along with configuration options related to JSON formatting:
from robot.running import TestSuite
# Create suite based on data on the file system.
suite = TestSuite.from_file_system('/path/to/data')
# Get JSON data as a string.
data = suite.to_json()
# Save JSON data to a file with custom indentation.
suite.to_json('data.rbt', indent=2)
If you would rather work with Python data and then convert that to JSON or some other format yourself, you can use TestSuite.to_dict instead.
A suite can be constructed from JSON data using the TestSuite.from_json method. It works both with JSON strings and paths to JSON files:
from robot.running import TestSuite
# Create suite from JSON data in a file.
suite = TestSuite.from_json('data.rbt')
# Create suite from a JSON string.
suite = TestSuite.from_json('{"name": "Suite", "tests": [{"name": "Test"}]}')
# Execute suite. Notice that log and report needs to be created separately.
suite.run(output='example.xml')
If you have data as a Python dictionary, you can use TestSuite.from_dict instead. Regardless of how a suite is recreated, it exists only in memory and original data files on the file system are not recreated.
As the above example demonstrates, the created suite can be executed using the TestSuite.run method. It may, however, be easier to execute a JSON file directly as explained in the following section.
When executing tests or tasks using the robot
command, JSON files with
the custom .rbt extension are parsed automatically. This includes
running individual JSON files like robot tests.rbt
and running directories
containing .rbt files. If you would rather use the standard
.json extension, you need to configure which files are parsed.
Suite source in the data got from TestSuite.to_json
and TestSuite.to_dict
is in absolute format. If a suite is recreated later on a different machine,
the source may thus not match the directory structure on that machine. To
avoid that, it is possible to use the TestSuite.adjust_source method to
make the suite source relative before getting the data and add a correct root
directory after the suite is recreated:
from robot.running import TestSuite
# Create a suite, adjust source and convert to JSON.
suite = TestSuite.from_file_system('/path/to/data')
suite.adjust_source(relative_to='/path/to')
suite.to_json('data.rbt')
# Recreate suite elsewhere and adjust source accordingly.
suite = TestSuite.from_json('data.rbt')
suite.adjust_source(root='/new/path/to')
Imports, variables and keywords created in suite files are included in the generated JSON along with tests and tasks. The exact JSON structure is documented in the running.json schema file.
When Robot Framework parses the test data files, it ignores:
#
), when it is the first
character of a cell. This means that hash marks can be used to enter
comments in the test data.When Robot Framework ignores some data, this data is not available in any resulting reports and, additionally, most tools used with Robot Framework also ignore them. To add information that is visible in Robot Framework outputs, place it to the documentation or other metadata of test cases or suites, or log it with the BuiltIn keywords Log or Comment.
The escape character in Robot Framework test data is the backslash
(\) and additionally built-in variables ${EMPTY}
and ${SPACE}
can often be used for escaping. Different escaping mechanisms are
discussed in the sections below.
The backslash character can be used to escape special characters so that their literal values are used.
Character | Meaning | Examples |
---|---|---|
\$ |
Dollar sign, never starts a scalar variable. | \${notvar} |
\@ |
At sign, never starts a list variable. | \@{notvar} |
\& |
Ampersand, never starts a dictionary variable. | \&{notvar} |
\% |
Percent sign, never starts an environment variable. | \%{notvar} |
\# |
Hash sign, never starts a comment. | \# not comment |
\= |
Equal sign, never part of named argument syntax. | not\=named |
\| |
Pipe character, not a separator in the pipe separated format. | ls -1 *.txt \| wc -l |
\\ |
Backslash character, never escapes anything. | c:\\temp, \\${var} |
The backslash character also allows creating special escape sequences that are recognized as characters that would otherwise be hard or impossible to create in the test data.
Sequence | Meaning | Examples |
---|---|---|
\n |
Newline character. | first line\n2nd line |
\r |
Carriage return character | text\rmore text |
\t |
Tab character. | text\tmore text |
\xhh |
Character with hex value hh . |
null byte: \x00, ä: \xE4 |
\uhhhh |
Character with hex value hhhh . |
snowman: \u2603 |
\Uhhhhhhhh |
Character with hex value hhhhhhhh . |
love hotel: \U0001f3e9 |
Note
All strings created in the test data, including characters like
\x02
, are Unicode and must be explicitly converted to
byte strings if needed. This can be done, for example, using
Convert To Bytes or Encode String To Bytes keywords
in BuiltIn and String libraries, respectively, or with
something like value.encode('UTF-8')
in Python code.
Note
If invalid hexadecimal values are used with \x
, \u
or \U
escapes, the end result is the original value without
the backslash character. For example, \xAX
(not hex) and
\U00110000
(too large value) result with xAX
and U00110000
, respectively. This behavior may change in
the future, though.
Note
Built-in variable ${\n}
can be used if operating system
dependent line terminator is needed (\r\n
on Windows and
\n
elsewhere).
When using the space separated format, the number of spaces used as
a separator can vary and thus empty values cannot be recognized unless they
are escaped. Empty cells can be escaped either with the backslash character
or with built-in variable ${EMPTY}
. The latter is typically recommended
as it is easier to understand.
*** Test Cases ***
Using backslash
Do Something first arg \
Do Something \ second arg
Using ${EMPTY}
Do Something first arg ${EMPTY}
Do Something ${EMPTY} second arg
When using the pipe separated format, empty values need to be escaped only when they are at the end of the row:
| *** Test Cases *** | | | |
| Using backslash | Do Something | first arg | \ |
| | Do Something | | second arg |
| | | | |
| Using ${EMPTY} | Do Something | first arg | ${EMPTY} |
| | Do Something | | second arg |
Spaces, especially consecutive spaces, as part of arguments for keywords or needed otherwise are problematic for two reasons:
In these cases spaces need to be escaped. Similarly as when escaping empty
values, it is possible to do that either by using the backslash character or
by using the built-in variable ${SPACE}
.
Escaping with backslash | Escaping with ${SPACE} |
Notes |
---|---|---|
\ leading space | ${SPACE}leading space |
|
trailing space \ | trailing space${SPACE} |
Backslash must be after the space. |
\ \ | ${SPACE} |
Backslash needed on both sides. |
consecutive \ \ spaces | consecutive${SPACE * 3}spaces |
Using extended variable syntax. |
As the above examples show, using the ${SPACE}
variable often makes the
test data easier to understand. It is especially handy in combination with
the extended variable syntax when more than one space is needed.
If there is more data than readily fits a row, it is possible to split it
and start continuing rows with ellipsis (...
). Ellipses can be indented
to match the indentation of the starting row and they must always be followed
by the normal test data separator.
In most places split lines have exact same semantics as lines that are not split. Exceptions to this rule are suite, test and keyword documentation as well suite metadata. With them split values are automatically joined together with the newline character to ease creating multiline values.
The ...
syntax allows also splitting variables in the Variable section.
When long scalar variables (e.g. ${STRING}
) are split to multiple rows,
the final value is got by concatenating the rows together. The separator is
a space by default, but that can be changed by starting the value with
SEPARATOR=<sep>
.
Splitting lines is illustrated in the following two examples containing exactly same data without and with splitting.
*** Settings ***
Documentation Here we have documentation for this suite.\nDocumentation is often quite long.\n\nIt can also contain multiple paragraphs.
Default Tags default tag 1 default tag 2 default tag 3 default tag 4 default tag 5
*** Variables ***
${STRING} This is a long string. It has multiple sentences. It does not have newlines.
${MULTILINE} This is a long multiline string.\nThis is the second line.\nThis is the third and the last line.
@{LIST} this list is quite long and items in it can also be long
&{DICT} first=This value is pretty long. second=This value is even longer. It has two sentences.
*** Test Cases ***
Example
[Tags] you probably do not have this many tags in real life
Do X first argument second argument third argument fourth argument fifth argument sixth argument
${var} = Get X first argument passed to this keyword is pretty long second argument passed to this keyword is long too
*** Settings ***
Documentation Here we have documentation for this suite.
... Documentation is often quite long.
...
... It can also contain multiple paragraphs.
Default Tags default tag 1 default tag 2 default tag 3
... default tag 4 default tag 5
*** Variables ***
${STRING} This is a long string.
... It has multiple sentences.
... It does not have newlines.
${MULTILINE} SEPARATOR=\n
... This is a long multiline string.
... This is the second line.
... This is the third and the last line.
@{LIST} this list is quite long and
... items in it can also be long
&{DICT} first=This value is pretty long.
... second=This value is even longer. It has two sentences.
*** Test Cases ***
Example
[Tags] you probably do not have this many
... tags in real life
Do X first argument second argument third argument
... fourth argument fifth argument sixth argument
${var} = Get X
... first argument passed to this keyword is pretty long
... second argument passed to this keyword is long too
Robot Framework localization efforts were started in Robot Framework 6.0 that allowed translation of section headers, settings, Given/When/Then prefixes used in Behavior Driven Development (BDD), and true and false strings used in automatic Boolean argument conversion. The plan is to extend localization support in the future, for example, to log and report and possibly also to control structures.
This section explains how to activate languages, what built-in languages are supported, how to create custom language files and how new translations can be contributed.
The main mechanism to activate languages is specifying them from the command line
using the --language option. When enabling built-in languages,
it is possible to use either the language name like Finnish
or the language
code like fi
. Both names and codes are case and space insensitive and also
the hyphen (-
) is ignored. To enable multiple languages, the
--language option needs to be used multiple times:
robot --language Finnish testit.robot robot --language pt --language ptbr testes.robot
The same --language option is also used when activating custom language files. With them the value can be either a path to the file or, if the file is in the module search path, the module name:
robot --language Custom.py tests.robot robot --language MyLang tests.robot
For backwards compatibility reasons, and to support partial translations, English is always activated automatically. Future versions may allow disabling it.
It is also possible to enable languages directly in data files by having
a line Language: <value>
(case-insensitive) before any of the section
headers. The value after the colon is interpreted the same way as with
the --language option:
Language: Finnish *** Asetukset *** Dokumentaatio Example using Finnish.
If there is a need to enable multiple languages, the Language:
line
can be repeated. These configuration lines cannot be in comments so something like
# Language: Finnish
has no effect.
Due to technical limitations, the per-file language configuration affects also parsing subsequent files as well as the whole execution. This behavior is likely to change in the future and should not be relied upon. If you use per-file configuration, use it with all files or enable languages globally with the --language option.
The following languages are supported out-of-the-box. Click the language name to see the actual translations:
All these translations have been provided by the awesome Robot Framework community. If a language you are interested in is not included, you can consider contributing it!
If a language you would need is not available as a built-in language, or you
want to create a totally custom language for some specific need, you can easily
create a custom language file. Language files are Python files that contain
one or more language definitions that are all loaded when the language file
is taken into use. Language definitions are created by extending the
robot.api.Language
base class and overriding class attributes as needed:
from robot.api import Language
class Example(Language):
test_cases_header = 'Validations'
tags_setting = 'Labels'
given_prefixes = ['Assuming']
true_strings = ['OK', '\N{THUMBS UP SIGN}']
Assuming the above code would be in file example.py, a path to that
file or just the module name example
could be used when the language file
is activated.
The above example adds only some of the possible translations. That is fine
because English is automatically enabled anyway. Most values must be specified
as strings, but BDD prefixes and true/false strings allow more than one value
and must be given as lists. For more examples, see Robot Framework's internal
languages module that contains the Language
class as well as all built-in
language definitions.
If you want to add translation for a new language or enhance existing, head
to Crowdin that we use for collaboration. For more details, see the
separate Localization project, and for questions and free discussion join
the #localization
channel on our Slack.
Robot Framework syntax creates a simple programming language, and similarly as with other languages, it is important to think about the coding style. Robot Framework syntax is pretty flexible on purpose, but there are some generally recommended conventions:
${EXAMPLE}
and local variables
using lower-case letters like ${example}
.One case where there currently is no strong convention is keyword capitalization. Robot Framework itself typically uses title case like Example Keyword in documentation and elsewhere, and this style is often used in Robot Framework data as well. It does not work too well with longer, sentence-like keywords such as Log into system as an admin, though.
Teams and organizations using Robot Framework should have their own coding standards. The community developed Robot Framework Style Guide is an excellent starting point that can be amended as needed. It is also possible to enforce these conventions by using the Robocop linter and the Robotidy code formatter.
This section describes the overall test case syntax. Organizing test cases into test suites using suite files and suite directories is discussed in the next section.
When using Robot Framework for other automation purposes than test automation, it is recommended to create tasks instead of tests. The task syntax is for most parts identical to the test syntax, and the differences are explained in the Creating tasks section.
Test cases are constructed in test case sections from the available keywords. Keywords can be imported from test libraries or resource files, or created in the keyword section of the test case file itself.
The first column in the test case section contains test case names. A test case starts from the row with something in this column and continues to the next test case name or to the end of the section. It is an error to have something between the section headers and the first test.
The second column normally has keyword names. An exception to this rule is setting variables from keyword return values, when the second and possibly also the subsequent columns contain variable names and a keyword name is located after them. In either case, columns after the keyword name contain possible arguments to the specified keyword.
*** Test Cases ***
Valid Login
Open Login Page
Input Username demo
Input Password mode
Submit Credentials
Welcome Page Should Be Open
Setting Variables
Do Something first argument second argument
${value} = Get Some Value
Should Be Equal ${value} Expected value
Note
Although test case names can contain any character, using ?
and
especially *
is not generally recommended because they are
considered to be wildcards when selecting test cases.
For example, trying to run only a test with name Example *
like --test 'Example *'
will actually run any test starting with
Example.
Test cases can also have their own settings. Setting names are always in the second column, where keywords normally are, and their values are in the subsequent columns. Setting names have square brackets around them to distinguish them from keywords. The available settings are listed below and explained later in this section.
Note
Setting names are case-insensitive, but the format used above is
recommended. Settings used to be also space-insensitive, but that was
deprecated in Robot Framework 3.1 and trying to use something like
[T a g s]
causes an error in Robot Framework 3.2. Possible spaces
between brackets and the name (e.g. [ Tags ]
) are still allowed.
Example test case with settings:
*** Test Cases ***
Test With Settings
[Documentation] Another dummy test
[Tags] dummy owner-johndoe
Log Hello, world!
The earlier examples have already demonstrated keywords taking different arguments, and this section discusses this important functionality more thoroughly. How to actually implement user keywords and library keywords with different arguments is discussed in separate sections.
Keywords can accept zero or more arguments, and some arguments may have default values. What arguments a keyword accepts depends on its implementation, and typically the best place to search this information is keyword's documentation. In the examples in this section the documentation is expected to be generated using the Libdoc tool, but the same information is available on documentation generated by generic documentation tools such as pydoc.
Most keywords have a certain number of arguments that must always be
given. In the keyword documentation this is denoted by specifying the
argument names separated with a comma like first, second,
third
. The argument names actually do not matter in this case, except
that they should explain what the argument does, but it is important
to have exactly the same number of arguments as specified in the
documentation. Using too few or too many arguments will result in an
error.
The test below uses keywords Create Directory and Copy
File from the OperatingSystem library. Their arguments are
specified as path
and source, destination
, which means
that they take one and two arguments, respectively. The last keyword,
No Operation from BuiltIn, takes no arguments.
*** Test Cases ***
Example
Create Directory ${TEMPDIR}/stuff
Copy File ${CURDIR}/file.txt ${TEMPDIR}/stuff
No Operation
Arguments often have default values which can either be given or
not. In the documentation the default value is typically separated
from the argument name with an equal sign like name=default
value
. It is possible that all the arguments have default
values, but there cannot be any positional arguments after arguments
with default values.
Using default values is illustrated by the example below that uses
Create File keyword which has arguments path, content=,
encoding=UTF-8
. Trying to use it without any arguments or more than
three arguments would not work.
*** Test Cases ***
Example
Create File ${TEMPDIR}/empty.txt
Create File ${TEMPDIR}/utf-8.txt Hyvä esimerkki
Create File ${TEMPDIR}/iso-8859-1.txt Hyvä esimerkki ISO-8859-1
It is also possible that a keyword accepts any number of arguments.
These so called varargs can be combined with mandatory arguments
and arguments with default values, but they are always given after
them. In the documentation they have an asterisk before the argument
name like *varargs
.
For example, Remove Files and Join Paths keywords from
the OperatingSystem library have arguments *paths
and base, *parts
,
respectively. The former can be used with any number of arguments, but
the latter requires at least one argument.
*** Test Cases ***
Example
Remove Files ${TEMPDIR}/f1.txt ${TEMPDIR}/f2.txt ${TEMPDIR}/f3.txt
@{paths} = Join Paths ${TEMPDIR} f1.txt f2.txt f3.txt f4.txt
The named argument syntax makes using arguments with default values more flexible, and allows explicitly labeling what a certain argument value means. Technically named arguments work exactly like keyword arguments in Python.
It is possible to name an argument given to a keyword by prefixing the value
with the name of the argument like arg=value
. This is especially
useful when multiple arguments have default values, as it is
possible to name only some the arguments and let others use their defaults.
For example, if a keyword accepts arguments arg1=a, arg2=b, arg3=c
,
and it is called with one argument arg3=override
, arguments
arg1
and arg2
get their default values, but arg3
gets value override
. If this sounds complicated, the named arguments
example below hopefully makes it more clear.
The named argument syntax is both case and space sensitive. The former
means that if you have an argument arg
, you must use it like
arg=value
, and neither Arg=value
nor ARG=value
works. The latter means that spaces are not allowed before the =
sign, and possible spaces after it are considered part of the given value.
When the named argument syntax is used with user keywords, the argument
names must be given without the ${}
decoration. For example, user
keyword with arguments ${arg1}=first, ${arg2}=second
must be used
like arg2=override
.
Using normal positional arguments after named arguments like, for example,
| Keyword | arg=value | positional |
, does not work.
The relative order of the named arguments does not matter.
It is possible to use variables in both named argument names and values.
If the value is a single scalar variable, it is passed to the keyword as-is.
This allows using any objects, not only strings, as values also when using
the named argument syntax. For example, calling a keyword like arg=${object}
will pass the variable ${object}
to the keyword without converting it to
a string.
If variables are used in named argument names, variables are resolved before matching them against argument names.
The named argument syntax requires the equal sign to be written literally
in the keyword call. This means that variable alone can never trigger the
named argument syntax, not even if it has a value like foo=bar
. This is
important to remember especially when wrapping keywords into other keywords.
If, for example, a keyword takes a variable number of arguments like
@{args}
and passes all of them to another keyword using the same @{args}
syntax, possible named=arg
syntax used in the calling side is not recognized.
This is illustrated by the example below.
*** Test Cases ***
Example
Run Program shell=True # This will not come as a named argument to Run Process
*** Keywords ***
Run Program
[Arguments] @{args}
Run Process program.py @{args} # Named arguments are not recognized from inside @{args}
If keyword needs to accept and pass forward any named arguments, it must be changed to accept free named arguments. See free named argument examples for a wrapper keyword version that can pass both positional and named arguments forward.
The named argument syntax is used only when the part of the argument
before the equal sign matches one of the keyword's arguments. It is possible
that there is a positional argument with a literal value like foo=quux
,
and also an unrelated argument with name foo
. In this case the argument
foo
either incorrectly gets the value quux
or, more likely,
there is a syntax error.
In these rare cases where there are accidental matches, it is possible to
use the backslash character to escape the syntax like foo\=quux
.
Now the argument will get a literal value foo=quux
. Note that escaping
is not needed if there are no arguments with name foo
, but because it
makes the situation more explicit, it may nevertheless be a good idea.
As already explained, the named argument syntax works with keywords. In addition to that, it also works when importing libraries.
Naming arguments is supported by user keywords and by most test libraries. The only exceptions are Python keywords explicitly using positional-only arguments.
The following example demonstrates using the named arguments syntax with library keywords, user keywords, and when importing the Telnet test library.
*** Settings ***
Library Telnet prompt=$ default_log_level=DEBUG
*** Test Cases ***
Example
Open connection 10.0.0.42 port=${PORT} alias=example
List files options=-lh
List files path=/tmp options=-l
*** Keywords ***
List files
[Arguments] ${path}=. ${options}=
Execute command ls ${options} ${path}
Robot Framework supports free named arguments, often also called free
keyword arguments or kwargs, similarly as Python supports **kwargs.
What this means is that a keyword can receive all arguments that use
the named argument syntax (name=value
) and do not match any arguments
specified in the signature of the keyword.
Free named arguments are supported by same keyword types than normal named
arguments. How keywords specify that they accept free named arguments
depends on the keyword type. For example, Python based keywords simply use
**kwargs
and user keywords use &{kwargs}
.
Free named arguments support variables similarly as named arguments. In practice that means that variables
can be used both in names and values, but the escape sign must always be
visible literally. For example, both foo=${bar}
and ${foo}=${bar}
are
valid, as long as the variables that are used exist. An extra limitation is
that free argument names must always be strings.
As the first example of using free named arguments, let's take a look at
Run Process keyword in the Process library. It has a signature
command, *arguments, **configuration
, which means that it takes the command
to execute (command
), its arguments as variable number of arguments
(*arguments
) and finally optional configuration parameters as free named
arguments (**configuration
). The example below also shows that variables
work with free keyword arguments exactly like when using the named argument
syntax.
*** Test Cases ***
Free Named Arguments
Run Process program.py arg1 arg2 cwd=/home/user
Run Process program.py argument shell=True env=${ENVIRON}
See Free keyword arguments (**kwargs) section under Creating test libraries for more information about using the free named arguments syntax in your custom test libraries.
As the second example, let's create a wrapper user keyword for running the
program.py
in the above example. The wrapper keyword Run Program
accepts all positional and named arguments and passes them forward to
Run Process along with the name of the command to execute.
*** Test Cases ***
Free Named Arguments
Run Program arg1 arg2 cwd=/home/user
Run Program argument shell=True env=${ENVIRON}
*** Keywords ***
Run Program
[Arguments] @{args} &{config}
Run Process program.py @{args} &{config}
Starting from Robot Framework 3.1, keywords can accept argument that must
always be named using the named argument syntax. If, for example,
a keyword would accept a single named-only argument example
, it would
always need to be used like example=value
and using just value
would
not work. This syntax is inspired by the keyword-only arguments
syntax supported by Python 3.
For most parts named-only arguments work the same way as named arguments. The main difference is that libraries implemented with Python 2 using the static library API do not support this syntax.
As an example of using the named-only arguments with user keywords, here
is a variation of the Run Program in the above free named argument
examples that only supports configuring shell
:
*** Test Cases ***
Named-only Arguments
Run Program arg1 arg2 # 'shell' is False (default)
Run Program argument shell=True # 'shell' is True
*** Keywords ***
Run Program
[Arguments] @{args} ${shell}=False
Run Process program.py @{args} shell=${shell}
A totally different approach to specify arguments is embedding them into keyword names. This syntax is supported by both test library keywords and user keywords.
A test case fails if any of the keyword it uses fails. Normally this means that execution of that test case is stopped, possible test teardown is executed, and then execution continues from the next test case. It is also possible to use special continuable failures if stopping test execution is not desired.
The error message assigned to a failed test case is got directly from the failed keyword. Often the error message is created by the keyword itself, but some keywords allow configuring them.
In some circumstances, for example when continuable failures are used, a test case can fail multiple times. In that case the final error message is got by combining the individual errors. Very long error messages are automatically cut from the middle to keep reports easier to read, but full error messages are always visible in log files as messages of the failed keywords.
By default error messages are normal text, but
they can contain HTML formatting. This
is enabled by starting the error message with marker string *HTML*
.
This marker will be removed from the final error message shown in reports
and logs. Using HTML in a custom message is shown in the second example below.
*** Test Cases ***
Normal Error
Fail This is a rather boring example...
HTML Error
${number} = Get Number
Should Be Equal ${number} 42 *HTML* Number is not my <b>MAGIC</b> number.
The test case name comes directly from the Test Case section: it is
exactly what is entered into the test case column. Test cases in one
test suite should have unique names. Pertaining to this, you can also
use the automatic variable ${TEST_NAME}
within the test
itself to refer to the test name. It is available whenever a test is
being executed, including all user keywords, as well as the test setup
and the test teardown.
Starting from Robot Framework 3.2, possible variables in the test case name are resolved so that the final name will contain the variable value. If the variable does not exist, its name is left unchanged.
*** Variables ***
${MAX AMOUNT} ${5000000}
*** Test Cases ***
Amount cannot be larger than ${MAX AMOUNT}
# ...
The [Documentation] setting allows setting free form documentation for a test case. That text is shown in the command line output and in the resulting logs and reports. If documentation gets long, it can be split into multiple rows. It is possible to use simple HTML formatting and variables can be used to make the documentation dynamic. Possible non-existing variables are left unchanged.
*** Test Cases ***
Simple
[Documentation] Simple and short documentation.
No Operation
Multiple lines
[Documentation] First row of the documentation.
...
... Documentation continues here. These rows form
... a paragraph when shown in HTML outputs.
No Operation
Formatting
[Documentation]
... This list has:
... - *bold*
... - _italics_
... - link: http://robotframework.org
No Operation
Variables
[Documentation] Executed at ${HOST} by ${USER}
No Operation
It is important that test cases have clear and descriptive names, and in that case they normally do not need any documentation. If the logic of the test case needs documenting, it is often a sign that keywords in the test case need better names and they are to be enhanced, instead of adding extra documentation. Finally, metadata, such as the environment and user information in the last example above, is often better specified using tags.
Using tags in Robot Framework is a simple, yet powerful mechanism for classifying test cases and also user keywords. Tags are free text and Robot Framework itself has no special meaning for them except for the reserved tags discussed below. Tags can be used at least for the following purposes:
There are multiple ways how to specify tags for test cases explained below:
-tag
syntax.Example:
*** Settings ***
Test Tags requirement: 42 smoke
*** Variables ***
${HOST} 10.0.1.42
*** Test Cases ***
No own tags
[Documentation] Test has tags 'requirement: 42' and 'smoke'.
No Operation
Own tags
[Documentation] Test has tags 'requirement: 42', 'smoke' and 'not ready'.
[Tags] not ready
No Operation
Own tags with variable
[Documentation] Test has tags 'requirement: 42', 'smoke' and 'host: 10.0.1.42'.
[Tags] host: ${HOST}
No Operation
Remove common tag
[Documentation] Test has only tag 'requirement: 42'.
[Tags] -smoke
No Operation
Remove common tag using a pattern
[Documentation] Test has only tag 'smoke'.
[Tags] -requirement: *
No Operation
Set Tags and Remove Tags keywords
[Documentation] This test has tags 'smoke', 'example' and 'another'.
Set Tags example another
Remove Tags requirement: *
As the example shows, tags can be created using variables, but otherwise they preserve the exact name used in the data. When tags are compared, for example, to collect statistics, to select test to be executed, or to remove duplicates, comparisons are case, space and underscore insensitive.
As demonstrated by the above examples, removing tags using -tag
syntax supports
simple patterns like -requirement: *
. Tags starting with a hyphen have no
special meaning otherwise than with the [Tags] setting. If there is
a need to set a tag starting with a hyphen with [Tags], it is possible
to use the escaped format like \-tag
.
Note
The Test Tags setting is new in Robot Framework 6.0. Earlier versions support Force Tags and Default Tags settings discussed in the next section.
Note
The -tag
syntax for removing common tags is new in Robot Framework 7.0.
Prior to Robot Framework 6.0, tags could be specified to tests in the Setting section using two different settings:
Both of these settings still work, but they are considered deprecated. A visible deprecation warning will be added in the future, most likely in Robot Framework 8.0, and eventually these settings will be removed. Tools like Tidy can be used to ease transition.
Updating Force Tags requires only renaming it to Test Tags.
The Default Tags setting will be removed altogether, but the -tag
functionality introduced in Robot Framework 7.0 provides same underlying
functionality. The following examples demonstrate the needed changes.
Old syntax:
*** Settings ***
Force Tags all
Default Tags default
*** Test Cases ***
Common only
[Documentation] Test has tags 'all' and 'default'.
No Operation
No default
[Documentation] Test has only tag 'all'.
[Tags]
No Operation
Own and no default
[Documentation] Test has tags 'all' and 'own'.
[Tags] own
No Operation
New syntax:
*** Settings ***
Test Tags all default
*** Test Cases ***
Common only
[Documentation] Test has tags 'all' and 'default'.
No Operation
No default
[Documentation] Test has only tag 'all'.
[Tags] -default
No Operation
Own and no default
[Documentation] Test has tags 'all' and 'own'.
[Tags] own -default
No Operation
Users are generally free to use whatever tags that work in their context.
There are, however, certain tags that have a predefined meaning for Robot
Framework itself, and using them for other purposes can have unexpected
results. All special tags Robot Framework has and will have in the future
have the robot:
prefix. To avoid problems, users should thus not use any
tag with this prefixes unless actually activating the special functionality.
The current reserved tags are listed below, but more such tags are likely
to be added in the future.
robot:continue-on-failure
and robot:recursive-continue-on-failure
robot:stop-on-failure
and robot:recursive-stop-on-failure
robot:skip-on-failure
robot:skip
robot:exclude
robot:private
robot:no-dry-run
robot:exit
robot:flatten
As of RobotFramework 4.1, reserved tags are suppressed by default in
tag statistics. They will be shown when they are explicitly
included via the --tagstatinclude robot:*
command line option.
Robot Framework has similar test setup and teardown functionality as many other test automation frameworks. In short, a test setup is something that is executed before a test case, and a test teardown is executed after a test case. In Robot Framework setups and teardowns are just normal keywords with possible arguments.
A setup and a teardown are always a single keyword. If they need to take care of multiple separate tasks, it is possible to create higher-level user keywords for that purpose. An alternative solution is executing multiple keywords using the BuiltIn keyword Run Keywords.
The test teardown is special in two ways. First of all, it is executed also when a test case fails, so it can be used for clean-up activities that must be done regardless of the test case status. In addition, all the keywords in the teardown are also executed even if one of them fails. This continue on failure functionality can be used also with normal keywords, but inside teardowns it is on by default.
The easiest way to specify a setup or a teardown for test cases in a
test case file is using the Test Setup and Test
Teardown settings in the Setting section. Individual test cases can
also have their own setup or teardown. They are defined with the
[Setup] or [Teardown] settings in the test case
section and they override possible Test Setup and
Test Teardown settings. Having no keyword after a
[Setup] or [Teardown] setting means having no
setup or teardown. It is also possible to use value NONE
to indicate that
a test has no setup/teardown.
*** Settings ***
Test Setup Open Application App A
Test Teardown Close Application
*** Test Cases ***
Default values
[Documentation] Setup and teardown from setting section
Do Something
Overridden setup
[Documentation] Own setup, teardown from setting section
[Setup] Open Application App B
Do Something
No teardown
[Documentation] Default setup, no teardown at all
Do Something
[Teardown]
No teardown 2
[Documentation] Setup and teardown can be disabled also with special value NONE
Do Something
[Teardown] NONE
Using variables
[Documentation] Setup and teardown specified using variables
[Setup] ${SETUP}
Do Something
[Teardown] ${TEARDOWN}
The name of the keyword to be executed as a setup or a teardown can be a variable. This facilitates having different setups or teardowns in different environments by giving the keyword name as a variable from the command line.
Note
Test suites can have a setup and teardown of their own. A suite setup is executed before any test cases or sub test suites in that test suite, and similarly a suite teardown is executed after them.
Test templates convert normal keyword-driven test cases into data-driven tests. Whereas the body of a keyword-driven test case is constructed from keywords and their possible arguments, test cases with template contain only the arguments for the template keyword. Instead of repeating the same keyword multiple times per test and/or with all tests in a file, it is possible to use it only per test or just once per file.
Template keywords can accept both normal positional and named arguments, as well as arguments embedded to the keyword name. Unlike with other settings, it is not possible to define a template using a variable.
How a keyword accepting normal positional arguments can be used as a template is illustrated by the following example test cases. These two tests are functionally fully identical.
*** Test Cases ***
Normal test case
Example keyword first argument second argument
Templated test case
[Template] Example keyword
first argument second argument
As the example illustrates, it is possible to specify the
template for an individual test case using the [Template]
setting. An alternative approach is using the Test Template
setting in the Setting section, in which case the template is applied
for all test cases in that test case file. The [Template]
setting overrides the possible template set in the Setting section, and
an empty value for [Template] means that the test has no
template even when Test Template is used. It is also possible
to use value NONE
to indicate that a test has no template.
If a templated test case has multiple data rows in its body, the template is applied for all the rows one by one. This means that the same keyword is executed multiple times, once with data on each row. Templated tests are also special so that all the rounds are executed even if one or more of them fails. It is possible to use this kind of continue on failure mode with normal tests too, but with the templated tests the mode is on automatically.
*** Settings ***
Test Template Example keyword
*** Test Cases ***
Templated test case
first round 1 first round 2
second round 1 second round 2
third round 1 third round 2
Using keywords with default values or accepting variable number of arguments, as well as using named arguments and free named arguments, work with templates exactly like they work otherwise. Using variables in arguments is also supported normally.
Templates support a variation of the embedded argument syntax. With templates this syntax works so that if the template keyword has variables in its name, they are considered placeholders for arguments and replaced with the actual arguments used with the template. The resulting keyword is then used without positional arguments. This is best illustrated with an example:
*** Test Cases ***
Normal test case with embedded arguments
The result of 1 + 1 should be 2
The result of 1 + 2 should be 3
Template with embedded arguments
[Template] The result of ${calculation} should be ${expected}
1 + 1 2
1 + 2 3
*** Keywords ***
The result of ${calculation} should be ${expected}
${result} = Calculate ${calculation}
Should Be Equal ${result} ${expected}
When embedded arguments are used with templates, the number of arguments in the template keyword name must match the number of arguments it is used with. The argument names do not need to match the arguments of the original keyword, though, and it is also possible to use different arguments altogether:
*** Test Cases ***
Different argument names
[Template] The result of ${foo} should be ${bar}
1 + 1 2
1 + 2 3
Only some arguments
[Template] The result of ${calculation} should be 3
1 + 2
4 - 1
New arguments
[Template] The ${meaning} of ${life} should be 42
result 21 * 2
The main benefit of using embedded arguments with templates is that argument names are specified explicitly. When using normal arguments, the same effect can be achieved by naming the columns that contain arguments. This is illustrated by the data-driven style example in the next section.
FOR
loopsIf templates are used with FOR loops, the template is applied for all the steps inside the loop. The continue on failure mode is in use also in this case, which means that all the steps are executed with all the looped elements even if there are failures.
*** Test Cases ***
Template with FOR loop
[Template] Example keyword
FOR ${item} IN @{ITEMS}
${item} 2nd arg
END
FOR ${index} IN RANGE 42
1st arg ${index}
END
IF/ELSE
structuresIF/ELSE structures can be also used together with templates. This can be useful, for example, when used together with FOR loops to filter executed arguments.
*** Test Cases ***
Template with FOR and IF
[Template] Example keyword
FOR ${item} IN @{ITEMS}
IF ${item} < 5
${item} 2nd arg
END
END
There are several different ways in which test cases may be written. Test cases that describe some kind of workflow may be written either in keyword-driven or behavior-driven style. Data-driven style can be used to test the same workflow with varying input data.
Workflow tests, such as the Valid Login test described earlier, are constructed from several keywords and their possible arguments. Their normal structure is that first the system is taken into the initial state (Open Login Page in the Valid Login example), then something is done to the system (Input Name, Input Password, Submit Credentials), and finally it is verified that the system behaved as expected (Welcome Page Should Be Open).
Another style to write test cases is the data-driven approach where test cases use only one higher-level keyword, often created as a user keyword, that hides the actual test workflow. These tests are very useful when there is a need to test the same scenario with different input and/or output data. It would be possible to repeat the same keyword with every test, but the test template functionality allows specifying the keyword to use only once.
*** Settings ***
Test Template Login with invalid credentials should fail
*** Test Cases *** USERNAME PASSWORD
Invalid User Name invalid ${VALID PASSWORD}
Invalid Password ${VALID USER} invalid
Invalid User Name and Password invalid invalid
Empty User Name ${EMPTY} ${VALID PASSWORD}
Empty Password ${VALID USER} ${EMPTY}
Empty User Name and Password ${EMPTY} ${EMPTY}
Tip
Naming columns like in the example above makes tests easier to understand. This is possible because on the header row other cells except the first one are ignored.
The above example has six separate tests, one for each invalid user/password combination, and the example below illustrates how to have only one test with all the combinations. When using test templates, all the rounds in a test are executed even if there are failures, so there is no real functional difference between these two styles. In the above example separate combinations are named so it is easier to see what they test, but having potentially large number of these tests may mess-up statistics. Which style to use depends on the context and personal preferences.
*** Test Cases ***
Invalid Password
[Template] Login with invalid credentials should fail
invalid ${VALID PASSWORD}
${VALID USER} invalid
invalid whatever
${EMPTY} ${VALID PASSWORD}
${VALID USER} ${EMPTY}
${EMPTY} ${EMPTY}
It is also possible to write test cases as requirements that also non-technical project stakeholders must understand. These executable requirements are a corner stone of a process commonly called Acceptance Test Driven Development (ATDD) or Specification by Example.
One way to write these requirements/tests is Given-When-Then style popularized by Behavior Driven Development (BDD). When writing test cases in this style, the initial state is usually expressed with a keyword starting with word Given, the actions are described with keyword starting with When and the expectations with a keyword starting with Then. Keyword starting with And or But may be used if a step has more than one action.
*** Test Cases ***
Valid Login
Given login page is open
When valid username and password are inserted
and credentials are submitted
Then welcome page should be open
Prefixes Given, When, Then, And and But can be omitted when creating keywords. For example, Given login page is open in the above example is typically implemented without the word Given so that the name is just Login page is open. Omitting prefixes allows using the same keyword with different prefixes. For example, Welcome page should be open could be used as Then welcome page should be open or and welcome page should be open.
Note
These prefixes can be localized. See the Translations appendix for supported translations.
When writing concrete examples it is useful to be able to pass actual data to keyword implementations. This can be done by embedding arguments into keyword name.
In addition to test automation, Robot Framework can be used for other automation purposes, including robotic process automation (RPA). It has always been possible, but Robot Framework 3.1 added official support for automating tasks, not only tests. For most parts creating tasks works the same way as creating tests and the only real difference is in terminology. Tasks can also be organized into suites exactly like test cases.
Tasks are created based on the available keywords exactly like test cases, and the task syntax is in general identical to the test case syntax. The main difference is that tasks are created in Task sections instead of Test Case sections:
*** Tasks ***
Process invoice
Read information from PDF
Validate information
Submit information to backend system
Validate information is visible in web UI
It is an error to have both tests and tasks in same file.
Robot Framework test cases are created in test case files, which can be organized into directories. These files and directories create a hierarchical test suite structure. Same concepts apply also when creating tasks, but the terminology differs.
Robot Framework test cases are created using test case sections in suite files, also known as test case files. Such a file automatically creates a test suite from all the test cases it contains. There is no upper limit for how many test cases there can be, but it is recommended to have less than ten, unless the data-driven approach is used, where one test case consists of only one high-level keyword.
The following settings in the Setting section can be used to customize the suite:
Note
Setting names are case-insensitive, but the format used above is recommended.
Test case files can be organized into directories, and these directories create higher-level test suites. A test suite created from a directory cannot have any test cases directly, but it contains other test suites with test cases, instead. These directories can then be placed into other directories creating an even higher-level suite. There are no limits for the structure, so test cases can be organized as needed.
When a test directory is executed, the files and directories it contains are processed recursively as follows:
If a file or directory that is processed does not contain any test cases, it is silently ignored (a message is written to the syslog) and the processing continues.
A test suite created from a directory can have similar settings as a suite created from a test case file. Because a directory alone cannot have that kind of information, it must be placed into a special test suite initialization file. An initialization file name must always be of the format __init__.ext, where the extension must be one of the supported file formats (typically __init__.robot). The name format is borrowed from Python, where files named in this manner denote that a directory is a module.
Starting from Robot Framework 6.1, it is also possible to define a suite initialization file for automatically created suite when starting the test execution by giving multiple paths.
Initialization files have the same structure and syntax as test case files, except that they cannot have test case sections and not all settings are supported. Variables and keywords created or imported in initialization files are not available in the lower level test suites. If you need to share variables or keywords, you can put them into resource files that can be imported both by initialization and test case files.
The main usage for initialization files is specifying test suite related settings similarly as in suite files, but setting some test case related settings is also possible. How to use different settings in the initialization files is explained below.
*** Settings ***
Documentation Example suite
Suite Setup Do Something ${MESSAGE}
Test Tags example
Library SomeLibrary
*** Variables ***
${MESSAGE} Hello, world!
*** Keywords ***
Do Something
[Arguments] ${args}
Some Keyword ${arg}
Another Keyword
The test suite name is constructed from the file or directory name by default. The name is created so that the extension is ignored, possible underscores are replaced with spaces, and names fully in lower case are title cased. For example, some_tests.robot becomes Some Tests and My_test_directory becomes My test directory.
The file or directory name can contain a prefix to control the execution order of the suites. The prefix is separated from the base name by two underscores and, when constructing the actual test suite name, both the prefix and underscores are removed. For example files 01__some_tests.robot and 02__more_tests.robot create test suites Some Tests and More Tests, respectively, and the former is executed before the latter.
Starting from Robot Framework 6.1, it is also possible to give a custom name to a suite by using the Name setting in the Setting section:
*** Settings ***
Name Custom suite name
The name of the top-level suite can be overridden from the command line with the --name option.
The documentation for a test suite is set using the Documentation setting in the Settings section. It can be used both in suite files and in suite initialization files. Suite documentation has exactly the same characteristics regarding to where it is shown and how it can be created as test case documentation. For details about the syntax see the Documentation formatting appendix.
*** Settings ***
Documentation An example suite documentation with *some* _formatting_.
... Long documentation can be split into multiple lines.
The documentation of the top-level suite can be overridden from the command line with the --doc option.
In addition to documentation, suites can also have free metadata. This metadata is defined as name-value pairs in the Settings section using the Metadata setting. It is shown in reports and logs similarly as documentation.
Name of the metadata is the first argument given to the Metadata setting and the remaining arguments specify its value. The value is handled similarly as documentation, which means that it supports HTML formatting and variables, and that longer values can be split into multiple rows.
*** Settings ***
Metadata Version 2.0
Metadata Robot Framework http://robotframework.org
Metadata Platform ${PLATFORM}
Metadata Longer Value
... Longer metadata values can be split into multiple
... rows. Also *simple* _formatting_ is supported.
The free metadata of the top-level suite can be set from the command line with the --metadata option.
Not only test cases but also test suites can have a setup and a teardown. A suite setup is executed before running any of the suite's test cases or child test suites, and a suite teardown is executed after them. All test suites can have a setup and a teardown; with suites created from a directory they must be specified in a suite initialization file.
Similarly as with test cases, a suite setup and teardown are keywords that may take arguments. They are defined in the Setting section with Suite Setup and Suite Teardown settings, respectively. Keyword names and possible arguments are located in the columns after the setting name.
If a suite setup fails, all test cases in it and its child test suites are immediately assigned a fail status and they are not actually executed. This makes suite setups ideal for checking preconditions that must be met before running test cases is possible.
A suite teardown is normally used for cleaning up after all the test cases have been executed. It is executed even if the setup of the same suite fails. If the suite teardown fails, all test cases in the suite are marked failed, regardless of their original execution status. Note that all the keywords in suite teardowns are executed even if one of them fails.
The name of the keyword to be executed as a setup or a teardown can be a variable. This facilitates having different setups or teardowns in different environments by giving the keyword name as a variable from the command line.
Test libraries contain those lowest-level keywords, often called library keywords, which actually interact with the system under test. All test cases always use keywords from some library, often through higher-level user keywords. This section explains how to take test libraries into use and how to use the keywords they provide. Creating test libraries is described in a separate section.
Test libraries are typically imported using the Library setting, but it is also possible to use the Import Library keyword.
Library
settingTest libraries are normally imported using the Library setting in the Setting section and having the library name in the subsequent column. Unlike most of the other data, the library name is both case- and space-sensitive. If a library is in a package, the full name including the package name must be used.
In those cases where the library needs arguments, they are listed in the columns after the library name. It is possible to use default values, variable number of arguments, and named arguments in test library imports similarly as with arguments to keywords. Both the library name and arguments can be set using variables.
*** Settings ***
Library OperatingSystem
Library my.package.TestLibrary
Library MyLibrary arg1 arg2
Library ${LIBRARY}
It is possible to import test libraries in suite files, resource files and suite initialization files. In all these cases, all the keywords in the imported library are available in that file. With resource files, those keywords are also available in other files using them.
Import Library
keywordAnother possibility to take a test library into use is using the keyword Import Library from the BuiltIn library. This keyword takes the library name and possible arguments similarly as the Library setting. Keywords from the imported library are available in the test suite where the Import Library keyword was used. This approach is useful in cases where the library is not available when the test execution starts and only some other keywords make it available.
*** Test Cases ***
Example
Do Something
Import Library MyLibrary arg1 arg2
KW From MyLibrary
Libraries to import can be specified either by using the library name or the path to the library. These approaches work the same way regardless if the library is imported using the Library setting or the Import Library keyword.
The most common way to specify a test library to import is using its name, like it has been done in all the examples in this section. In these cases Robot Framework tries to find the class or module implementing the library from the module search path. Libraries that are installed somehow ought to be in the module search path automatically, but with other libraries the search path may need to be configured separately.
The biggest benefit of this approach is that when the module search path has been configured, often using a custom start-up script, normal users do not need to think where libraries actually are installed. The drawback is that getting your own, possible very simple, libraries into the search path may require some additional configuration.
Another mechanism for specifying the library to import is using a path to it in the file system. This path is considered relative to the directory where current test data file is situated similarly as paths to resource and variable files. The main benefit of this approach is that there is no need to configure the module search path.
If the library is a file, the path to it must contain extension,
i.e. .py. If a library is implemented
as a directory, the path to it must have a trailing forward slash (/
)
if the path is relative. With absolute paths the trailing slash is optional.
Following examples demonstrate these different usages.
*** Settings ***
Library PythonLibrary.py
Library relative/path/PythonDirLib/ possible arguments
Library ${RESOURCES}/Example.class
A limitation of this approach is that libraries implemented as Python classes must be in a module with the same name as the class.
The library name is shown in test logs before keyword names, and if multiple keywords have the same name, they must be used so that the keyword name is prefixed with the library name. The library name is got normally from the module or class name implementing it, but there are some situations where changing it is desirable:
The basic syntax for specifying the new name is having the text
AS
(case-sensitive) after the library name and then
having the new name after that. The specified name is shown in
logs and must be used in the test data when using keywords' full name
(LibraryName.Keyword Name).
*** Settings ***
Library packagename.TestLib AS TestLib
Library ${LIBRARY} AS MyName
Possible arguments to the library are placed between the
original library name and the AS
marker. The following example
illustrates how the same library can be imported several times with
different arguments:
*** Settings ***
Library SomeLibrary localhost 1234 AS LocalLib
Library SomeLibrary server.domain 8080 AS RemoteLib
*** Test Cases ***
Example
LocalLib.Some Keyword some arg second arg
RemoteLib.Some Keyword another arg whatever
LocalLib.Another Keyword
Setting a custom name to a test library works both when importing a library in the Setting section and when using the Import Library keyword.
Note
Prior to Robot Framework 6.0 the marker to use when giving a custom name
to a library was WITH NAME
instead of AS
. The old syntax continues
to work, but it is considered deprecated and will eventually be removed.
Some test libraries are distributed with Robot Framework and these libraries are called standard libraries. The BuiltIn library is special, because it is taken into use automatically and thus its keywords are always available. Other standard libraries need to be imported in the same way as any other libraries, but there is no need to install them.
The available normal standard libraries are listed below with links to their documentations:
In addition to the normal standard libraries listed above, there is also Remote library that is totally different than the other standard libraries. It does not have any keywords of its own but it works as a proxy between Robot Framework and actual test library implementations. These libraries can be running on other machines than the core framework and can even be implemented using languages not supported by Robot Framework natively.
See separate Remote library interface section for more information about this concept.
Any test library that is not one of the standard libraries is, by definition, an external library. The Robot Framework open source community has implemented several generic libraries, such as SeleniumLibrary and SwingLibrary, which are not packaged with the core framework. A list of publicly available libraries can be found from http://robotframework.org.
Generic and custom libraries can obviously also be implemented by teams using Robot Framework. See Creating test libraries section for more information about that topic.
Different external libraries can have a totally different mechanism for installing them and taking them into use. Sometimes they may also require some other dependencies to be installed separately. All libraries should have clear installation and usage documentation and they should preferably automate the installation process.
Variables are an integral feature of Robot Framework, and they can be used in most places in test data. Most commonly, they are used in arguments for keywords in Test Case and Keyword sections, but also all settings allow variables in their values. A normal keyword name cannot be specified with a variable, but the BuiltIn keyword Run Keyword can be used to get the same effect.
Robot Framework has its own variables that can be used as scalars, lists
or dictionaries using syntax ${SCALAR}
, @{LIST}
and &{DICT}
,
respectively. In addition to this, environment variables can be used
directly with syntax %{ENV_VAR}
.
Variables are useful, for example, in these cases:
${RESOURCES}
instead of c:\resources
, or ${HOST}
instead of 10.0.0.1:8080
). Because variables can be set from the
command line when tests are started, changing system-specific
variables is easy (for example, --variable HOST:10.0.0.2:1234
--variable RESOURCES:/opt/resources
). This also facilitates
localization testing, which often involves running the same tests
with different strings.${URL}
is shorter than
http://long.domain.name:8080/path/to/service?foo=1&bar=2&zap=42
.If a non-existent variable is used in the test data, the keyword using
it fails. If the same syntax that is used for variables is needed as a
literal string, it must be escaped with a backslash as in \${NAME}
.
This section explains how to use variables, including the normal scalar
variable syntax ${var}
, how to use variables in list and dictionary
contexts like @{var}
and &{var}
, respectively, and how to use environment
variables like %{var}
. Different ways how to create variables are discussed
in the subsequent sections.
Robot Framework variables, similarly as keywords, are
case-insensitive, and also spaces and underscores are
ignored. However, it is recommended to use capital letters with
global variables (for example, ${PATH}
or ${TWO WORDS}
)
and small letters with local variables that are only available in certain
test cases or user keywords (for example, ${my var}
). Much more
importantly, though, case should be used consistently.
Variable name consists of the variable type identifier ($
, @
, &
, %
),
curly braces ({
, }
) and the actual variable name between the braces.
Unlike in some programming languages where similar variable syntax is
used, curly braces are always mandatory. Variable names can basically have
any characters between the curly braces. However, using only alphabetic
characters from a to z, numbers, underscore and space is recommended, and
it is even a requirement for using the extended variable syntax.
The most common way to use variables in Robot Framework test data is using
the scalar variable syntax like ${var}
. When this syntax is used, the
variable name is replaced with its value as-is. Most of the time variable
values are strings, but variables can contain any object, including numbers,
lists, dictionaries, or even custom objects.
The example below illustrates the usage of scalar variables. Assuming
that the variables ${GREET}
and ${NAME}
are available
and assigned to strings Hello
and world
, respectively,
both the example test cases are equivalent.
*** Test Cases ***
Constants
Log Hello
Log Hello, world!!
Variables
Log ${GREET}
Log ${GREET}, ${NAME}!!
When a scalar variable is used alone without any text or other variables
around it, like in ${GREET}
above, the variable is replaced with
its value as-is and the value can be any object. If the variable is not used
alone, like ${GREER}, ${NAME}!!
above, its value is first converted into
a string and then concatenated with the other data.
Note
Variable values are used as-is without conversions also when
passing arguments to keywords using the named arguments
syntax like argname=${var}
.
The example below demonstrates the difference between having a
variable in alone or with other content. First, let us assume
that we have a variable ${STR}
set to a string Hello,
world!
and ${OBJ}
set to an instance of the following Python
object:
class MyObj:
def __str__():
return "Hi, terra!"
With these two variables set, we then have the following test data:
*** Test Cases ***
Objects
KW 1 ${STR}
KW 2 ${OBJ}
KW 3 I said "${STR}"
KW 4 You said "${OBJ}"
Finally, when this test data is executed, different keywords receive the arguments as explained below:
Hello, world!
${OBJ}
I said "Hello, world!"
You said "Hi, terra!"
Note
Converting variables to Unicode obviously fails if the variable
cannot be represented as Unicode. This can happen, for example,
if you try to use byte sequences as arguments to keywords so that
you catenate the values together like ${byte1}${byte2}
.
A workaround is creating a variable that contains the whole value
and using it alone in the cell (e.g. ${bytes}
) because then
the value is used as-is.
When a variable is used as a scalar like ${EXAMPLE}
, its value is be
used as-is. If a variable value is a list or list-like, it is also possible
to use it as a list variable like @{EXAMPLE}
. In this case the list is expanded
and individual items are passed in as separate arguments. This is easiest to explain
with an example. Assuming that a variable @{USER}
has value ['robot', 'secret']
,
the following two test cases are equivalent:
*** Test Cases ***
Constants
Login robot secret
List Variable
Login @{USER}
Robot Framework stores its own variables in one internal storage and allows using them as scalars, lists or dictionaries. Using a variable as a list requires its value to be a Python list or list-like object. Robot Framework does not allow strings to be used as lists, but other iterable objects such as tuples or dictionaries are accepted.
Starting from Robot Framework 4.0, list expansion can be used in combination with list item access making these usages possible:
*** Test Cases ***
Nested container
${nested} = Evaluate [['a', 'b', 'c'], {'key': ['x', 'y']}]
Log Many @{nested}[0] # Logs 'a', 'b' and 'c'.
Log Many @{nested}[1][key] # Logs 'x' and 'y'.
Slice
${items} = Create List first second third
Log Many @{items}[1:] # Logs 'second' and 'third'.
It is possible to use list variables with other arguments, including other list variables.
*** Test Cases ***
Example
Keyword @{LIST} more args
Keyword ${SCALAR} @{LIST} constant
Keyword @{LIST} @{ANOTHER} @{ONE MORE}
List variables can be used only with some of the settings. They can be used in arguments to imported libraries and variable files, but library and variable file names themselves cannot be list variables. Also with setups and teardowns list variable can not be used as the name of the keyword, but can be used in arguments. With tag related settings they can be used freely. Using scalar variables is possible in those places where list variables are not supported.
*** Settings ***
Library ExampleLibrary @{LIB ARGS} # This works
Library ${LIBRARY} @{LIB ARGS} # This works
Library @{LIBRARY AND ARGS} # This does not work
Suite Setup Some Keyword @{KW ARGS} # This works
Suite Setup ${KEYWORD} @{KW ARGS} # This works
Suite Setup @{KEYWORD AND ARGS} # This does not work
Default Tags @{TAGS} # This works
As discussed above, a variable containing a list can be used as a list
variable to pass list items to a keyword as individual arguments.
Similarly a variable containing a Python dictionary or a dictionary-like
object can be used as a dictionary variable like &{EXAMPLE}
. In practice
this means that the dictionary is expanded and individual items are passed as
named arguments to the keyword. Assuming that a variable &{USER}
has
value {'name': 'robot', 'password': 'secret'}
, the following two test cases
are equivalent.
*** Test Cases ***
Constants
Login name=robot password=secret
Dict Variable
Login &{USER}
Starting from Robot Framework 4.0, dictionary expansion can be used in combination with
dictionary item access making usages like &{nested}[key]
possible.
It is possible to use dictionary variables with other arguments, including other dictionary variables. Because named argument syntax requires positional arguments to be before named argument, dictionaries can only be followed by named arguments or other dictionaries.
*** Test Cases ***
Example
Keyword &{DICT} named=arg
Keyword positional @{LIST} &{DICT}
Keyword &{DICT} &{ANOTHER} &{ONE MORE}
Dictionary variables cannot generally be used with settings. The only exception are imports, setups and teardowns where dictionaries can be used as arguments.
*** Settings ***
Library ExampleLibrary &{LIB ARGS}
Suite Setup Some Keyword &{KW ARGS} named=arg
It is possible to access items of subscriptable variables, e.g. lists and dictionaries,
using special syntax like ${var}[item]
or ${var}[nested][item]
.
Starting from Robot Framework 4.0, it is also possible to use item access together with
list expansion and dictionary expansion by using syntax @{var}[item]
and
&{var}[item]
, respectively.
Note
Prior to Robot Framework 3.1 the normal item access syntax was @{var}[item]
with lists and &{var}[item]
with dictionaries. Robot Framework 3.1 introduced
the generic ${var}[item]
syntax along with some other nice enhancements and
the old item access syntax was deprecated in Robot Framework 3.2.
It is possible to access a certain item of a variable containing a sequence
(e.g. list, string or bytes) with the syntax ${var}[index]
, where index
is the index of the selected value. Indices start from zero, negative indices
can be used to access items from the end, and trying to access an item with
too large an index causes an error. Indices are automatically converted to
integers, and it is also possible to use variables as indices.
*** Test Cases ***
Positive index
Login ${USER}[0] ${USER}[1]
Title Should Be Welcome ${USER}[0]!
Negative index
Keyword ${SEQUENCE}[-1]
Index defined as variable
Keyword ${SEQUENCE}[${INDEX}]
Sequence item access supports also the same "slice" functionality as Python
with syntax like ${var}[1:]
. With this syntax you do not get a single
item but a slice of the original sequence. Same way as with Python you can
specify the start index, the end index, and the step:
*** Test Cases ***
Start index
Keyword ${SEQUENCE}[1:]
End index
Keyword ${SEQUENCE}[:4]
Start and end
Keyword ${SEQUENCE}[2:-1]
Step
Keyword ${SEQUENCE}[::2]
Keyword ${SEQUENCE}[1:-1:10]
Note
The slice syntax is new in Robot Framework 3.1. It was extended to work
with list expansion like @{var}[1:]
in Robot Framework 4.0.
Note
Prior to Robot Framework 3.2, item and slice access was only supported with variables containing lists, tuples, or other objects considered list-like. Nowadays all sequences, including strings and bytes, are supported.
It is possible to access a certain value of a dictionary variable
with the syntax ${NAME}[key]
, where key
is the name of the
selected value. Keys are considered to be strings, but non-strings
keys can be used as variables. Dictionary values accessed in this
manner can be used similarly as scalar variables.
If a key is a string, it is possible to access its value also using
attribute access syntax ${NAME.key}
. See Creating dictionary variables
for more details about this syntax.
*** Test Cases ***
Dictionary variable item
Login ${USER}[name] ${USER}[password]
Title Should Be Welcome ${USER}[name]!
Key defined as variable
Log Many ${DICT}[${KEY}] ${DICT}[${42}]
Attribute access
Login ${USER.name} ${USER.password}
Title Should Be Welcome ${USER.name}!
Also nested subscriptable variables can be accessed using the same
item access syntax like ${var}[item1][item2]
. This is especially useful
when working with JSON data often returned by REST services. For example,
if a variable ${DATA}
contains [{'id': 1, 'name': 'Robot'},
{'id': 2, 'name': 'Mr. X'}]
, this tests would pass:
*** Test Cases ***
Nested item access
Should Be Equal ${DATA}[0][name] Robot
Should Be Equal ${DATA}[1][id] ${2}
Robot Framework allows using environment variables in the test data using
the syntax %{ENV_VAR_NAME}
. They are limited to string values. It is
possible to specify a default value, that is used if the environment
variable does not exists, by separating the variable name and the default
value with an equal sign like %{ENV_VAR_NAME=default value}
.
Environment variables set in the operating system before the test execution are available during it, and it is possible to create new ones with the keyword Set Environment Variable or delete existing ones with the keyword Delete Environment Variable, both available in the OperatingSystem library. Because environment variables are global, environment variables set in one test case can be used in other test cases executed after it. However, changes to environment variables are not effective after the test execution.
*** Test Cases ***
Environment variables
Log Current user: %{USER}
Run %{JAVA_HOME}${/}javac
Environment variables with defaults
Set port %{APPLICATION_PORT=8080}
Note
Support for specifying the default value is new in Robot Framework 3.2.
Variables can spring into existence from different sources.
The most common source for variables are Variable sections in suite files and resource files. Variable sections are convenient, because they allow creating variables in the same place as the rest of the test data, and the needed syntax is very simple. Their main disadvantages are that values are always strings and they cannot be created dynamically. If either of these is a problem, variable files can be used instead.
The simplest possible variable assignment is setting a string into a
scalar variable. This is done by giving the variable name (including
${}
) in the first column of the Variable section and the value in
the second one. If the second column is empty, an empty string is set
as a value. Also an already defined variable can be used in the value.
*** Variables ***
${NAME} Robot Framework
${VERSION} 2.0
${ROBOT} ${NAME} ${VERSION}
It is also possible, but not obligatory,
to use the equals sign =
after the variable name to make assigning
variables slightly more explicit.
*** Variables ***
${NAME} = Robot Framework
${VERSION} = 2.0
If a scalar variable has a long value, it can be split into multiple rows
by using the ...
syntax. By default rows are concatenated together using
a space, but this can be changed by using a having separator
configuration
option after the last value:
*** Variables ***
${EXAMPLE} This value is joined
... together with a space.
${MULTILINE} First line.
... Second line.
... Third line.
... separator=\n
The separator
option is new in Robot Framework 7.0, but also older versions
support configuring the separator. With them the first value can contain a
special SEPARATOR
marker:
*** Variables ***
${MULTILINE} SEPARATOR=\n
... First line.
... Second line.
... Third line.
Both the separator
option and the SEPARATOR
marker are case-sensitive.
Using the separator
option is recommended, unless there is a need to
support also older versions.
Creating list variables is as easy as creating scalar variables. Again, the variable name is in the first column of the Variable section and values in the subsequent columns. A list variable can have any number of values, starting from zero, and if many values are needed, they can be split into several rows.
*** Variables ***
@{NAMES} Matti Teppo
@{NAMES2} @{NAMES} Seppo
@{NOTHING}
@{MANY} one two three four
... five six seven
Dictionary variables can be created in the Variable section similarly as
list variables. The difference is that items need to be created using
name=value
syntax or existing dictionary variables. If there are multiple
items with same name, the last value has precedence. If a name contains
a literal equal sign, it can be escaped with a backslash like \=
.
*** Variables ***
&{USER 1} name=Matti address=xxx phone=123
&{USER 2} name=Teppo address=yyy phone=456
&{MANY} first=1 second=${2} ${3}=third
&{EVEN MORE} &{MANY} first=override empty=
... =empty key\=here=value
Dictionary variables have two extra properties
compared to normal Python dictionaries. First of all, values of these
dictionaries can be accessed like attributes, which means that it is possible
to use extended variable syntax like ${VAR.key}
. This only works if the
key is a valid attribute name and does not match any normal attribute
Python dictionaries have. For example, individual value &{USER}[name]
can
also be accessed like ${USER.name}
(notice that $
is needed in this
context), but using ${MANY.3}
is not possible.
Tip
With nested dictionary variables keys are accessible like
${VAR.nested.key}
. This eases working with nested data structures.
Another special property of dictionary variables is
that they are ordered. This means that if these dictionaries are iterated,
their items always come in the order they are defined. This can be useful
if dictionaries are used as list variables with FOR loops or otherwise.
When a dictionary is used as a list variable, the actual value contains
dictionary keys. For example, @{MANY}
variable would have value ['first',
'second', 3]
.
Starting from Robot Framework 7.0, it is possible to create the variable name dynamically based on another variable:
*** Variables ***
${X} Y
${${X}} Z # Name is created based on '${X}'.
*** Test Cases ***
Dynamically created name
Should Be Equal ${Y} Z
Variable files are the most powerful mechanism for creating different kind of variables. It is possible to assign variables to any object using them, and they also enable creating variables dynamically. The variable file syntax and taking variable files into use is explained in section Resource and variable files.
Variables can be set from the command line either individually with the --variable (-v) option or using a variable file with the --variablefile (-V) option. Variables set from the command line are globally available for all executed test data files, and they also override possible variables with the same names in the Variable section and in variable files imported in the test data.
The syntax for setting individual variables is --variable
name:value, where name
is the name of the variable without
${}
and value
is its value. Several variables can be
set by using this option several times. Only scalar variables can be
set using this syntax and they can only get string values.
--variable EXAMPLE:value
--variable HOST:localhost:7272 --variable USER:robot
In the examples above, variables are set so that
${EXAMPLE}
gets the value value
${HOST}
and ${USER}
get the values
localhost:7272
and robot
The basic syntax for taking variable files into use from the command line is --variablefile path/to/variables.py, and Taking variable files into use section has more details. What variables actually are created depends on what variables there are in the referenced variable file.
If both variable files and individual variables are given from the command line, the latter have higher priority.
Return values from keywords can also be set into variables. This allows communication between different keywords even in different test libraries.
Variables set in this manner are otherwise similar to any other variables, but they are available only in the local scope where they are created. Thus it is not possible, for example, to set a variable like this in one test case and use it in another. This is because, in general, automated test cases should not depend on each other, and accidentally setting a variable that is used elsewhere could cause hard-to-debug errors. If there is a genuine need for setting a variable in one test case and using it in another, it is possible to use BuiltIn keywords as explained in the next section.
Any value returned by a keyword can be assigned to a scalar variable. As illustrated by the example below, the required syntax is very simple:
*** Test Cases ***
Returning
${x} = Get X an argument
Log We got ${x}!
In the above example the value returned by the Get X keyword
is first set into the variable ${x}
and then used by the Log
keyword. Having the equals sign =
after the variable name is
not obligatory, but it makes the assignment more explicit. Creating
local variables like this works both in test case and user keyword level.
Notice that although a value is assigned to a scalar variable, it can be used as a list variable if it has a list-like value and as a dictionary variable if it has a dictionary-like value.
*** Test Cases ***
Example
${list} = Create List first second third
Length Should Be ${list} 3
Log Many @{list}
Starting from Robot Framework 6.1, when working with variables that support
item assignment such as lists or dictionaries, it is possible to set their values
by specifying the index or key of the item using the syntax ${var}[item]
where the item
part can itself contain a variable:
*** Test Cases ***
Item assignment to list
${list} = Create List one two three four
${list}[0] = Set Variable first
${list}[${1}] = Set Variable second
${list}[2:3] = Evaluate ['third']
${list}[-1] = Set Variable last
Log Many @{list} # Logs 'first', 'second', 'third' and 'last'
Item assignment to dictionary
${dict} = Create Dictionary first_name=unknown
${dict}[first_name] = Set Variable John
${dict}[last_name] = Set Variable Doe
Log ${dictionary} # Logs {'first_name': 'John', 'last_name': 'Doe'}
Starting from Robot Framework 7.0, it is possible to create the name of the assigned variable dynamically based on another variable:
*** Test Cases ***
Dynamically created name
${x} = Set Variable y
${${x}} = Set Variable z # Name is created based on '${x}'.
Should Be Equal ${y} z
If a keyword returns a list or any list-like object, it is possible to assign it to a list variable:
*** Test Cases ***
Example
@{list} = Create List first second third
Length Should Be ${list} 3
Log Many @{list}
Because all Robot Framework variables are stored in the same namespace, there is not much difference between assigning a value to a scalar variable or a list variable. This can be seen by comparing the last two examples above. The main differences are that when creating a list variable, Robot Framework automatically verifies that the value is a list or list-like, and the stored variable value will be a new list created from the return value. When assigning to a scalar variable, the return value is not verified and the stored value will be the exact same object that was returned.
If a keyword returns a dictionary or any dictionary-like object, it is possible to assign it to a dictionary variable:
*** Test Cases ***
Example
&{dict} = Create Dictionary first=1 second=${2} ${3}=third
Length Should Be ${dict} 3
Do Something &{dict}
Log ${dict.first}
Because all Robot Framework variables are stored in the same namespace, it would also be possible to assign a dictionary into a scalar variable and use it later as a dictionary when needed. There are, however, some actual benefits in creating a dictionary variable explicitly. First of all, Robot Framework verifies that the returned value is a dictionary or dictionary-like similarly as it verifies that list variables can only get a list-like value.
A bigger benefit is that the value is converted into a special dictionary
that it uses also when creating dictionary variables in the Variable section.
Values in these dictionaries can be accessed using attribute access like
${dict.first}
in the above example. These dictionaries are also ordered, but
if the original dictionary was not ordered, the resulting order is arbitrary.
If a keyword returns a list or a list-like object, it is possible to assign individual values into multiple scalar variables or into scalar variables and a list variable.
*** Test Cases ***
Assign multiple
${a} ${b} ${c} = Get Three
${first} @{rest} = Get Three
@{before} ${last} = Get Three
${begin} @{middle} ${end} = Get Three
Assuming that the keyword Get Three returns a list [1, 2, 3]
,
the following variables are created:
${a}
, ${b}
and ${c}
with values 1
, 2
, and 3
, respectively.${first}
with value 1
, and @{rest}
with value [2, 3]
.@{before}
with value [1, 2]
and ${last}
with value 3
.${begin}
with value 1
, @{middle}
with value [2]
and ${end} with
value 3
.It is an error if the returned list has more or less values than there are scalar variables to assign. Additionally, only one list variable is allowed and dictionary variables can only be assigned alone.
To make it easier to understand what happens during execution, the beginning of value that is assigned is automatically logged. The default is to show 200 first characters, but this can be changed by using the --maxassignlength command line option when running tests. If the value is zero or negative, the whole assigned value is hidden.
--maxassignlength 1000
--maxassignlength 0
The reason the value is not logged fully is that it could be really big. If you always want to see a certain value fully, it is possible to use the BuiltIn Log keyword to log it after the assignment.
Note
The --maxassignlength option is new in Robot Framework 5.0.
VAR
syntaxStarting from Robot Framework 7.0, it is possible to create variables inside
tests and user keywords using the VAR
syntax. The VAR
marker is case-sensitive
and it must be followed by a variable name and value. Other than the mandatory
VAR
, the overall syntax is mostly the same as when creating variables
in the Variable section.
The new syntax is aims to make creating variables simpler and more uniform. It is especially indented to replace the BuiltIn keywords Set Variable, Set Test Variable, Set Suite Variable and Set Global Variable, but it can be used instead of Catenate, Create List and Create Dictionary as well.
In simple cases scalar variables are created by just giving a variable name
and its value. The value can be a hard-coded string or it can itself contain
a variable. If the value is long, it is possible to split it into multiple
columns and rows. In that case parts are joined together with a space by default,
but the separator to use can be specified with the separator
configuration
option. It is possible to have an optional =
after the variable name the same
way as when creating variables based on return values from keywords and in
the Variable section.
*** Test Cases ***
Scalar examples
VAR ${simple} variable
VAR ${equals} = this works too
VAR ${variable} value contains ${simple}
VAR ${sentence} This is a bit longer variable value
... that is split into multiple rows.
... These parts are joined with a space.
VAR ${multiline} This is another longer value.
... This time there is a custom separator.
... As the result this becomes a multiline string.
... separator=\n
List and dictionary variables are created similarly as scalar variables.
When creating dictionaries, items must be specified using the name=value
syntax.
*** Test Cases ***
List examples
VAR @{two items} Robot Framework
VAR @{empty list}
VAR @{lot of stuff}
... first item
... second item
... third item
... fourth item
... last item
Dictionary examples
VAR &{two items} name=Robot Framework url=http://robotframework.org
VAR &{empty dict}
VAR &{lot of stuff}
... first=1
... second=2
... third=3
... fourth=4
... last=5
Variables created with the VAR
syntax are are available only within the test
or user keyword where they are created. That can, however, be altered by using
the scope
configuration option. Supported values are:
LOCAL
TEST
TASK
TEST
that can be used when creating tasks.SUITE
SUITES
GLOBAL
Although Robot Framework variables are case-insensitive, it is recommended to use capital letters with non-local variable names.
*** Variables ***
${SUITE} this value is overridden
*** Test Cases ***
Scope example
VAR ${local} local value
VAR ${TEST} test value scope=TEST
VAR ${SUITE} suite value scope=SUITE
VAR ${SUITES} nested suite value scope=SUITES
VAR ${GLOBAL} global value scope=GLOBAL
Should Be Equal ${local} local value
Should Be Equal ${TEST} test value
Should Be Equal ${SUITE} suite value
Should Be Equal ${SUITES} nested suite value
Should Be Equal ${GLOBAL} global value
Keyword
Should Be Equal ${TEST} new test value
Should Be Equal ${SUITE} new suite value
Should Be Equal ${SUITES} new nested suite value
Should Be Equal ${GLOBAL} new global value
Scope example, part 2
Should Be Equal ${SUITE} new suite value
Should Be Equal ${SUITES} new nested suite value
Should Be Equal ${GLOBAL} new global value
*** Keywords ***
Keyword
Should Be Equal ${TEST} test value
Should Be Equal ${SUITE} suite value
Should Be Equal ${SUITES} nested suite value
Should Be Equal ${GLOBAL} global value
VAR ${TEST} new ${TEST} scope=TEST
VAR ${SUITE} new ${SUITE} scope=SUITE
VAR ${SUITES} new ${SUITES} scope=SUITES
VAR ${GLOBAL} new ${GLOBAL} scope=GLOBAL
Should Be Equal ${TEST} new test value
Should Be Equal ${SUITE} new suite value
Should Be Equal ${SUITES} new nested suite value
Should Be Equal ${GLOBAL} new global value
The VAR
syntax works with IF/ELSE structures which makes it easy to create
variables conditionally. In simple cases using inline IF can be convenient.
*** Test Cases ***
IF/ELSE example
IF "${ENV}" == "devel"
VAR ${address} 127.0.0.1
VAR ${name} demo
ELSE
VAR ${address} 192.168.1.42
VAR ${name} robot
END
Inline IF
IF "${ENV}" == "devel" VAR ${name} demo ELSE VAR ${name} robot
If there is a need, variable name can also be created dynamically based on another variable.
*** Test Cases ***
Dynamic name
VAR ${x} y # Normal assignment.
VAR ${${x}} z # Name created dynamically.
Should Be Equal ${y} z
Note
The VAR
syntax is recommended over these keywords when using
Robot Framework 7.0 or newer.
The BuiltIn library has keywords Set Test Variable, Set Suite Variable and Set Global Variable which can be used for setting variables dynamically during the test execution. If a variable already exists within the new scope, its value will be overwritten, and otherwise a new variable is created.
Variables set with Set Test Variable keyword are available everywhere within the scope of the currently executed test case. For example, if you set a variable in a user keyword, it is available both in the test case level and also in all other user keywords used in the current test. Other test cases will not see variables set with this keyword. It is an error to call Set Test Variable outside the scope of a test (e.g. in a Suite Setup or Teardown).
Variables set with Set Suite Variable keyword are available everywhere within the scope of the currently executed test suite. Setting variables with this keyword thus has the same effect as creating them using the Variable section in the test data file or importing them from variable files. Other test suites, including possible child test suites, will not see variables set with this keyword.
Variables set with Set Global Variable keyword are globally available in all test cases and suites executed after setting them. Setting variables with this keyword thus has the same effect as creating from the command line using the options --variable or --variablefile. Because this keyword can change variables everywhere, it should be used with care.
Note
Set Test/Suite/Global Variable keywords set named variables directly into test, suite or global variable scope and return nothing. On the other hand, another BuiltIn keyword Set Variable sets local variables using return values.
Robot Framework provides some built-in variables that are available automatically.
Built-in variables related to the operating system ease making the test data operating-system-agnostic.
Variable | Explanation |
---|---|
${CURDIR} | An absolute path to the directory where the test data file is located. This variable is case-sensitive. |
${TEMPDIR} | An absolute path to the system temporary directory. In UNIX-like systems this is typically /tmp, and in Windows c:\Documents and Settings\<user>\Local Settings\Temp. |
${EXECDIR} | An absolute path to the directory where test execution was started from. |
${/} | The system directory path separator. / in UNIX-like
systems and \ in Windows. |
${:} | The system path element separator. : in UNIX-like
systems and ; in Windows. |
${\n} | The system line separator. \n in UNIX-like systems and \r\n in Windows. |
*** Test Cases ***
Example
Create Binary File ${CURDIR}${/}input.data Some text here${\n}on two lines
Set Environment Variable CLASSPATH ${TEMPDIR}${:}${CURDIR}${/}foo.jar
The variable syntax can be used for creating both integers and floating point numbers, as illustrated in the example below. This is useful when a keyword expects to get an actual number, and not a string that just looks like a number, as an argument.
*** Test Cases ***
Example 1A
Connect example.com 80 # Connect gets two strings as arguments
Example 1B
Connect example.com ${80} # Connect gets a string and an integer
Example 2
Do X ${3.14} ${-1e-4} # Do X gets floating point numbers 3.14 and -0.0001
It is possible to create integers also from binary, octal, and
hexadecimal values using 0b
, 0o
and 0x
prefixes, respectively.
The syntax is case insensitive.
*** Test Cases ***
Example
Should Be Equal ${0b1011} ${11}
Should Be Equal ${0o10} ${8}
Should Be Equal ${0xff} ${255}
Should Be Equal ${0B1010} ${0XA}
Also Boolean values and Python None
can
be created using the variable syntax similarly as numbers.
*** Test Cases ***
Boolean
Set Status ${true} # Set Status gets Boolean true as an argument
Create Y something ${false} # Create Y gets a string and Boolean false
None
Do XYZ ${None} # Do XYZ gets Python None as an argument
These variables are case-insensitive, so for example ${True}
and
${true}
are equivalent.
It is possible to create spaces and empty strings using variables
${SPACE}
and ${EMPTY}
, respectively. These variables are
useful, for example, when there would otherwise be a need to escape
spaces or empty cells with a backslash. If more than one space is
needed, it is possible to use the extended variable syntax like
${SPACE * 5}
. In the following example, Should Be
Equal keyword gets identical arguments but those using variables are
easier to understand than those using backslashes.
*** Test Cases ***
One space
Should Be Equal ${SPACE} \ \
Four spaces
Should Be Equal ${SPACE * 4} \ \ \ \ \
Ten spaces
Should Be Equal ${SPACE * 10} \ \ \ \ \ \ \ \ \ \ \
Quoted space
Should Be Equal "${SPACE}" " "
Quoted spaces
Should Be Equal "${SPACE * 2}" " \ "
Empty
Should Be Equal ${EMPTY} \
There is also an empty list variable @{EMPTY}
and an empty dictionary
variable &{EMPTY}
. Because they have no content, they basically
vanish when used somewhere in the test data. They are useful, for example,
with test templates when the template keyword is used without
arguments or when overriding list or dictionary variables in different
scopes. Modifying the value of @{EMPTY}
or &{EMPTY}
is not possible.
*** Test Cases ***
Template
[Template] Some keyword
@{EMPTY}
Override
Set Global Variable @{LIST} @{EMPTY}
Set Suite Variable &{DICT} &{EMPTY}
Note
${SPACE}
represents the ASCII space (\x20
) and other spaces
should be specified using the escape sequences like \xA0
(NO-BREAK SPACE) and \u3000
(IDEOGRAPHIC SPACE).
Some automatic variables can also be used in the test data. These variables can have different values during the test execution and some of them are not even available all the time. Altering the value of these variables does not affect the original values, but some values can be changed dynamically using keywords from the BuiltIn library.
Variable | Explanation | Available |
---|---|---|
${TEST NAME} | The name of the current test case. | Test case |
@{TEST TAGS} | Contains the tags of the current test case in alphabetical order. Can be modified dynamically using Set Tags and Remove Tags keywords. | Test case |
${TEST DOCUMENTATION} | The documentation of the current test case. Can be set dynamically using using Set Test Documentation keyword. | Test case |
${TEST STATUS} | The status of the current test case, either PASS or FAIL. | Test teardown |
${TEST MESSAGE} | The message of the current test case. | Test teardown |
${PREV TEST NAME} | The name of the previous test case, or an empty string if no tests have been executed yet. | Everywhere |
${PREV TEST STATUS} | The status of the previous test case: either PASS, FAIL, or an empty string when no tests have been executed. | Everywhere |
${PREV TEST MESSAGE} | The possible error message of the previous test case. | Everywhere |
${SUITE NAME} | The full name of the current test suite. | Everywhere |
${SUITE SOURCE} | An absolute path to the suite file or directory. | Everywhere |
${SUITE DOCUMENTATION} | The documentation of the current test suite. Can be set dynamically using using Set Suite Documentation keyword. | Everywhere |
&{SUITE METADATA} | The free metadata of the current test suite. Can be set using Set Suite Metadata keyword. | Everywhere |
${SUITE STATUS} | The status of the current test suite, either PASS or FAIL. | Suite teardown |
${SUITE MESSAGE} | The full message of the current test suite, including statistics. | Suite teardown |
${KEYWORD STATUS} | The status of the current keyword, either PASS or FAIL. | User keyword teardown |
${KEYWORD MESSAGE} | The possible error message of the current keyword. | User keyword teardown |
${LOG LEVEL} | Current log level. | Everywhere |
${OUTPUT DIR} | An absolute path to the output directory as a string. | Everywhere |
${OUTPUT FILE} | An absolute path to the output file as a string or
a string NONE if the output file is not created. |
Everywhere |
${LOG FILE} | An absolute path to the log file as a string or
a string NONE if the log file is not created. |
Everywhere |
${REPORT FILE} | An absolute path to the report file as a string or
a string NONE if the report file is not created. |
Everywhere |
${DEBUG FILE} | An absolute path to the debug file as a string or
a string NONE if the debug file is not created. |
Everywhere |
&{OPTIONS} | A dictionary exposing command line options. The
dictionary keys match the command line options and
can be accessed both like
|
Everywhere |
Suite related variables ${SUITE SOURCE}
, ${SUITE NAME}
, ${SUITE DOCUMENTATION}
and &{SUITE METADATA}
as well as options related to command line options like
${LOG FILE}
and &{OPTIONS}
are available already when libraries and variable
files are imported. Possible variables in these automatic variables are not yet
resolved at the import time, though.
Variables coming from different sources have different priorities and are available in different scopes.
Variables from the command line
Variables set in the command line have the highest priority of all variables that can be set before the actual test execution starts. They override possible variables created in Variable sections in test case files, as well as in resource and variable files imported in the test data.
Individually set variables (--variable option) override the variables set using variable files (--variablefile option). If you specify same individual variable multiple times, the one specified last will override earlier ones. This allows setting default values for variables in a start-up script and overriding them from the command line. Notice, though, that if multiple variable files have same variables, the ones in the file specified first have the highest priority.
Variable section in a test case file
Variables created using the Variable section in a test case file are available for all the test cases in that file. These variables override possible variables with same names in imported resource and variable files.
Variables created in the Variable sections are available in all other sections in the file where they are created. This means that they can be used also in the Setting section, for example, for importing more variables from resource and variable files.
Imported resource and variable files
Variables imported from the resource and variable files have the lowest priority of all variables created in the test data. Variables from resource files and variable files have the same priority. If several resource and/or variable file have same variables, the ones in the file imported first are taken into use.
If a resource file imports resource files or variable files, variables in its own Variable section have a higher priority than variables it imports. All these variables are available for files that import this resource file.
Note that variables imported from resource and variable files are not available in the Variable section of the file that imports them. This is due to the Variable section being processed before the Setting section where the resource files and variable files are imported.
Variables set during test execution
Variables set during the test execution either using return values from keywords or using Set Test/Suite/Global Variable keywords always override possible existing variables in the scope where they are set. In a sense they thus have the highest priority, but on the other hand they do not affect variables outside the scope they are defined.
Built-in variables
Built-in variables like${TEMPDIR}
and${TEST_NAME}
have the highest priority of all variables. They cannot be overridden using Variable section or from command line, but even they can be reset during the test execution. An exception to this rule are number variables, which are resolved dynamically if no variable is found otherwise. They can thus be overridden, but that is generally a bad idea. Additionally${CURDIR}
is special because it is replaced already during the test data processing time.
Depending on where and how they are created, variables can have a global, test suite, test case or local scope.
Global variables are available everywhere in the test data. These variables are normally set from the command line with the --variable and --variablefile options, but it is also possible to create new global variables or change the existing ones by using the VAR syntax or the Set Global Variable keyword anywhere in the test data. Additionally also built-in variables are global.
It is recommended to use capital letters with all global variables.
Variables with the test suite scope are available anywhere in the test suite where they are defined or imported. They can be created in Variable sections, imported from resource and variable files, or set during the test execution using the VAR syntax or the Set Suite Variable keyword.
The test suite scope is not recursive, which means that variables available in a higher-level test suite are not available in lower-level suites. If necessary, resource and variable files can be used for sharing variables.
Since these variables can be considered global in the test suite where they are used, it is recommended to use capital letters also with them.
Variables with the test case scope are visible in a test case and in all user keywords the test uses. Initially there are no variables in this scope, but it is possible to create them by using the VAR syntax or the Set Test Variable keyword anywhere in a test case. Trying to create test variables in suite setup or suite teardown causes and error.
Also variables in the test case scope are to some extend global. It is thus generally recommended to use capital letters with them too.
Test cases and user keywords have a local variable scope that is not seen by other tests or keywords. Local variables can be created using return values from executed keywords and with the VAR syntax, and user keywords also get them as arguments.
It is recommended to use lower-case letters with local variables.
Extended variable syntax allows accessing attributes of an object assigned
to a variable (for example, ${object.attribute}
) and even calling
its methods (for example, ${obj.getName()}
). It works both with
scalar and list variables, but is mainly useful with the former.
Extended variable syntax is a powerful feature, but it should be used with care. Accessing attributes is normally not a problem, on the contrary, because one variable containing an object with several attributes is often better than having several variables. On the other hand, calling methods, especially when they are used with arguments, can make the test data pretty complicated to understand. If that happens, it is recommended to move the code into a test library.
The most common usages of extended variable syntax are illustrated in the example below. First assume that we have the following variable file and test case:
class MyObject:
def __init__(self, name):
self.name = name
def eat(self, what):
return '%s eats %s' % (self.name, what)
def __str__(self):
return self.name
OBJECT = MyObject('Robot')
DICTIONARY = {1: 'one', 2: 'two', 3: 'three'}
*** Test Cases ***
Example
KW 1 ${OBJECT.name}
KW 2 ${OBJECT.eat('Cucumber')}
KW 3 ${DICTIONARY[2]}
When this test data is executed, the keywords get the arguments as explained below:
Robot
Robot eats Cucumber
two
The extended variable syntax is evaluated in the following order:
{
until
the first occurrence of a character that is not an alphanumeric character
or a space. For example, base variables of ${OBJECT.name}
and ${DICTIONARY[2]}
) are OBJECT
and DICTIONARY
,
respectively.Many standard Python objects, including strings and numbers, have methods that can be used with the extended variable syntax either explicitly or implicitly. Sometimes this can be really useful and reduce the need for setting temporary variables, but it is also easy to overuse it and create really cryptic test data. Following examples show few pretty good usages.
*** Test Cases ***
String
${string} = Set Variable abc
Log ${string.upper()} # Logs 'ABC'
Log ${string * 2} # Logs 'abcabc'
Number
${number} = Set Variable ${-2}
Log ${number * 10} # Logs -20
Log ${number.__abs__()} # Logs 2
Note that even though abs(number)
is recommended over
number.__abs__()
in normal Python code, using
${abs(number)}
does not work. This is because the variable name
must be in the beginning of the extended syntax. Using __xxx__
methods in the test data like this is already a bit questionable, and
it is normally better to move this kind of logic into test libraries.
Extended variable syntax works also in list variable context.
If, for example, an object assigned to a variable ${EXTENDED}
has
an attribute attribute
that contains a list as a value, it can be
used as a list variable @{EXTENDED.attribute}
.
It is possible to set attributes of
objects stored to scalar variables using keyword return values and
a variation of the extended variable syntax. Assuming we have
variable ${OBJECT}
from the previous examples, attributes could
be set to it like in the example below.
*** Test Cases ***
Example
${OBJECT.name} = Set Variable New name
${OBJECT.new_attr} = Set Variable New attribute
The extended variable assignment syntax is evaluated using the following rules:
${OBJECT.name}
in the example above) that variable
will be assigned a new value and the extended syntax is not used.${
and
the last dot, for example, OBJECT
in ${OBJECT.name}
and foo.bar
in ${foo.bar.zap}
. As the second example
illustrates, the base name may contain normal extended variable
syntax.}
, for
example, name
in ${OBJECT.name}
. If the name does not
start with a letter or underscore and contain only these characters
and numbers, the attribute is considered invalid and the extended
syntax is not used. A new variable with the full name is created
instead.Note
Unlike when assigning variables normally using return values from keywords, changes to variables done using the extended assign syntax are not limited to the current scope. Because no new variable is created but instead the state of an existing variable is changed, all tests and keywords that see that variable will also see the changes.
Variables are allowed also inside variables, and when this syntax is
used, variables are resolved from the inside out. For example, if you
have a variable ${var${x}}
, then ${x}
is resolved
first. If it has the value name
, the final value is then the
value of the variable ${varname}
. There can be several nested
variables, but resolving the outermost fails, if any of them does not
exist.
In the example below, Do X gets the value ${JOHN HOME}
or ${JANE HOME}
, depending on if Get Name returns
john
or jane
. If it returns something else, resolving
${${name} HOME}
fails.
*** Variables ***
${JOHN HOME} /home/john
${JANE HOME} /home/jane
*** Test Cases ***
Example
${name} = Get Name
Do X ${${name} HOME}
Variable syntax can also be used for evaluating Python expressions. The
basic syntax is ${{expression}}
i.e. there are double curly braces around
the expression. The expression
can be any valid Python expression such as
${{1 + 2}}
or ${{['a', 'list']}}
. Spaces around the expression are allowed,
so also ${{ 1 + 2 }}
and ${{ ['a', 'list'] }}
are valid. In addition to
using normal scalar variables, also list variables and
dictionary variables support @{{expression}}
and &{{expression}}
syntax,
respectively.
Main usages for this pretty advanced functionality are:
${{len('${var}') > 3}}
, ${{$var[0] if $var is not None else None}}
).${{decimal.Decimal('0.11')}}
, ${{datetime.date(2019, 11, 5)}}
).${{random.randint(0, 100)}}
,
${{datetime.date.today()}}
).${{[1, 2, 3, 4]}}
,
${{ {'id': 1, 'name': 'Example', 'children': [7, 9]} }}
).${{math.pi}}
, ${{platform.system()}}
).This is somewhat similar functionality than the extended variable syntax
discussed earlier. As the examples above illustrate, this syntax is even more
powerful as it provides access to Python built-ins like len()
and modules
like math
. In addition to being able to use variables like ${var}
in
the expressions (they are replaced before evaluation), variables are also
available using the special $var
syntax during evaluation. The whole expression
syntax is explained in the Evaluating expressions appendix.
Tip
Instead of creating complicated expressions, it is often better to move the logic into a custom library. That eases maintenance, makes test data easier to understand and can also enhance execution speed.
Note
The inline Python evaluation syntax is new in Robot Framework 3.2.
Keyword sections are used to create new higher-level keywords by combining existing keywords together. These keywords are called user keywords to differentiate them from lowest level library keywords that are implemented in test libraries. The syntax for creating user keywords is very close to the syntax for creating test cases, which makes it easy to learn.
In many ways, the overall user keyword syntax is identical to the test case syntax. User keywords are created in Keyword sections which differ from Test Case sections only by the name that is used to identify them. User keyword names are in the first column similarly as test cases names. Also user keywords are created from keywords, either from keywords in test libraries or other user keywords. Keyword names are normally in the second column, but when setting variables from keyword return values, they are in the subsequent columns.
*** Keywords ***
Open Login Page
Open Browser http://host/login.html
Title Should Be Login Page
Title Should Start With
[Arguments] ${expected}
${title} = Get Title
Should Start With ${title} ${expected}
Most user keywords take some arguments. This important feature is used already in the second example above, and it is explained in detail later in this section, similarly as user keyword return values.
User keywords can be created in suite files, resource files, and suite initialization files. Keywords created in resource files are available for files using them, whereas other keywords are only available in the files where they are created.
User keywords can have similar settings as test cases, and they have the same square bracket syntax separating them from keyword names. All available settings are listed below and explained later in this section.
Note
The format used above is recommended, but setting names are
case-insensitive and spaces are allowed between brackets and the name.
For example, [ TAGS ]
:setting is valid.
The user keyword name is defined in the first column of the Keyword section. Of course, the name should be descriptive, and it is acceptable to have quite long keyword names. Actually, when creating use-case-like test cases, the highest-level keywords are often formulated as sentences or even paragraphs.
User keywords can have a documentation that is set with the [Documentation] setting. It supports same formatting, splitting to multiple lines, and other features as test case documentation. This setting documents the user keyword in the test data. It is also shown in a more formal keyword documentation, which the Libdoc tool can create from resource files. Finally, the first logical row of the documentation, until the first empty row, is shown as a keyword documentation in test logs.
*** Keywords ***
One line documentation
[Documentation] One line documentation.
No Operation
Multiline documentation
[Documentation] The first line creates the short doc.
...
... This is the body of the documentation.
... It is not shown in Libdoc outputs but only
... the short doc is shown in logs.
No Operation
Short documentation in multiple lines
[Documentation] If the short doc gets longer, it can span
... multiple physical lines.
...
... The body is separated from the short doc with
... an empty line.
No Operation
Sometimes keywords need to be removed, replaced with new ones, or
deprecated for other reasons. User keywords can be marked deprecated
by starting the documentation with *DEPRECATED*
, which will
cause a warning when the keyword is used. For more information, see
the Deprecating keywords section.
Note
Prior to Robot Framework 3.1, the short documentation contained only the first physical line of the keyword documentation.
Both user keywords and library keywords can have tags. Similarly as when tagging test cases, there are two settings affecting user keyword tags:
-tag
syntax.*** Settings ***
Keyword Tags gui html
*** Keywords ***
No own tags
[Documentation] Keyword has tags 'gui' and 'html'.
No Operation
Own tags
[Documentation] Keyword has tags 'gui', 'html', 'own' and 'tags'.
[Tags] own tags
No Operation
Remove common tag
[Documentation] Test has tags 'gui' and 'own'.
[Tags] own -html
No Operation
Keyword tags can be specified using variables, the -tag
syntax supports
patterns, and so on, exactly as test case tags.
In addition to using the dedicated settings, keyword tags can be specified on
the last line of the documentation with Tags:
prefix so that tags are separated
with a comma. For example, following two keywords get same three tags:
*** Keywords ***
Settings tags using separate setting
[Tags] my fine tags
No Operation
Settings tags using documentation
[Documentation] I have documentation. And my documentation has tags.
... Tags: my, fine, tags
No Operation
Keyword tags are shown in logs and in documentation generated by Libdoc, where the keywords can also be searched based on tags. The --removekeywords and --flattenkeywords commandline options also support selecting keywords by tag, and new usages for keywords tags are possibly added in later releases.
Similarly as with test case tags, user keyword tags with the robot:
prefix are reserved for special features by Robot Framework
itself. Users should thus not use any tag with these prefixes unless actually
activating the special functionality. Starting from Robot Framework 6.1,
flattening keyword during execution time can be taken into use using
reserved tag robot:flatten
.
Note
Keyword Tags is new in Robot Framework 6.0. With earlier versions all keyword tags need to be specified using the [Tags] setting.
Note
The -tag
syntax for removing common tags is new in Robot Framework 7.0.
Most user keywords need to take some arguments. The syntax for
specifying them is probably the most complicated feature normally
needed with Robot Framework, but even that is relatively easy,
particularly in most common cases. Arguments are normally specified with
the [Arguments] setting, and argument names use the same
syntax as variables, for example ${arg}
.
The simplest way to specify arguments (apart from not having them at all) is using only positional arguments. In most cases, this is all that is needed.
The syntax is such that first the [Arguments] setting is
given and then argument names are defined in the subsequent
cells. Each argument is in its own cell, using the same syntax as with
variables. The keyword must be used with as many arguments as there
are argument names in its signature. The actual argument names do not
matter to the framework, but from users' perspective they should
be as descriptive as possible. It is recommended
to use lower-case letters in variable names, either as
${my_arg}
, ${my arg}
or ${myArg}
.
*** Keywords ***
One Argument
[Arguments] ${arg_name}
Log Got argument ${arg_name}
Three Arguments
[Arguments] ${arg1} ${arg2} ${arg3}
Log 1st argument: ${arg1}
Log 2nd argument: ${arg2}
Log 3rd argument: ${arg3}
When creating user keywords, positional arguments are sufficient in most situations. It is, however, sometimes useful that keywords have default values for some or all of their arguments. Also user keywords support default values, and the needed new syntax does not add very much to the already discussed basic syntax.
In short, default values are added to arguments, so that first there is
the equals sign (=
) and then the value, for example ${arg}=default
.
There can be many arguments with defaults, but they all must be given after
the normal positional arguments. The default value can contain a variable
created on test, suite or global scope, but local variables of the keyword
executor cannot be used. Default value can
also be defined based on earlier arguments accepted by the keyword.
Note
The syntax for default values is space sensitive. Spaces
before the =
sign are not allowed, and possible spaces
after it are considered part of the default value itself.
*** Keywords ***
One Argument With Default Value
[Arguments] ${arg}=default value
[Documentation] This keyword takes 0-1 arguments
Log Got argument ${arg}
Two Arguments With Defaults
[Arguments] ${arg1}=default 1 ${arg2}=${VARIABLE}
[Documentation] This keyword takes 0-2 arguments
Log 1st argument ${arg1}
Log 2nd argument ${arg2}
One Required And One With Default
[Arguments] ${required} ${optional}=default
[Documentation] This keyword takes 1-2 arguments
Log Required: ${required}
Log Optional: ${optional}
Default Based On Earlier Argument
[Arguments] ${a} ${b}=${a} ${c}=${a} and ${b}
Should Be Equal ${a} ${b}
Should Be Equal ${c} ${a} and ${b}
When a keyword accepts several arguments with default values and only
some of them needs to be overridden, it is often handy to use the
named arguments syntax. When this syntax is used with user
keywords, the arguments are specified without the ${}
decoration. For example, the second keyword above could be used like
below and ${arg1}
would still get its default value.
*** Test Cases ***
Example
Two Arguments With Defaults arg2=new value
As all Pythonistas must have already noticed, the syntax for specifying default arguments is heavily inspired by Python syntax for function default values.
Sometimes even default values are not enough and there is a need
for a keyword accepting variable number of arguments. User keywords
support also this feature. All that is needed is having list variable such
as @{varargs}
after possible positional arguments in the keyword signature.
This syntax can be combined with the previously described default values, and
at the end the list variable gets all the leftover arguments that do not match
other arguments. The list variable can thus have any number of items, even zero.
*** Keywords ***
Any Number Of Arguments
[Arguments] @{varargs}
Log Many @{varargs}
One Or More Arguments
[Arguments] ${required} @{rest}
Log Many ${required} @{rest}
Required, Default, Varargs
[Arguments] ${req} ${opt}=42 @{others}
Log Required: ${req}
Log Optional: ${opt}
Log Others:
FOR ${item} IN @{others}
Log ${item}
END
Notice that if the last keyword above is used with more than one
argument, the second argument ${opt}
always gets the given
value instead of the default value. This happens even if the given
value is empty. The last example also illustrates how a variable
number of arguments accepted by a user keyword can be used in a for
loop. This combination of two rather advanced functions can
sometimes be very useful.
The keywords in the examples above could be used, for example, like this:
*** Test Cases ***
Varargs with user keywords
Any Number Of Arguments
Any Number Of Arguments arg
Any Number Of Arguments arg1 arg2 arg3 arg4
One Or More Arguments required
One Or More Arguments arg1 arg2 arg3 arg4
Required, Default, Varargs required
Required, Default, Varargs required optional
Required, Default, Varargs arg1 arg2 arg3 arg4 arg5
Again, Pythonistas probably notice that the variable number of arguments syntax is very close to the one in Python.
User keywords can also accept free named arguments by having a dictionary
variable like &{named}
as the absolutely last argument. When the keyword
is called, this variable will get all named arguments that do not match
any positional argument or named-only argument in the keyword
signature.
*** Keywords ***
Free Named Only
[Arguments] &{named}
Log Many &{named}
Positional And Free Named
[Arguments] ${required} &{extra}
Log Many ${required} &{extra}
Run Program
[Arguments] @{args} &{config}
Run Process program.py @{args} &{config}
The last example above shows how to create a wrapper keyword that accepts any positional or named argument and passes them forward. See free named argument examples for a full example with same keyword.
Free named arguments support with user keywords works similarly as kwargs
work in Python. In the signature and also when passing arguments forward,
&{kwargs}
is pretty much the same as Python's **kwargs
.
Starting from Robot Framework 3.1, user keywords support named-only
arguments that are inspired by Python 3 keyword-only arguments.
This syntax is typically used by having normal arguments after
variable number of arguments (@{varargs}
). If the keywords does not
use varargs, it is possible to use just @{}
to denote that the subsequent
arguments are named-only:
*** Keywords ***
With Varargs
[Arguments] @{varargs} ${named}
Log Many @{varargs} ${named}
Without Varargs
[Arguments] @{} ${first} ${second}
Log Many ${first} ${second}
Named-only arguments can be used together with positional arguments as well as with free named arguments. When using free named arguments, they must be last:
*** Keywords ***
With Positional
[Arguments] ${positional} @{} ${named}
Log Many ${positional} ${named}
With Free Named
[Arguments] @{varargs} ${named only} &{free named}
Log Many @{varargs} ${named only} &{free named}
When passing named-only arguments to keywords, their order does not matter other than they must follow possible positional arguments. The keywords above could be used, for example, like this:
*** Test Cases ***
Example
With Varargs named=value
With Varargs positional second positional named=foobar
Without Varargs first=1 second=2
Without Varargs second=toka first=eka
With Positional foo named=bar
With Positional named=2 positional=1
With Free Named positional named only=value x=1 y=2
With Free Named foo=a bar=b named only=c quux=d
Named-only arguments can have default values similarly as normal user keyword arguments. A minor difference is that the order of arguments with and without default values is not important.
*** Keywords ***
With Default
[Arguments] @{} ${named}=default
Log Many ${named}
With And Without Defaults
[Arguments] @{} ${optional}=default ${mandatory} ${mandatory 2} ${optional 2}=default 2 ${mandatory 3}
Log Many ${optional} ${mandatory} ${mandatory 2} ${optional 2} ${mandatory 3}
The previous section explained how to pass arguments to keywords so that they are listed separately after the keyword name. Robot Framework has also another approach to pass arguments, embedding them directly to the keyword name, used by the second test below:
*** Test Cases ***
Normal arguments
Select from list cat
Embedded arguments
Select cat from list
As the example illustrates, embedding arguments to keyword names can make the data easier to read and understand even for people without any Robot Framework experience.
The previous example showed how using a keyword Select cat from list is
more fluent than using Select from list so that cat
is passed to
it as an argument. We obviously could implement Select cat from list
as a normal keyword accepting no arguments, but then we needed to implement
various other keywords like Select dog from list for other animals.
Embedded arguments simplify this and we can instead implement just one
keyword with name Select ${animal} from list and use it with any
animal:
*** Test Cases ***
Embedded arguments
Select cat from list
Select dog from list
*** Keywords ***
Select ${animal} from list
Open Page Pet Selection
Select Item From List animal_list ${animal}
As the above example shows, embedded arguments are specified simply by using
variables in keyword names. The arguments used in the name are naturally
available inside the keyword and they have different values depending on how
the keyword is called. In the above example, ${animal}
has value cat
when
the keyword is used for the first time and dog
when it is used for
the second time.
Starting from Robot Framework 6.1, it is possible to create user keywords that accept both embedded and "normal" arguments:
*** Test Cases ***
Embedded and normal arguments
Number of cats should be 2
Number of dogs should be count=3
*** Keywords ***
Number of ${animals} should be
[Arguments] ${count}
Open Page Pet Selection
Select Items From List animal_list ${animals}
Number of Selected List Items Should Be ${count}
Other than the special name, keywords with embedded arguments are created just like other user keywords. They are also used the same way as other keywords except that spaces and underscores are not ignored in their names when keywords are matched. They are, however, case-insensitive like other keywords. For example, the Select ${animal} from list keyword could be used like select cow from list, but not like Select cow fromlist.
Embedded arguments do not support default values or variable number of arguments like normal arguments do. If such functionality is needed, normal arguments should be used instead. Passing embedded arguments as variables is possible, but that can reduce readability:
*** Variables ***
${SELECT} cat
*** Test Cases ***
Embedded arguments with variable
Select ${SELECT} from list
*** Keywords ***
Select ${animal} from list
Open Page Pet Selection
Select Item From List animal_list ${animal}
One tricky part in using embedded arguments is making sure that the
values used when calling the keyword match the correct arguments. This
is a problem especially if there are multiple arguments and characters
separating them may also appear in the given values. For example,
Select Los Angeles Lakers in the following example matches
Select ${city} ${team} so that ${city}
contains Los
and
${team}
contains Angeles Lakers
:
*** Test Cases ***
Example
Select Chicago Bulls
Select Los Angeles Lakers
*** Keywords ***
Select ${city} ${team}
Log Selected ${team} from ${city}.
An easy solution to this problem is surrounding arguments with double quotes or other characters not used in the actual values. This fixed example works so that cities and teams match correctly:
*** Test Cases ***
Example
Select "Chicago" "Bulls"
Select "Los Angeles" "Lakers"
*** Keywords ***
Select "${city}" "${team}"
Log Selected ${team} from ${city}.
This approach is not enough to resolve all conflicts, but it helps in common cases and is generally recommended. Another benefit is that it makes arguments stand out from rest of the keyword.
Prior to Robot Framework 7.1, embedded arguments starting the keyword name also
matched possible given/when/then/and/but prefixes typically used in Behavior
Driven Development (BDD). For example, ${name} goes home matched
Given Janne goes home so that ${name}
got value Given Janne
.
Nowadays the prefix is ignored and ${name}
will be Janne
as expected.
If older Robot Framework versions need to be supported, it is easiest to quote
the argument like in "${name}" goes home to get consistent behavior.
An alternative solution for limiting what values arguments match is using custom regular expressions.
When using embedded arguments, it is pretty common that there are multiple
keyword implementations that match the keyword that is used. For example,
Execute "ls" with "lf" in the example below matches both of the keywords.
It matching Execute "${cmd}" with "${opts}" is pretty obvious and what
we want, but it also matches Execute "${cmd}" so that ${cmd}
matches
ls" with "-lh
.
*** Settings ***
Library Process
*** Test Cases ***
Automatic conflict resolution
Execute "ls"
Execute "ls" with "-lh"
*** Keywords ***
Execute "${cmd}"
Run Process ${cmd} shell=True
Execute "${cmd}" with "${opts}"
Run Process ${cmd} ${opts} shell=True
When this kind of conflicts occur, Robot Framework tries to automatically select the best match and use that. In the above example, Execute "${cmd}" with "${opts}" is considered a better match than the more generic Execute "${cmd}" and running the example thus succeeds without conflicts.
It is not always possible to find a single match that is better than others. For example, the second test below fails because Robot Framework matches both of the keywords equally well. This kind of conflicts need to be resolved manually either by renaming keywords or by using custom regular expressions.
*** Test Cases ***
No conflict
Automation framework
Robot uprising
Unresolvable conflict
Robot Framework
*** Keywords ***
${type} Framework
Should Be Equal ${type} Automation
Robot ${action}
Should Be Equal ${action} uprising
Keywords that accept only "normal" arguments or no arguments at all are considered to match better than keywords accepting embedded arguments. For example, if the following keyword is added to the above example, Robot Framework used by the latter test matches it and the test succeeds:
*** Keywords ***
Robot Framework
No Operation
Before looking which match is best, Robot Framework checks are some of the matching keywords implemented in the same file as the caller keyword. If there are such keywords, they are given precedence over other keywords. Alternatively, library search order can be used to control the order in which Robot Framework looks for keywords in resources and libraries.
Note
Automatically resolving conflicts if multiple keywords with embedded arguments match is a new feature in Robot Framework 6.0. With older versions custom regular expressions explained below can be used instead.
When keywords with embedded arguments are called, the values are matched
internally using regular expressions (regexps for short). The default
logic goes so that every argument in the name is replaced with a pattern .*?
that matches any string and tries to match as little as possible. This logic works
fairly well normally, but as discussed above, sometimes keywords
match wrong values and sometimes there are conflicts that cannot
be resolved . A solution in these cases is specifying a custom regular
expression that makes sure that the keyword matches only what it should in that
particular context. To be able to use this feature, and to fully
understand the examples in this section, you need to understand at
least the basics of the regular expression syntax.
A custom embedded argument regular expression is defined after the
base name of the argument so that the argument and the regexp are
separated with a colon. For example, an argument that should match
only numbers can be defined like ${arg:\d+}
.
Using custom regular expressions is illustrated by the following examples.
Notice that the first one shows how the earlier problem with
Select ${city} ${team} not matching Select Los Angeles Lakers
properly can be resolved without quoting. That is achieved by implementing
the keyword so that ${team}
can only contain non-whitespace characters.
*** Settings ***
Library DateTime
*** Test Cases ***
Do not match whitespace characters
Select Chicago Bulls
Select Los Angeles Lakers
Match numbers and characters from set
1 + 2 = 3
53 - 11 = 42
Match either date or literal 'today'
Deadline is 2022-09-21
Deadline is today
*** Keywords ***
Select ${city} ${team:\S+}
Log Selected ${team} from ${city}.
${number1:\d+} ${operator:[+-]} ${number2:\d+} = ${expected:\d+}
${result} = Evaluate ${number1} ${operator} ${number2}
Should Be Equal As Integers ${result} ${expected}
Deadline is ${date:(\d{4}-\d{2}-\d{2}|today)}
IF '${date}' == 'today'
${date} = Get Current Date
ELSE
${date} = Convert Date ${date}
END
Log Deadline is on ${date}.
Being implemented with Python, Robot Framework naturally uses Python's
re module that has pretty standard regular expressions syntax.
This syntax is otherwise fully supported with embedded arguments, but
regexp extensions in format (?...)
cannot be used. If the regular
expression syntax is invalid, creating the keyword fails with an error
visible in test execution errors.
Regular expressions use the backslash character (\) heavily both
to form special sequences (e.g. \d
) and to escape characters that have
a special meaning in regexps (e.g. \$
). Typically in Robot Framework data
backslash characters need to be escaped with another backslash, but
that is not required in this context. If there is a need to have a literal
backslash in the pattern, then the backslash must be escaped like
${path:c:\\temp\\.*}
.
Possible lone opening and closing curly braces in the pattern must be escaped
like ${open:\{}
and ${close:\}}
or otherwise Robot Framework is not able
to parse the variable syntax correctly. If there are matching braces like in
${digits:\d{2}}
, escaping is not needed.
Note
Prior to Robot Framework 3.2, it was mandatory to escape all
closing curly braces in the pattern like ${digits:\d{2\}}
.
This syntax is unfortunately not supported by Robot Framework 3.2
or newer and keywords using it must be updated when upgrading.
Note
Prior to Robot Framework 6.0, using literal backslashes in the pattern
required double escaping them like ${path:c:\\\\temp\\\\.*}
.
Patterns using literal backslashes need to be updated when upgrading.
When embedded arguments are used with custom regular expressions, Robot Framework automatically enhances the specified regexps so that they match variables in addition to the text matching the pattern. For example, the following test case would pass using the keywords from the earlier example.
*** Variables ***
${DATE} 2011-06-27
*** Test Cases ***
Example
Deadline is ${DATE}
${1} + ${2} = ${3}
A limitation of using variables is that their actual values are not matched against custom regular expressions. As the result keywords may be called with values that their custom regexps would not allow. This behavior is deprecated starting from Robot Framework 6.0 and values will be validated in the future. For more information see issue #4462.
A big benefit of having arguments as part of the keyword name is that it makes it easier to use higher-level sentence-like keywords when using the behavior-driven style to write tests. As the example below shows, this support is typically used in combination with the possibility to omit Given, When and Then prefixes in keyword definitions:
*** Test Cases ***
Add two numbers
Given I have Calculator open
When I add 2 and 40
Then result should be 42
Add negative numbers
Given I have Calculator open
When I add 1 and -2
Then result should be -1
*** Keywords ***
I have ${program} open
Start Program ${program}
I add ${number 1} and ${number 2}
Input Number ${number 1}
Push Button +
Input Number ${number 2}
Push Button =
Result should be ${expected}
${result} = Get Result
Should Be Equal ${result} ${expected}
Note
Embedded arguments feature in Robot Framework is inspired by how step definitions are created in the popular BDD tool Cucumber.
Similarly as library keywords, also user keywords can return values. When using Robot Framework 5.0 or newer, the recommended approach is using the native RETURN statement. The old [Return] setting was deprecated in Robot Framework 7.0 and also BuiltIn keywords Return From Keyword and Return From Keyword If are considered deprecated.
Regardless how values are returned, they can be assigned to variables in test cases and in other user keywords.
RETURN
statementThe recommended approach to return values is using the RETURN
statement.
It accepts optional return values and can be used with IF and inline IF
structures. Its usage is easiest explained with examples:
*** Keywords ***
Return One Value
[Arguments] ${arg}
[Documentation] Return a value unconditionally.
... Notice that keywords after RETURN are not executed.
${value} = Convert To Upper Case ${arg}
RETURN ${value}
Fail Not executed
Return Three Values
[Documentation] Return multiple values.
RETURN a b c
Conditional Return
[Arguments] ${arg}
[Documentation] Return conditionally.
Log Before
IF ${arg} == 1
Log Returning!
RETURN
END
Log After
Find Index
[Arguments] ${test} ${items}
[Documentation] Advanced example involving FOR loop, inline IF and @{list} variable syntax.
FOR ${index} ${item} IN ENUMERATE @{items}
IF $item == $test RETURN ${index}
END
RETURN ${-1}
If you want to test the above examples yourself, you can use them with these test cases:
*** Settings ***
Library String
*** Test Cases ***
One return value
${ret} = Return One Value argument
Should Be Equal ${ret} ARGUMENT
Multiple return values
${a} ${b} ${c} = Return Three Values
Should Be Equal ${a}, ${b}, ${c} a, b, c
Conditional return
Conditional Return 1
Conditional Return 2
Advanced
@{list} = Create List foo bar baz
${index} = Find Index bar ${list}
Should Be Equal ${index} ${1}
${index} = Find Index non existing ${list}
Should Be Equal ${index} ${-1}
Note
RETURN
is new in Robot Framework 5.0. Use approaches explained
below if you need to support older versions.
The [Return] setting defines what the keyword should return after it has been executed. Although it is recommended to have it at the end of keyword where it logically belongs, its position does not affect how it is used.
An inherent limitation of the [Return] setting is that cannot be used
conditionally. Thus only the first two earlier RETURN
statement examples
can be created using it.
*** Keywords ***
Return One Value
[Arguments] ${arg}
${value} = Convert To Upper Case ${arg}
[Return] ${value}
Return Three Values
[Return] a b c
Note
The [Return] setting was deprecated in Robot Framework 7.0
and the RETURN
statement should be used instead. If there is a need
to support older Robot Framework versions that do not support RETURN
,
it is possible to use the special keywords discussed in the next section.
BuiltIn keywords Return From Keyword and Return From Keyword If
allow returning from a user keyword conditionally in the middle of the keyword.
Both of them also accept optional return values that are handled exactly like
with the RETURN
statement and the [Return] setting discussed above.
The introduction of the RETURN
statement makes these keywords redundant.
Examples below contain same keywords as earlier RETURN
examples but these
ones are more verbose:
*** Keywords ***
Return One Value
[Arguments] ${arg}
${value} = Convert To Upper Case ${arg}
Return From Keyword ${value}
Fail Not executed
Return Three Values
Return From Keyword a b c
Conditional Return
[Arguments] ${arg}
Log Before
IF ${arg} == 1
Log Returning!
Return From Keyword
END
Log After
Find Index
[Arguments] ${test} ${items}
FOR ${index} ${item} IN ENUMERATE @{items}
Return From Keyword If $item == $test ${index}
END
Return From Keyword ${-1}
Note
These keywords are effectively deprecated and the RETURN
statement should be
used unless there is a need to support also older versions than Robot Framework
5.0. There is no visible deprecation warning when using these keywords yet, but
they will be loudly deprecated and eventually removed in the future.
A user keyword can have a setup and a teardown similarly as tests. They are specified using [Setup] and [Teardown] settings, respectively, directly to the keyword having them. Unlike with tests, it is not possible to specify a common setup or teardown to all keywords in a certain file.
A setup and a teardown are always a single keyword, but they can themselves be
user keywords executing multiple keywords internally. It is possible to specify
them as variables, and using a special NONE
value (case-insensitive) is
the same as not having a setup or a teardown at all.
User keyword setup is not much different to the first keyword inside the created user keyword. The only functional difference is that a setup can be specified as a variable, but it can also be useful to be able to explicitly mark a keyword to be a setup.
User keyword teardowns are, exactly as test teardowns, executed also if the user keyword fails. They are thus very useful when needing to do something at the end of the keyword regardless of its status. To ensure that all cleanup activities are done, the continue on failure mode is enabled by default with user keyword teardowns the same way as with test teardowns.
*** Keywords ***
Setup and teardown
[Setup] Log New in RF 7!
Do Something
[Teardown] Log Old feature.
Using variables
[Setup] ${SETUP}
Do Something
[Teardown] ${TEARDOWN}
Note
User keyword setups are new in Robot Framework 7.0.
User keywords can be tagged with a special robot:private
tag to indicate
that they should only be used in the file where they are created:
*** Keywords ***
Public Keyword
Private Keyword
Private Keyword
[Tags] robot:private
No Operation
Using the robot:private
tag does not outright prevent using the keyword
outside the file where it is created, but such usages will cause a warning.
If there is both a public and a private keyword with the same name,
the public one will be used but also this situation causes a warning.
Private keywords are included in spec files created by Libdoc but not in its HTML output files.
Note
Private user keywords are new in Robot Framework 6.0.
User keywords and variables in suite files and suite initialization files can only be used in files where they are created, but resource files provide a mechanism for sharing them. The high level syntax for creating resource files is exactly the same as when creating suite files and supported file formats are the same as well. The main difference is that resource files cannot have tests.
Variable files provide a powerful mechanism for creating and sharing variables. For example, they allow values other than strings and enable creating variables dynamically. Their flexibility comes from the fact that they are created using Python or YAML, which also makes them somewhat more complicated than Variable sections.
Resource files are typically created using the plain text format, but also reStructuredText format and JSON format are supported.
Resource files are imported using the Resource setting in the Settings section so that the path to the resource file is given as an argument to the setting. The recommended extension for resource files is .resource. For backwards compatibility reasons also .robot, .txt and .tsv work, but using .resource may be mandated in the future.
If the resource file path is absolute, it is used directly. Otherwise, the resource file is first searched relatively to the directory where the importing file is located. If the file is not found there, it is then searched from the directories in Python's module search path. Searching resource files from the module search path makes it possible to bundle them into Python packages as package data and importing them like package/example.resource.
The resource file path can contain variables, and it is recommended to use
them to make paths system-independent (for example,
${RESOURCES}/login.resource or just ${RESOURCE_PATH}).
Additionally, forward slashes (/
) in the path
are automatically changed to backslashes (\) on Windows.
*** Settings ***
Resource example.resource
Resource ../resources/login.resource
Resource package/example.resource
Resource ${RESOURCES}/common.resource
The user keywords and variables defined in a resource file are available in the file that takes that resource file into use. Similarly available are also all keywords and variables from the libraries, resource files and variable files imported by the said resource file.
Note
The .resource extension is new in Robot Framework 3.1.
The higher-level structure of resource files is the same as that of suite files otherwise, but they cannot contain tests or tasks. Additionally, the Setting section in resource files can contain only imports (Library, Resource, Variables), Documentation and Keyword Tags. The Variable section and Keyword section are used exactly the same way as in suite files.
If several resource files have a user keyword with the same name, they must be used so that the keyword name is prefixed with the resource file name without the extension (for example, myresources.Some Keyword and common.Some Keyword). Moreover, if several resource files contain the same variable, the one that is imported first is taken into use.
Keywords created in a resource file can be documented using [Documentation] setting. The resource file itself can have Documentation in the Setting section similarly as suites.
Libdoc and various editors use these documentations, and they are naturally available for anyone opening resource files. The first logical line of the documentation of a keyword, until the first empty line, is logged when the keyword is run, but otherwise resource file documentation is ignored during the test execution.
*** Settings ***
Documentation An example resource file
Library SeleniumLibrary
Resource ${RESOURCES}/common.resource
*** Variables ***
${HOST} localhost:7272
${LOGIN URL} http://${HOST}/
${WELCOME URL} http://${HOST}/welcome.html
${BROWSER} Firefox
*** Keywords ***
Open Login Page
[Documentation] Opens browser to login page
Open Browser ${LOGIN URL} ${BROWSER}
Title Should Be Login Page
Input Name
[Arguments] ${name}
Input Text username_field ${name}
Input Password
[Arguments] ${password}
Input Text password_field ${password}
The reStructuredText format that can be used with suite files works also with resource files. Such resource files can use either .rst or .rest extension and they are otherwise imported exactly as normal resource files:
*** Settings ***
Resource example.rst
When parsing resource files using the reStructuredText format, Robot Framework
ignores all data outside code blocks containing Robot Framework data exactly
the same way as when parsing reStructuredText suite files.
For example, the following resource file imports OperatingSystem library,
defines ${MESSAGE}
variable and creates My Keyword keyword:
Resource file using reStructuredText
------------------------------------
This text is outside code blocks and thus ignored.
.. code:: robotframework
*** Settings ***
Library OperatingSystem
*** Variables ***
${MESSAGE} Hello, world!
Also this text is outside code blocks and ignored. Code blocks not
containing Robot Framework data are ignored as well.
.. code:: robotframework
# Both space and pipe separated formats are supported.
| *** Keywords *** | | |
| My Keyword | [Arguments] | ${path} |
| | Directory Should Exist | ${path} |
Resource files can be created using JSON the same way as suite files. Such JSON resource files must use either the standard .json extension or the custom .rsrc extension. They are otherwise imported exactly as normal resource files:
*** Settings ***
Resource example.rsrc
Resource files can be converted to JSON using ResourceFile.to_json and recreated using ResourceFile.from_json:
from robot.running import ResourceFile
# Create resource file based on data on the file system.
resource = ResourceFile.from_file_system('example.resource')
# Save JSON data to a file.
resource.to_json('example.rsrc')
# Recreate resource from JSON data.
resource = ResourceFile.from_json('example.rsrc')
Variable files contain variables that can be used in the test data. Variables can also be created using Variable sections or set from the command line, but variable files allow creating them dynamically and also make it easy to create other variable values than strings.
Variable files are typically implemented as modules and there are two different approaches for creating variables:
MY_VAR = 'my value'
creates a variable ${MY_VAR}
with the specified
text as its value. One limitation of this approach is that it does
not allow using arguments.get_variables
(or getVariables
) method that returns variables as a mapping.
Because the method can take arguments this approach is very flexible.Alternatively variable files can be implemented as classes
that the framework will instantiate. Also in this case it is possible to create
variables as attributes or get them dynamically from the get_variables
method. Variable files can also be created as YAML and JSON.
All test data files can import variable files using the Variables setting in the Setting section. Variable files are typically imported using a path to the file same way as resource files are imported using the Resource setting. Similarly to resource files, the path to the imported variable file is considered relative to the directory where the importing file is, and if not found, it is searched from directories in the module search path. The path can also contain variables, and slashes are converted to backslashes on Windows.
Examples:
*** Settings ***
Variables myvariables.py
Variables ../data/variables.py
Variables ${RESOURCES}/common.yaml
Starting from Robot Framework 5.0, variable files implemented using Python can also be imported using the module name similarly as libraries. When using this approach, the module needs to be in the module search path.
Examples:
*** Settings ***
Variables myvariables
Variables rootmodule.Variables
If a variable file accepts arguments, they are specified after the path or name of the variable file to import:
*** Settings ***
Variables arguments.py arg1 ${ARG2}
Variables arguments argument
All variables from a variable file are available in the test data file that imports it. If several variable files are imported and they contain a variable with the same name, the one in the earliest imported file is taken into use. Additionally, variables created in Variable sections and set from the command line override variables from variable files.
Another way to take variable files into use is using the command line option
--variablefile. Variable files are referenced using a path or
module name similarly as when importing them using the Variables
setting. Possible arguments are joined to the path with a colon (:
):
--variablefile myvariables.py --variablefile path/variables.py --variablefile /absolute/path/common.py --variablefile variablemodule --variablefile arguments.py:arg1:arg2 --variablefile rootmodule.Variables:arg1:arg2
Variable files taken into use from the command line are also searched from the module search path similarly as variable files imported in the Setting section. Relative paths are considered relative to the directory where execution is started from.
If a variable file is given as an absolute Windows path, the colon after the drive letter is not considered a separator:
--variablefile C:\path\variables.py
It is also possible to use a semicolon
(;
) as an argument separator. This is useful if variable file arguments
themselves contain colons, but requires surrounding the whole value with
quotes on UNIX-like operating systems:
--variablefile C:\path\variables.py;D:\data.xls --variablefile "myvariables.py;argument:with:colons"
Variables in variable files taken use on the command line are globally available in all test data files, similarly as individual variables set with the --variable option. If both --variablefile and --variable options are used and there are variables with same names, those that are set individually with --variable option take precedence.
When variable files are taken into use, they are imported as Python
modules and all their module level attributes that do not start with
an underscore (_
) are, by default, considered to be variables. Because
variable names are case-insensitive, both lower- and upper-case names are
possible, but in general, capital letters are recommended for global
variables and attributes.
VARIABLE = "An example string"
ANOTHER_VARIABLE = "This is pretty easy!"
INTEGER = 42
STRINGS = ["one", "two", "kolme", "four"]
NUMBERS = [1, INTEGER, 3.14]
MAPPING = {"one": 1, "two": 2, "three": 3}
In the example above, variables ${VARIABLE}
, ${ANOTHER VARIABLE}
, and
so on, are created. The first two variables are strings, the third one is
an integer, then there are two lists, and the final value is a dictionary.
All these variables can be used as a scalar variable, lists and the
dictionary also a list variable like @{STRINGS}
(in the dictionary's case
that variable would only contain keys), and the dictionary also as a
dictionary variable like &{MAPPING}
.
To make creating a list variable or a dictionary variable more explicit,
it is possible to prefix the variable name with LIST__
or DICT__
,
respectively:
from collections import OrderedDict
LIST__ANIMALS = ["cat", "dog"]
DICT__FINNISH = OrderedDict([("cat", "kissa"), ("dog", "koira")])
These prefixes will not be part of the final variable name, but they cause
Robot Framework to validate that the value actually is list-like or
dictionary-like. With dictionaries the actual stored value is also turned
into a special dictionary that is used also when creating dictionary
variables in the Variable section. Values of these dictionaries are accessible
as attributes like ${FINNISH.cat}
. These dictionaries are also ordered, but
preserving the source order requires also the original dictionary to be
ordered.
The variables in both the examples above could be created also using the Variable section below.
*** Variables ***
${VARIABLE} An example string
${ANOTHER VARIABLE} This is pretty easy!
${INTEGER} ${42}
@{STRINGS} one two kolme four
@{NUMBERS} ${1} ${INTEGER} ${3.14}
&{MAPPING} one=${1} two=${2} three=${3}
@{ANIMALS} cat dog
&{FINNISH} cat=kissa dog=koira
Note
Variables are not replaced in strings got from variable files.
For example, VAR = "an ${example}"
would create
variable ${VAR}
with a literal string value
an ${example}
regardless would variable ${example}
exist or not.
Variables in variable files are not limited to having only strings or
other base types as values like Variable sections. Instead, their
variables can contain any objects. In the example below, the variable
${MAPPING}
contains a Python dictionary and also has two variables
created from a custom object implemented in the same file.
MAPPING = {'one': 1, 'two': 2}
class MyObject:
def __init__(self, name):
self.name = name
OBJ1 = MyObject('John')
OBJ2 = MyObject('Jane')
Because variable files are created using a real programming language, they can have dynamic logic for setting variables.
import os
import random
import time
USER = os.getlogin() # current login name
RANDOM_INT = random.randint(0, 10) # random integer in range [0,10]
CURRENT_TIME = time.asctime() # timestamp like 'Thu Apr 6 12:45:21 2006'
if time.localtime()[3] > 12:
AFTERNOON = True
else:
AFTERNOON = False
The example above uses standard Python libraries to set different variables, but you can use your own code to construct the values. The example below illustrates the concept, but similarly, your code could read the data from a database, from an external file or even ask it from the user.
import math
def get_area(diameter):
radius = diameter / 2
area = math.pi * radius * radius
return area
AREA1 = get_area(1)
AREA2 = get_area(2)
When Robot Framework processes variable files, all their attributes
that do not start with an underscore are expected to be
variables. This means that even functions or classes created in the
variable file or imported from elsewhere are considered variables. For
example, the last example would contain the variables ${math}
and ${get_area}
in addition to ${AREA1}
and
${AREA2}
.
Normally the extra variables do not cause problems, but they could override some other variables and cause hard-to-debug errors. One possibility to ignore other attributes is prefixing them with an underscore:
import math as _math
def _get_area(diameter):
radius = diameter / 2.0
area = _math.pi * radius * radius
return area
AREA1 = _get_area(1)
AREA2 = _get_area(2)
If there is a large number of other attributes, instead of prefixing
them all, it is often easier to use a special attribute
__all__
and give it a list of attribute names to be processed
as variables.
import math
__all__ = ['AREA1', 'AREA2']
def get_area(diameter):
radius = diameter / 2.0
area = math.pi * radius * radius
return area
AREA1 = get_area(1)
AREA2 = get_area(2)
Note
The __all__
attribute is also, and originally, used
by Python to decide which attributes to import
when using the syntax from modulename import *
.
The third option to select what variables are actually created is using
a special get_variables
function discussed below.
An alternative approach for getting variables is having a special
get_variables
function (also camelCase syntax getVariables
is possible)
in a variable file. If such a function exists, Robot Framework calls it and
expects to receive variables as a Python dictionary with variable names as keys
and variable values as values. Created variables can
be used as scalars, lists, and dictionaries exactly like when getting
variables directly from a module, and it is possible to use LIST__
and
DICT__
prefixes to make creating list and dictionary variables more explicit.
The example below is functionally identical to the first example related to
getting variables directly from a module.
def get_variables():
variables = {"VARIABLE ": "An example string",
"ANOTHER VARIABLE": "This is pretty easy!",
"INTEGER": 42,
"STRINGS": ["one", "two", "kolme", "four"],
"NUMBERS": [1, 42, 3.14],
"MAPPING": {"one": 1, "two": 2, "three": 3}}
return variables
get_variables
can also take arguments, which facilitates changing
what variables actually are created. Arguments to the function are set just
as any other arguments for a Python function. When taking variable files
into use, arguments are specified after the path
to the variable file, and in the command line they are separated from the
path with a colon or a semicolon.
The dummy example below shows how to use arguments with variable files. In a more realistic example, the argument could be a path to an external text file or database where to read variables from.
variables1 = {'scalar': 'Scalar variable',
'LIST__list': ['List','variable']}
variables2 = {'scalar' : 'Some other value',
'LIST__list': ['Some','other','value'],
'extra': 'variables1 does not have this at all'}
def get_variables(arg):
if arg == 'one':
return variables1
else:
return variables2
Starting from Robot Framework 7.0, arguments to variable files support automatic
argument conversion as well as named argument syntax. For example, a variable
file with get_variables(first: int = 0, second: str = '')
could be imported
like this:
*** Settings ***
Variables example.py 42 # Converted to integer.
Variables example.py second=value # Named argument syntax.
It is possible to implement variables files also as a class.
Because variable files are always imported using a file system path, the class must have the same name as the module it is located in.
The framework will create an instance of the class using no arguments and
variables will be gotten from the instance. Similarly as with modules,
variables can be defined as attributes directly
in the instance or gotten from a special get_variables
method.
When variables are defined directly in an instance, all attributes containing callable values are ignored to avoid creating variables from possible methods the instance has. If you would actually need callable variables, you need to use other approaches to create variable files.
The first examples create variables from attributes.
It creates variables ${VARIABLE}
and @{LIST}
from class
attributes and ${ANOTHER VARIABLE}
from an instance attribute.
class StaticExample:
variable = 'value'
LIST__list = [1, 2, 3]
_not_variable = 'starts with an underscore'
def __init__(self):
self.another_variable = 'another value'
The second examples utilizes dynamic approach for getting variables. It
creates only one variable ${DYNAMIC VARIABLE}
.
class DynamicExample:
def get_variables(self, *args):
return {'dynamic variable': ' '.join(args)}
Variable files can also be implemented as YAML files. YAML is a data serialization language with a simple and human-friendly syntax that is nevertheless easy for machines to parse. The following example demonstrates a simple YAML file:
string: Hello, world!
integer: 42
list:
- one
- two
dict:
one: yksi
two: kaksi
with spaces: kolme
YAML variable files can be used exactly like normal variable files from the command line using --variablefile option, in the Settings section using Variables setting, and dynamically using the Import Variables keyword. They are automatically recognized by their extension that must be either .yaml or .yml. If the above YAML file is imported, it will create exactly the same variables as this Variable section:
*** Variables ***
${STRING} Hello, world!
${INTEGER} ${42}
@{LIST} one two
&{DICT} one=yksi two=kaksi with spaces=kolme
YAML files used as variable files must always be mappings on the top level. As the above example demonstrates, keys and values in the mapping become variable names and values, respectively. Variable values can be any data types supported by YAML syntax. If names or values contain non-ASCII characters, YAML variables files must be UTF-8 encoded.
Mappings used as values are automatically converted to special dictionaries
that are used also when creating dictionary variables in the Variable section.
Most importantly, values of these dictionaries are accessible as attributes
like ${DICT.one}
, assuming their names are valid as Python attribute names.
If the name contains spaces or is otherwise not a valid attribute name, it is
always possible to access dictionary values using syntax like
${DICT}[with spaces]
syntax.
Variable files can also be implemented as JSON files. Similarly as YAML discussed in the previous section, JSON is a data serialization format targeted both for humans and machines. It is based on JavaScript syntax and it is not as human-friendly as YAML, but it still relatively easy to understand and modify. The following example contains exactly the same data as the earlier YAML example:
{
"string": "Hello, world!",
"integer": 42,
"list": [
"one",
"two"
],
"dict": {
"one": "yksi",
"two": "kaksi",
"with spaces": "kolme"
}
}
JSON variable files are automatically recognized by their .json extension and they can be used exactly like YAML variable files. They also have exactly same requirements for structure, encoding, and so on. Unlike YAML, Python supports JSON out-of-the-box so no extra modules need to be installed.
Note
Support for JSON variable files is new in Robot Framework 6.1.
This section describes various structures that can be used to control the test execution flow. These structures are familiar from most programming languages and they allow conditional execution, repeatedly executing a block of keywords and fine-grained error handling. For readability reasons these structures should be used judiciously, and more complex use cases should be preferably implemented in test libraries.
FOR
loopsRepeating same actions several times is quite a common need in test
automation. With Robot Framework, test libraries can have any kind of
loop constructs, and most of the time loops should be implemented in
them. Robot Framework also has its own FOR
loop syntax, which is
useful, for example, when there is a need to repeat keywords from
different libraries.
FOR
loops can be used with both test cases and user keywords. Except for
really simple cases, user keywords are better, because they hide the
complexity introduced by FOR
loops. The basic FOR
loop syntax,
FOR item IN sequence
, is derived from Python, but similar
syntax is supported also by various other programming languages.
FOR
loopIn a normal FOR
loop, one variable is assigned based on a list of values,
one value per iteration. The syntax starts with FOR
(case-sensitive) as
a marker, then the loop variable, then a mandatory IN
(case-sensitive) as
a separator, and finally the values to iterate. These values can contain
variables, including list variables.
The keywords used in the FOR
loop are on the following rows and the loop
ends with END
(case-sensitive) on its own row. Keywords inside the loop
do not need to be indented, but that is highly recommended to make the syntax
easier to read.
*** Test Cases ***
Example
FOR ${animal} IN cat dog
Log ${animal}
Log 2nd keyword
END
Log Outside loop
Second Example
FOR ${var} IN one two ${3} four ${five}
... kuusi 7 eight nine ${last}
Log ${var}
END
The FOR
loop in Example above is executed twice, so that first
the loop variable ${animal}
has the value cat
and then
dog
. The loop consists of two Log keywords. In the
second example, loop values are split into two rows and the
loop is run altogether ten times.
It is often convenient to use FOR
loops with list variables. This is
illustrated by the example below, where @{ELEMENTS}
contains
an arbitrarily long list of elements and keyword Start Element is
used with all of them one by one.
*** Test Cases ***
Example
FOR ${element} IN @{ELEMENTS}
Start Element ${element}
END
FOR
loop syntaxPrior to Robot Framework 3.1, the FOR
loop syntax was different than nowadays.
The marker to start the loop was :FOR
instead of FOR
and loop contents needed
to be explicitly marked with a backslash instead of using the END
marker to end
the loop. The first example above would look like this using the old syntax:
*** Test Cases ***
Example
:FOR ${animal} IN cat dog
\ Log ${animal}
\ Log 2nd keyword
Log Outside loop
The old syntax was deprecated in Robot Framework 3.2 and the support for it was removed altogether in Robot Framework 4.0.
FOR
loopsStarting from Robot Framework 4.0, it is possible to use nested FOR
loops
simply by adding a loop inside another loop:
*** Keywords ***
Handle Table
[Arguments] @{table}
FOR ${row} IN @{table}
FOR ${cell} IN @{row}
Handle Cell ${cell}
END
END
There can be multiple nesting levels and loops can also be combined with other control structures:
*** Test Cases ***
Multiple nesting levels
FOR ${root} IN r1 r2
FOR ${child} IN c1 c2 c3
FOR ${grandchild} IN g1 g2
Log Many ${root} ${child} ${grandchild}
END
END
FOR ${sibling} IN s1 s2 s3
IF '${sibling}' != 's2'
Log Many ${root} ${sibling}
END
END
END
It is possible to iterate over multiple values in one iteration by using
multiple loop variables between the FOR
and IN
markers. There can be
any number of loop variables, but the number of values must be evenly
dividable by the number of variables. Each iteration consumes as many
values as there are variables.
If there are lot of values to iterate, it is often convenient to organize them below the loop variables, as in the first loop of the example below:
*** Test Cases ***
Multiple loop variables
FOR ${index} ${english} ${finnish} IN
... 1 cat kissa
... 2 dog koira
... 3 horse hevonen
Add Translation ${english} ${finnish} ${index}
END
FOR ${name} ${id} IN @{EMPLOYERS}
Create ${name} ${id}
END
FOR-IN-RANGE
loopAll FOR
loops in the previous section iterated over a sequence. That is the most
common use case, but sometimes it is convenient to have a loop that is executed
a certain number of times. For this purpose Robot Framework has a special
FOR index IN RANGE limit
loop syntax that is derived from the similar Python
idiom using the built-in range() function.
Similarly as other FOR
loops, the FOR-IN-RANGE
loop starts with
FOR
that is followed by a loop variable. In this format
there can be only one loop variable and it contains the current loop
index. After the variable there must be IN RANGE
marker (case-sensitive)
that is followed by loop limits.
In the simplest case, only the upper limit of the loop is specified. In this case, loop indices start from zero and increase by one until, but excluding, the limit. It is also possible to give both the start and end limits. Then indices start from the start limit, but increase similarly as in the simple case. Finally, it is possible to give also the step value that specifies the increment to use. If the step is negative, it is used as decrement.
It is possible to use simple arithmetic such as addition and subtraction with the range limits. This is especially useful when the limits are specified with variables. Start, end and step are typically given as integers, but using float values is possible as well.
*** Test Cases ***
Only upper limit
[Documentation] Loops over values from 0 to 9.
FOR ${index} IN RANGE 10
Log ${index}
END
Start and end
[Documentation] Loops over values from 1 to 10.
FOR ${index} IN RANGE 1 11
Log ${index}
END
Also step given
[Documentation] Loops over values 5, 15, and 25.
FOR ${index} IN RANGE 5 26 10
Log ${index}
END
Negative step
[Documentation] Loops over values 13, 3, and -7.
FOR ${index} IN RANGE 13 -13 -10
Log ${index}
END
Arithmetic
[Documentation] Arithmetic with variable.
FOR ${index} IN RANGE ${var} + 1
Log ${index}
END
Float parameters
[Documentation] Loops over values 3.14, 4.34, and 5.54.
FOR ${index} IN RANGE 3.14 6.09 1.2
Log ${index}
END
FOR-IN-ENUMERATE
loopSometimes it is useful to loop over a list and also keep track of your location
inside the list. Robot Framework has a special
FOR index ... IN ENUMERATE ...
syntax for this situation.
This syntax is derived from the Python built-in enumerate() function.
FOR-IN-ENUMERATE
loops syntax is just like the regular FOR
loop syntax,
except that the separator between variables and values is IN ENUMERATE
(case-sensitive). Typically they are used so that there is an additional index
variable before any other loop-variables. By default the index has a value 0
on the first iteration, 1
on the second, and so on.
For example, the following two test cases do the same thing:
*** Variables ***
@{LIST} a b c
*** Test Cases ***
Manage index manually
${index} = Set Variable -1
FOR ${item} IN @{LIST}
${index} = Evaluate ${index} + 1
My Keyword ${index} ${item}
END
FOR-IN-ENUMERATE
FOR ${index} ${item} IN ENUMERATE @{LIST}
My Keyword ${index} ${item}
END
Starting from Robot Framework 4.0, it is possible to specify a custom start index
by using start=<index>
syntax as the last item of the FOR ... IN ENUMERATE ...
header:
*** Variables ***
@{LIST} a b c
${START} 10
*** Test Cases ***
FOR-IN-ENUMERATE with start
FOR ${index} ${item} IN ENUMERATE @{LIST} start=1
My Keyword ${index} ${item}
END
Start as variable
FOR ${index} ${item} IN ENUMERATE @{LIST} start=${start}
My Keyword ${index} ${item}
END
The start=<index>
syntax must be explicitly used in the FOR
header and it cannot
itself come from a variable. If the last actual item to enumerate would start with
start=
, it needs to be escaped like start\=
.
Just like with regular FOR
loops, you can loop over multiple values per loop
iteration as long as the number of values in your list is evenly divisible by
the number of loop-variables (excluding the index variable):
*** Test Cases ***
FOR-IN-ENUMERATE with two values per iteration
FOR ${index} ${en} ${fi} IN ENUMERATE
... cat kissa
... dog koira
... horse hevonen
Log "${en}" in English is "${fi}" in Finnish (index: ${index})
END
If you only use one loop variable with FOR-IN-ENUMERATE
loops, that variable
will become a Python tuple containing the index and the iterated value:
*** Test Cases ***
FOR-IN-ENUMERATE with one loop variable
FOR ${x} IN ENUMERATE @{LIST}
Length Should Be ${x} 2
Log Index is ${x}[0] and item is ${x}[1].
END
Note
FOR-IN-ENUMERATE
loops with only one loop variable is a new
feature in Robot Framework 3.2.
FOR-IN-ZIP
loopSome tests build up several related lists, then loop over them together.
Robot Framework has a shortcut for this case: FOR ... IN ZIP ...
, which
is derived from the Python built-in zip() function.
This may be easiest to show with an example:
*** Variables ***
@{NUMBERS} ${1} ${2} ${5}
@{NAMES} one two five
*** Test Cases ***
Iterate over two lists manually
${length}= Get Length ${NUMBERS}
FOR ${index} IN RANGE ${length}
Log Many ${NUMBERS}[${index}] ${NAMES}[${index}]
END
FOR-IN-ZIP
FOR ${number} ${name} IN ZIP ${NUMBERS} ${NAMES}
Log Many ${number} ${name}
END
As the example above illustrates, FOR-IN-ZIP
loops require their own custom
separator IN ZIP
(case-sensitive) between loop variables and values.
Values used with FOR-IN-ZIP
loops must be lists or list-like objects.
Items to iterate over must always be given either as scalar variables like
${items}
or as list variables like @{lists}
that yield the actual
iterated lists. The former approach is more common and it was already
demonstrated above. The latter approach works like this:
*** Variables ***
@{NUMBERS} ${1} ${2} ${5}
@{NAMES} one two five
@{LISTS} ${NUMBERS} ${NAMES}
*** Test Cases ***
FOR-IN-ZIP with lists from variable
FOR ${number} ${name} IN ZIP @{LISTS}
Log Many ${number} ${name}
END
The number of lists to iterate over is not limited, but it must match the number of loop variables. Alternatively, there can be just one loop variable that then becomes a Python tuple getting items from all lists.
*** Variables ***
@{ABC} a b c
@{XYZ} x y z
@{NUM} 1 2 3
*** Test Cases ***
FOR-IN-ZIP with multiple lists
FOR ${a} ${x} ${n} IN ZIP ${ABC} ${XYZ} ${NUM}
Log Many ${a} ${x} ${n}
END
FOR-IN-ZIP with one variable
FOR ${items} IN ZIP ${ABC} ${XYZ} ${NUM}
Length Should Be ${items} 3
Log Many ${items}[0] ${items}[1] ${items}[2]
END
Starting from Robot Framework 6.1, it is possible to configure what to do if
lengths of the iterated items differ. By default, the shortest item defines how
many iterations there are and values at the end of longer ones are ignored.
This can be changed by using the mode
option that has three possible values:
STRICT
: Items must have equal lengths. If not, execution fails. This is
the same as using strict=True
with Python's zip function.SHORTEST
: Items in longer items are ignored. Infinite iterators are supported
in this mode as long as one of the items is exhausted. This is the default
behavior.LONGEST
: The longest item defines how many iterations there are. Missing
values in shorter items are filled-in with value specified using the fill
option or None
if it is not used. This is the same as using Python's
zip_longest function except that it has fillvalue
argument instead of
fill
.All these modes are illustrated by the following examples:
*** Variables ***
@{CHARACTERS} a b c d f
@{NUMBERS} 1 2 3
*** Test Cases ***
STRICT mode
[Documentation] This loop fails due to lists lengths being different.
FOR ${c} ${n} IN ZIP ${CHARACTERS} ${NUMBERS} mode=STRICT
Log ${c}: ${n}
END
SHORTEST mode
[Documentation] This loop executes three times.
FOR ${c} ${n} IN ZIP ${CHARACTERS} ${NUMBERS} mode=SHORTEST
Log ${c}: ${n}
END
LONGEST mode
[Documentation] This loop executes five times.
... On last two rounds `${n}` has value `None`.
FOR ${c} ${n} IN ZIP ${CHARACTERS} ${NUMBERS} mode=LONGEST
Log ${c}: ${n}
END
LONGEST mode with custom fill value
[Documentation] This loop executes five times.
... On last two rounds `${n}` has value `0`.
FOR ${c} ${n} IN ZIP ${CHARACTERS} ${NUMBERS} mode=LONGEST fill=0
Log ${c}: ${n}
END
Note
The behavior if list lengths differ will change in the future
so that the STRICT
mode will be the default. If that is not desired,
the SHORTEST
mode needs to be used explicitly.
Normal FOR
loops and FOR-IN-ENUMERATE
loops support iterating over keys
and values in dictionaries. This syntax requires at least one of the loop
values to be a dictionary variable.
It is possible to use multiple dictionary variables and to give additional
items in key=value
syntax. Items are iterated in the order they are defined
and if same key gets multiple values the last value will be used.
*** Variables ***
&{DICT} a=1 b=2 c=3
*** Test Cases ***
Dictionary iteration with FOR loop
FOR ${key} ${value} IN &{DICT}
Log Key is '${key}' and value is '${value}'.
END
Dictionary iteration with FOR-IN-ENUMERATE loop
FOR ${index} ${key} ${value} IN ENUMERATE &{DICT}
Log On round ${index} key is '${key}' and value is '${value}'.
END
Multiple dictionaries and extra items in 'key=value' syntax
&{more} = Create Dictionary e=5 f=6
FOR ${key} ${value} IN &{DICT} d=4 &{more} g=7
Log Key is '${key}' and value is '${value}'.
END
Typically it is easiest to use the dictionary iteration syntax so that keys
and values get separate variables like in the above examples. With normal FOR
loops it is also possible to use just a single variable that will become
a tuple containing the key and the value. If only one variable is used with
FOR-IN-ENUMERATE
loops, it becomes a tuple containing the index, the key and
the value. Two variables with FOR-IN-ENUMERATE
loops means assigning the index
to the first variable and making the second variable a tuple containing the key
and the value.
*** Test Cases ***
One loop variable
FOR ${item} IN &{DICT}
Log Key is '${item}[0]' and value is '${item}[1]'.
END
One loop variable with FOR-IN-ENUMERATE
FOR ${item} IN ENUMERATE &{DICT}
Log On round ${item}[0] key is '${item}[1]' and value is '${item}[2]'.
END
Two loop variables with FOR-IN-ENUMERATE
FOR ${index} ${item} IN ENUMERATE &{DICT}
Log On round ${index} key is '${item}[0]' and value is '${item}[1]'.
END
In addition to iterating over names and values in dictionaries, it is possible to iterate over keys and then possibly fetch the value based on it. This syntax requires using dictionaries as list variables:
*** Test Cases ***
Iterate over keys
FOR ${key} IN @{DICT}
Log Key is '${key}' and value is '${DICT}[${key}]'.
END
Note
Iterating over keys and values in dictionaries is a new feature in Robot Framework 3.2. With earlier version it is possible to iterate over dictionary keys like the last example above demonstrates.
FOR
loops with multiple iterations often create lots of output and
considerably increase the size of the generated output and log files.
It is possible to remove or flatten unnecessary keywords using
--removekeywords and --flattenkeywords command line options.
FOR
loops can be excessive in situations where there is only a need to
repeat a single keyword. In these cases it is often easier to use
BuiltIn keyword Repeat Keyword. This keyword takes a
keyword and how many times to repeat it as arguments. The times to
repeat the keyword can have an optional postfix times
or x
to make the syntax easier to read.
*** Test Cases ***
Example
Repeat Keyword 5 Some Keyword arg1 arg2
Repeat Keyword 42 times My Keyword
Repeat Keyword ${var} Another Keyword argument
WHILE loops
WHILE
loops combine features of FOR loops and IF/ELSE structures.
They specify a condition and repeat the loop body as long as the condition
remains true. This can be utilised, for example, to repeat a nondeterministic sequence
until the desired outcome happens, or in some cases they can be used as an
alternative to FOR loops.
Note
WHILE
loops are new in Robot Framework 5.0.
WHILE
syntax*** Test Cases ***
Example
VAR ${rc} 1
WHILE ${rc} != 0
${rc} = Keyword that returns zero on success
END
The WHILE
loop condition is evaluated in Python so that Python builtins like
len()
are available and modules are imported automatically to support usages
like math.pi * math.pow(${radius}, 2) < 10
.
Normal variables like ${rc}
in the above example are replaced before evaluation, but
variables are also available in the evaluation namespace using the special $rc
syntax.
The latter approach is handy when the string representation of the variable cannot be
used in the condition directly. For example, strings require quoting and multiline
strings and string themselves containing quotes cause additional problems. See the
Evaluating expressions appendix for more information and examples related to
the evaluation syntax.
Starting from Robot Framework 6.1, the condition in a WHILE
statement can be omitted.
This is interpreted as the condition always being true, which may be useful with the
limit
option described below.
WHILE
loop iterationsWith WHILE
loops, there is always a possibility to achieve an infinite loop,
either by intention or by mistake. This happens when the loop condition never
becomes false. While infinite loops have some utility in application programming,
in automation an infinite loop is rarely a desired outcome. If such a loop occurs
with Robot Framework, the execution must be forcefully stopped and no log or report
can be created. For this reason, WHILE
loops in Robot Framework have a default
limit of 10 000 iterations. If the limit is exceeded, the loop fails.
The limit can be set with the limit
configuration parameter either as a maximum
iteration count or as a maximum time for the whole loop. When the limit is an
iteration count, it is possible to use just integers like 100
and to add times
or x
suffix after the value like 100 times
. When the limit is a timeout,
it is possible to use time strings like 10 s
or 1 hour 10 minutes
.
The limit can also be disabled altogether by using NONE
(case-insensitive).
All these options are illustrated by the examples below.
*** Test Cases ***
Limit as iteration count
WHILE True limit=100
Log This is run 100 times.
END
WHILE True limit=10 times
Log This is run 10 times.
END
WHILE True limit=42x
Log This is run 42 times.
END
Limit as time
WHILE True limit=10 seconds
Log This is run 10 seconds.
END
No limit
WHILE True limit=NONE
Log This runs forever.
END
Note
Support for using times
and x
suffixes with iteration counts
is new in Robot Framework 7.0.
Keywords in a loop are not forcefully stopped if the limit is exceeded. Instead
the loop is exited similarly as if the loop condition would have become false.
A major difference is that the loop status will be FAIL
in this case.
Starting from Robot Framework 6.1, it is possible to use on_limit
parameter to
configure the behaviour when the limit is exceeded. It supports two values pass
and fail
, case insensitively. If the value is pass
, the execution will continue
normally when the limit is reached and the status of the WHILE
loop will be PASS
.
The value fail
works similarly as the default behaviour, e.g. the loop and the
test will fail if the limit is exceeded.
*** Test Cases ***
Continue when iteration limit is reached
WHILE True limit=5 on_limit=pass
Log Loop will be executed five times
END
Log This will be executed normally.
Continue when time limit is reached
WHILE True limit=10s on_limit=pass
Log Loop will be executed for 10 seconds.
Sleep 0.5s
END
Log This will be executed normally.
By default, the error message raised when the limit is reached is
WHILE loop was aborted because it did not finish within the limit of 0.5
seconds. Use the 'limit' argument to increase or remove the limit if
needed.
. Starting from Robot Framework 6.1, the error message can be changed
with the on_limit_message
configuration parameter.
*** Test Cases ***
Limit as iteration count
WHILE True limit=0.5s on_limit_message=Custom While loop error message
Log This is run 0.5 seconds.
END
Note
on_limit_message
configuration parameter is new in Robot Framework 6.1.
WHILE
loopsWHILE
loops can be nested and also combined with other control structures:
*** Test Cases ***
Nesting WHILE
${x} = Set Variable 10
WHILE ${x} > 0
${y} = Set Variable ${x}
WHILE ${y} > 0
${y} = Evaluate ${y} - 1
END
IF ${x} > 5
${x} = Evaluate ${x} - 1
ELSE
${x} = Evaluate ${x} - 2
END
END
WHILE
loops with multiple iterations often create lots of output and
considerably increase the size of the generated output and log files.
It is possible to remove or flatten unnecessary keywords using
--removekeywords and --flattenkeywords command line options.
BREAK
and CONTINUE
Both FOR and WHILE loop execution can be controlled with BREAK
and CONTINUE
statements. The former exits the whole loop prematurely and the latter stops
executing the current loop iteration and continues to the next one. In practice
they have the same semantics as break
and continue
statements in Python, Java,
and many other programming languages.
Both BREAK
and CONTINUE
are typically used conditionally with IF/ELSE
or TRY/EXCEPT structures, and especially the inline IF syntax is often
convenient with them. These statements must be used in the loop body,
possibly inside the aforementioned control structures, and using them in
keyword called in the loop body is invalid.
*** Test Cases ***
BREAK with FOR
${text} = Set Variable zero
FOR ${var} IN one two three
IF '${var}' == 'two' BREAK
${text} = Set Variable ${text}-${var}
END
Should Be Equal ${text} zero-one
CONTINUE with FOR
${text} = Set Variable zero
FOR ${var} IN one two three
IF '${var}' == 'two' CONTINUE
${text} = Set Variable ${text}-${var}
END
Should Be Equal ${text} zero-one-three
CONTINUE and BREAK with WHILE
WHILE True
TRY
${value} = Do Something
EXCEPT
CONTINUE
END
Do something with value ${value}
BREAK
END
Invalid BREAK usage
[Documentation] BREAK and CONTINUE can only be used in the loop body,
... not in keywords used in the loop.
FOR ${var} IN one two three
Invalid BREAK
END
*** Keywords ***
Invalid BREAK
[Documentation] This keyword fails due to invalid syntax.
BREAK
Note
BREAK
and CONTINUE
statements are new in Robot Framework 5.0 similarly
as WHILE
. Earlier versions supported controlling FOR
loops using
BuiltIn keywords Exit For Loop, Exit For Loop If,
Continue For Loop and Continue For Loop If. These
keywords still continue to work, but they will be deprecated and removed
in the future.
Note
Also the RETURN statement can be used to a exit loop. It only works when loops are used inside a user keyword.
IF/ELSE
syntaxSometimes there is a need to execute some keywords conditionally. Starting
from Robot Framework 4.0 there is a separate IF/ELSE
syntax, but
there are also other ways to execute keywords conditionally. Notice that if
the logic gets complicated, it is typically better to move it into a test library.
IF
syntaxRobot Framework's native IF
syntax starts with IF
(case-sensitive) and
ends with END
(case-sensitive). The IF
marker requires exactly one value that is
the condition to evaluate. Keywords to execute if the condition is true are on their
own rows between the IF
and END
markers. Indenting keywords in the IF
block is
highly recommended but not mandatory.
In the following example keywords Some keyword and Another keyword
are executed if ${rc}
is greater than zero:
*** Test Cases ***
Example
IF ${rc} > 0
Some keyword
Another keyword
END
The condition is evaluated in Python so that Python builtins like
len()
are available and modules are imported automatically to support usages like
platform.system() == 'Linux'
and math.ceil(${x}) == 1
.
Normal variables like ${rc}
in the above example are replaced before evaluation, but
variables are also available in the evaluation namespace using the special $rc
syntax.
The latter approach is handy when the string representation of the variable cannot be
used in the condition directly. For example, strings require quoting and multiline
strings and string themselves containing quotes cause additional problems. For more
information and examples related the evaluation syntax see the Evaluating expressions
appendix.
ELSE
branchesLike most other languages supporting conditional execution, Robot Framework IF
syntax also supports ELSE
branches that are executed if the IF
condition is
not true.
In this example Some keyword is executed if ${rc}
is greater than
zero and Another keyword is executed otherwise:
*** Test Cases ***
Example
IF ${rc} > 0
Some keyword
ELSE
Another keyword
END
ELSE IF
branchesRobot Framework also supports ELSE IF
branches that have their own condition
that is evaluated if the initial condition is not true. There can be any number
of ELSE IF
branches and they are gone through in the order they are specified.
If one of the ELSE IF
conditions is true, the block following it is executed
and remaining ELSE IF
branches are ignored. An optional ELSE
branch can follow
ELSE IF
branches and it is executed if all conditions are false.
In the following example different keyword is executed depending on is ${rc}
positive,
negative, zero, or something else like a string or None
:
*** Test Cases ***
Example
IF $rc > 0
Positive keyword
ELSE IF $rc < 0
Negative keyword
ELSE IF $rc == 0
Zero keyword
ELSE
Fail Unexpected rc: ${rc}
END
Notice that this example uses the ${rc}
variable in the special $rc
format to
avoid evaluation failures if it is not a number. See the aforementioned
Evaluating expressions appendix for more information about this syntax.
IF
Normal IF/ELSE
structure is a bit verbose if there is a need to execute only
a single statement. An alternative to it is using inline IF
syntax where
the statement to execute follows the IF
marker and condition directly and
no END
marker is needed. For example, the following two keywords are
equivalent:
*** Keywords ***
Normal IF
IF $condition1
Keyword argument
END
IF $condition2
RETURN
END
Inline IF
IF $condition1 Keyword argument
IF $condition2 RETURN
The inline IF
syntax supports also ELSE
and ELSE IF
branches:
*** Keywords ***
Inline IF/ELSE
IF $condition Keyword argument ELSE Another Keyword
Inline IF/ELSE IF/ELSE
IF $cond1 Keyword 1 ELSE IF $cond2 Keyword 2 ELSE IF $cond3 Keyword 3 ELSE Keyword 4
As the latter example above demonstrates, inline IF
with several ELSE IF
and ELSE
branches starts to get hard to understand. Long inline IF
structures can be split into multiple lines using the common ...
continuation syntax, but using a normal IF/ELSE
structure or moving the logic
into a test library is probably a better idea. Each inline IF
branch can
contain only one statement. If more statements are needed, normal IF/ELSE
structure needs to be used instead.
If there is a need for an assignment with inline IF
, the variable or variables
to assign must be before the starting IF
. Otherwise the logic is exactly
the same as when assigning variables based on keyword return values. If
assignment is used and no branch is run, the variable gets value None
.
*** Keywords ***
Inline IF/ELSE with assignment
${var} = IF $condition Keyword argument ELSE Another Keyword
Inline IF/ELSE with assignment having multiple variables
${host} ${port} = IF $production Get Production Config ELSE Get Testing Config
Note
Inline IF
syntax is new in Robot Framework 5.0.
IF
structuresIF
structures can be nested with each others and with FOR loops.
This is illustrated by the following example using advanced features such
as FOR-IN-ENUMERATE loop, named-only arguments with user keywords and
inline Python evaluation syntax (${{len(${items})}}
):
*** Keywords ***
Log items
[Arguments] @{items} ${log_values}=True
IF not ${items}
Log to console No items.
ELSE IF len(${items}) == 1
IF ${log_values}
Log to console One item: ${items}[0]
ELSE
Log to console One item.
END
ELSE
Log to console ${{len(${items})}} items.
IF ${log_values}
FOR ${index} ${item} IN ENUMERATE @{items} start=1
Log to console Item ${index}: ${item}
END
END
END
*** Test Cases ***
No items
Log items
One item without logging value
Log items xxx log_values=False
Multiple items
Log items a b c
There are also other methods to execute keywords conditionally:
IF/ELSE
syntax explained above is generally recommended, though.TRY/EXCEPT
syntaxWhen a keyword fails, Robot Framework's default behavior is to stop the current
test and executes its possible teardown. There can, however, be needs to handle
these failures during execution as well. Robot Framework 5.0 introduces native
TRY/EXCEPT
syntax for this purpose, but there also other ways to handle errors.
Robot Framework's TRY/EXCEPT
syntax is inspired by Python's exception handling
syntax. It has same TRY
, EXCEPT
, ELSE
and FINALLY
branches as Python and
they also mostly work the same way. A difference is that Python uses lower case
try
, except
, etc. but with Robot Framework all this kind of syntax must use
upper case letters. A bigger difference is that with Python exceptions are objects
and with Robot Framework you are dealing with error messages as strings.
EXCEPT
The basic TRY/EXCEPT
syntax can be used to handle failures based on
error messages:
*** Test Cases ***
First example
TRY
Some Keyword
EXCEPT Error message
Error Handler Keyword
END
Keyword Outside
In the above example, if Some Keyword
passes, the EXCEPT
branch is not run
and execution continues after the TRY/EXCEPT
structure. If the keyword fails
with a message Error message
(case-sensitive), the EXCEPT
branch is executed.
If the EXCEPT
branch succeeds, execution continues after the TRY/EXCEPT
structure. If it fails, the test fails and remaining keywords are not executed.
If Some Keyword
fails with any other exception, that failure is not handled
and the test fails without executing remaining keywords.
There can be more than one EXCEPT
branch. In that case they are matched one
by one and the first matching branch is executed. One EXCEPT
can also have
multiple messages to match, and such a branch is executed if any of its messages
match. In all these cases messages can be specified using variables in addition
to literal strings.
*** Test Cases ***
Multiple EXCEPT branches
TRY
Some Keyword
EXCEPT Error message # Try matching this first.
Error Handler 1
EXCEPT Another error # Try this if above did not match.
Error Handler 2
EXCEPT ${message} # Last match attempt, this time using a variable.
Error Handler 3
END
Multiple messages with one EXCEPT
TRY
Some Keyword
EXCEPT Error message Another error ${message} # Match any of these.
Error handler
END
It is also possible to have an EXCEPT
without messages, in which case it matches
any error. There can be only one such EXCEPT
and it must follow possible
other EXCEPT
branches:
*** Test Cases ***
Match any error
TRY
Some Keyword
EXCEPT # Match any error.
Error Handler
END
Match any after testing more specific errors
TRY
Some Keyword
EXCEPT Error message # Try matching this first
Error Handler 1
EXCEPT # Match any that did not match the above.
Error Handler 2
END
Note
It is not possible to catch exceptions caused by invalid syntax.
By default matching an error using EXCEPT
requires an exact match. That can be
changed using a configuration option type=
as an argument to the except clause.
Valid values for the option are GLOB
, REGEXP
or START
(case-insensitive)
to make the match a glob pattern match, a regular expression match, or
to match only the beginning of the error, respectively. Using value
LITERAL
has the same effect as the default behavior. If an EXCEPT
has multiple
messages, this option applies to all of them. The value of the option
can be defined with a variable as well.
*** Variables ***
${MATCH TYPE} regexp
*** Test Cases ***
Glob pattern
TRY
Some Keyword
EXCEPT ValueError: * type=GLOB
Error Handler 1
EXCEPT [Ee]rror ?? occurred ${pattern} type=glob
Error Handler 2
END
Regular expression
TRY
Some Keyword
EXCEPT ValueError: .* type=${MATCH TYPE}
Error Handler 1
EXCEPT [Ee]rror \\d+ occurred type=Regexp # Backslash needs to be escaped.
Error Handler 2
END
Match start
TRY
Some Keyword
EXCEPT ValueError: ${beginning} type=start
Error Handler
END
Explicit exact match
TRY
Some Keyword
EXCEPT ValueError: invalid literal for int() with base 10: 'ooops' type=LITERAL
Error Handler
EXCEPT Error 13 occurred type=LITERAL
Error Handler 2
END
Note
Remember that the backslash character often used with regular expressions is an escape character in Robot Framework data. It thus needs to be escaped with another backslash when using it in regular expressions.
When matching errors using patterns and when using EXCEPT
without any
messages to match any error, it is often useful to know the actual error that
occurred. Robot Framework supports that by making it possible to capture
the error message into a variable by adding AS ${var}
at the
end of the EXCEPT
statement:
*** Test Cases ***
Capture error
TRY
Some Keyword
EXCEPT ValueError: * type=GLOB AS ${error}
Error Handler 1 ${error}
EXCEPT [Ee]rror \\d+ (Invalid|Bad) usage type=REGEXP AS ${error}
Error Handler 2 ${error}
EXCEPT AS ${error}
Error Handler 3 ${error}
END
ELSE
to execute keywords when there are no errorsOptional ELSE
branches make it possible to execute keywords if there is no error.
There can be only one ELSE
branch and it is allowed only after one or more
EXCEPT
branches:
*** Test Cases ***
ELSE branch
TRY
Some Keyword
EXCEPT X
Log Error 'X' occurred!
EXCEPT Y
Log Error 'Y' occurred!
ELSE
Log No error occurred!
END
Keyword Outside
In the above example, if Some Keyword
passes, the ELSE
branch is executed,
and if it fails with message X
or Y
, the appropriate EXCEPT
branch run.
In all these cases execution continues after the whole TRY/EXCEPT/ELSE
structure.
If Some Keyword
fail any other way, EXCEPT
and ELSE
branches are not run
and the TRY/EXCEPT/ELSE
structure fails.
To handle both the case when there is any error and when there is no error,
it is possible to use an EXCEPT
without any message in combination with an ELSE
:
*** Test Cases ***
Handle everything
TRY
Some Keyword
EXCEPT AS ${err}
Log Error occurred: ${err}
ELSE
Log No error occurred!
END
FINALLY
to execute keywords regardless are there errors or notOptional FINALLY
branches make it possible to execute keywords both when there
is an error and when there is not. They are thus suitable for cleaning up
after a keyword execution somewhat similarly as teardowns. There can be only one
FINALLY
branch and it must always be last. They can be used in combination with
EXCEPT
and ELSE
branches and having also TRY/FINALLY
structure is possible:
*** Test Cases ***
TRY/EXCEPT/ELSE/FINALLY
TRY
Some keyword
EXCEPT
Log Error occurred!
ELSE
Log No error occurred.
FINALLY
Log Always executed.
END
TRY/FINALLY
Open Connection
TRY
Use Connection
FINALLY
Close Connection
END
There are also other methods to execute keywords conditionally:
TRY/EXCEPT
with a specified message. The syntax to specify
the error message is also identical except that this keyword uses glob pattern
matching, not exact match, by default. Using the native TRY/EXCEPT
functionality
is generally recommended unless there is a need to support older Robot Framework
versions that do not support it.PASS
or FAIL
along with possible return value
or error message. It is basically the same as using TRY/EXCEPT/ELSE
so that
EXCEPT
catches all errors. Using the native syntax is recommended unless
old Robot Framework versions need to be supported.FINALLY
branches.Keywords that are used with Robot Framework are either library keywords or user keywords. The former come from standard libraries or external libraries, and the latter are either created in the same file where they are used or then imported from resource files. When many keywords are in use, it is quite common that some of them have the same name, and this section describes how to handle possible conflicts in these situations.
When only a keyword name is used and there are several keywords with that name, Robot Framework attempts to determine which keyword has the highest priority based on its scope. The keyword's scope is determined on the basis of how the keyword in question is created:
Scopes alone are not a sufficient solution, because there can be keywords with the same name in several libraries or resources, and thus, they provide a mechanism to use only the keyword of the highest priority. In such cases, it is possible to use the full name of the keyword, where the keyword name is prefixed with the name of the resource or library and a dot is a delimiter.
With library keywords, the long format means only using the format LibraryName.Keyword Name. For example, the keyword Run from the OperatingSystem library could be used as OperatingSystem.Run, even if there was another Run keyword somewhere else. If the library is in a module or package, the full module or package name must be used (for example, com.company.Library.Some Keyword). If a custom name is given to a library when importing it, the specified name must be used also in the full keyword name.
Resource files are specified in the full keyword name, similarly as library names. The name of the resource is derived from the basename of the resource file without the file extension. For example, the keyword Example in a resource file myresources.html can be used as myresources.Example. Note that this syntax does not work, if several resource files have the same basename. In such cases, either the files or the keywords must be renamed. The full name of the keyword is case-, space- and underscore-insensitive, similarly as normal keyword names.
If there are multiple conflicts between keywords, specifying all the keywords in the long format can be quite a lot work. Using the long format also makes it impossible to create dynamic test cases or user keywords that work differently depending on which libraries or resources are available. A solution to both of these problems is specifying the keyword priorities explicitly using the keyword Set Library Search Order from the BuiltIn library.
Note
Although the keyword has the word library in its name, it works also with resource files. As discussed above, keywords in resources always have higher priority than keywords in libraries, though.
The Set Library Search Order accepts an ordered list or libraries and resources as arguments. When a keyword name in the test data matches multiple keywords, the first library or resource containing the keyword is selected and that keyword implementation used. If the keyword is not found from any of the specified libraries or resources, execution fails for conflict the same way as when the search order is not set.
For more information and examples, see the documentation of the keyword.
Sometimes keywords may take exceptionally long time to execute or just hang endlessly. Robot Framework allows you to set timeouts both for test cases and user keywords, and if a test or keyword is not finished within the specified time, the keyword that is currently being executed is forcefully stopped.
Stopping keywords in this manner may leave the library, the test environment or the system under test to an unstable state, and timeouts are recommended only when there is no safer option available. In general, libraries should be implemented so that keywords cannot hang or that they have their own timeout mechanism.
The test case timeout can be set either by using the Test Timeout setting in the Setting section or the [Timeout] setting with individual test cases. Test Timeout defines a default timeout for all the test cases in that suite, whereas [Timeout] applies a timeout to a particular test case and overrides the possible default value.
Using an empty [Timeout] means that the test has no timeout even
when Test Timeout is used. It is also possible to use explicit
NONE
value for this purpose. The timeout is effectively ignored also if
its value is zero or negative.
Regardless of where the test timeout is defined, the value given to it
contains the duration of the timeout. The duration must be given in Robot
Framework's time format, that is, either directly in seconds like 10
or in a format like 1 minute 30 seconds
. Timeouts can also be specified
as variables making it possible to give them, for example, from the command
line.
If there is a timeout and it expires, the keyword that is currently running is stopped and the test case fails. Keywords executed as part of test teardown are not interrupted if a test timeout occurs, though, but the test is nevertheless marked failed. If a keyword in teardown may hang, it can be stopped by using user keyword timeouts.
*** Settings ***
Test Timeout 2 minutes
*** Test Cases ***
Default timeout
[Documentation] Default timeout from Settings is used.
Some Keyword argument
Override
[Documentation] Override default, use 10 seconds timeout.
[Timeout] 10
Some Keyword argument
Variables
[Documentation] It is possible to use variables too.
[Timeout] ${TIMEOUT}
Some Keyword argument
No timeout
[Documentation] Empty timeout means no timeout even when Test Timeout has been used.
[Timeout]
Some Keyword argument
No timeout 2
[Documentation] Disabling timeout with NONE works too and is more explicit.
[Timeout] NONE
Some Keyword argument
Timeouts can be set for user keywords using the [Timeout] setting. The syntax is exactly the same as with test case timeout, but user keyword timeouts do not have any default value. If a user keyword timeout is specified using a variable, the value can be given also as a keyword argument.
*** Keywords ***
Hardcoded
[Arguments] ${arg}
[Timeout] 1 minute 42 seconds
Some Keyword ${arg}
Configurable
[Arguments] ${arg} ${timeout}
[Timeout] ${timeout}
Some Keyword ${arg}
Run Keyword with Timeout
[Arguments] ${keyword} @{args} &{kwargs} ${timeout}=1 minute
[Documentation] Wrapper that runs another keyword with a configurable timeout.
[Timeout] ${timeout}
Run Keyword ${keyword} @{args} &{kwargs}
A user keyword timeout is applicable during the execution of that user keyword. If the total time of the whole keyword is longer than the timeout value, the currently executed keyword is stopped. User keyword timeouts are applicable also during a test case teardown, whereas test timeouts are not.
If both the test case and some of its keywords (or several nested keywords) have a timeout, the active timeout is the one with the least time left.
Note
With earlier Robot Framework versions it was possible to specify a custom error message to use if a timeout expires. This functionality was deprecated in Robot Framework 3.0.1 and removed in Robot Framework 3.2.
When parallel execution is needed, it must be implemented in test library level so that the library executes the code on background. Typically this means that the library needs a keyword like Start Something that starts the execution and returns immediately, and another keyword like Get Results From Something that waits until the result is available and returns it. See Process library keywords Start Process and Wait For Process for an example.
Robot Framework test cases are executed from the command line, and the end result is, by default, an output file in XML format and an HTML report and log. After the execution, output files can be combined and otherwise post-processed with the Rebot tool.
robot [options] data python -m robot [options] data python path/to/robot/ [options] data
Execution is normally started using the robot command created as part of installation. Alternatively it is possible to execute the installed robot module using the selected Python interpreter. This is especially convenient if Robot Framework has been installed under multiple Python versions. Finally, if you know where the installed robot directory exists, it can be executed using Python as well.
Regardless of execution approach, the path (or paths) to the test data to be executed is given as an argument after the command. Additionally, different command line options can be used to alter the test execution or generated outputs in many ways.
Robot Framework test cases are created in files and directories, and they are executed by giving the path to the file or directory in question to the selected runner script. The path can be absolute or, more commonly, relative to the directory where tests are executed from. The given file or directory creates the top-level test suite, which, by default, gets its name from the file or directory name. Different execution possibilities are illustrated in the examples below. Note that in these examples, as well as in other examples in this section, only the robot script is used, but other execution approaches could be used similarly.
robot tests.robot robot path/to/my_tests/ robot c:\robot\tests.robot
Note
When executing a directory, all files and directories starting with a dot (.) or an underscore (_) are ignored and, by default, only files with the .robot extension executed. See the Selecting files to parse section for more details.
It is also possible to give paths to several test case files or directories at once, separated with spaces. In this case, Robot Framework creates the top-level test suite automatically, and the specified files and directories become its child test suites. The name of the created test suite is got from child suite names by concatenating them together with an ampersand (&) and spaces. For example, the name of the top-level suite in the first example below is My Tests & Your Tests. These automatically created names are often quite long and complicated. In most cases, it is thus better to use the --name option for overriding it, as in the second example below:
robot my_tests.robot your_tests.robot robot --name Example path/to/tests/pattern_*.robot
Starting from Robot Framework 6.1, it is also possible to define a test suite initialisation file for the automatically created top-level suite. The path to the init file is given similarly to the test case files:
robot __init__.robot my_tests.robot other_tests.robot
Robot Framework provides a number of command line options that can be used to control how test cases are executed and what outputs are generated. This section explains the option syntax, and what options actually exist. How they can be used is discussed elsewhere in this chapter.
When options are used, they must always be given between the runner script and the data sources. For example:
robot -L debug my_tests.robot robot --include smoke --variable HOST:10.0.0.42 path/to/tests/
Options always have a long name, such as --name, and the
most frequently needed options also have a short name, such as
-N. In addition to that, long options can be shortened as
long as they are unique. For example, --logle DEBUG
works,
while --lo log.html
does not, because the former matches only
--loglevel, but the latter matches several options. Short
and shortened options are practical when executing test cases
manually, but long options are recommended in start-up scripts,
because they are easier to understand.
The long option names are case-insensitive and hyphen-insensitive, which facilitates writing option names in an easy-to-read format. For example, --SuiteStatLevel and --suite-stat-level are equivalent to, but easier to read than, --suitestatlevel.
Note
Long options being hyphen-insensitive is new in Robot Framework 6.1.
Most of the options require a value, which is given after the option
name. Both short and long options accept the value separated
from the option name with a space, as in --include tag
or -i tag
. With long options, the separator can also be the
equals sign, for example --include=tag
, and with short options the
separator can be omitted, as in -itag
.
Some options can be specified several times. For example,
--variable VAR1:value --variable VAR2:another
sets two
variables. If the options that take only one value are used several
times, the value given last is effective.
Options accepting no values can be disabled by using the same option again
with no
prefix added or dropped. The last option has precedence regardless
of how many times options are used. For example, --dryrun --dryrun --nodryrun
--nostatusrc --statusrc
would not activate the dry-run mode and would return
normal status rc.
Many command line options take arguments as simple patterns. These glob-like patterns are matched according to the following rules:
*
matches any string, even an empty string.?
matches any single character.[abc]
matches one character in the bracket.[!abc]
matches one character not in the bracket.[a-z]
matches one character from the range in the bracket.[!a-z]
matches one character not from the range in the bracket./
and
\ and the newline character \n
are matches by the above
wildcards.Examples:
--test Example* # Matches tests with name starting 'Example'. --test Example[1-2] # Matches tests 'Example1' and 'Example2'. --include f?? # Matches tests with a tag that starts with 'f' is three characters long.
All matching in above examples is case, space and underscore insensitive.
For example, the second example would also match test named example 1
.
If the matched text happens to contain some of the wildcard characters and
they need to be matched literally, it is possible to do that by using
the [...]
syntax. The pattern [*]
matches the literal *
character,
[?]
matches ?
, and [[]
matches [
. Lone [
and ]
do not need to
be escaped.
Note
Support for brackets like [abc]
and [!a-z]
is new in
Robot Framework 3.1.
Most tag related options accept arguments as tag patterns. They support same
wildcards as simple patterns (e.g. examp??
, ex*le
), but they also support AND
,
OR
and NOT
operators explained below. These operators can be
used for combining two or more individual tags or patterns together.
AND
or &
The whole pattern matches if all individual patterns match. AND
and
&
are equivalent:
--include fooANDbar # Matches tests containing tags 'foo' and 'bar'. --exclude xx&yy&zz # Matches tests containing tags 'xx', 'yy', and 'zz'.
OR
The whole pattern matches if any individual pattern matches:
--include fooORbar # Matches tests containing either tag 'foo' or tag 'bar'. --exclude xxORyyORzz # Matches tests containing any of tags 'xx', 'yy', or 'zz'.
NOT
The whole pattern matches if the pattern on the left side matches but
the one on the right side does not. If used multiple times, none of
the patterns after the first NOT
must not match:
--include fooNOTbar # Matches tests containing tag 'foo' but not tag 'bar'. --exclude xxNOTyyNOTzz # Matches tests containing tag 'xx' but not tag 'yy' or tag 'zz'.
The pattern can also start with NOT
in which case the pattern matches if the pattern after NOT
does not match:
--include NOTfoo # Matches tests not containing tag 'foo' --include NOTfooANDbar # Matches tests not containing tags 'foo' and 'bar'
The above operators can also be used together. The operator precedence,
from highest to lowest, is AND
, OR
and NOT
:
--include xANDyORz # Matches tests containing either tags 'x' and 'y', or tag 'z'. --include xORyNOTz # Matches tests containing either tag 'x' or 'y', but not tag 'z'. --include xNOTyANDz # Matches tests containing tag 'x', but not tags 'y' and 'z'.
Although tag matching itself is case-insensitive, all operators are
case-sensitive and must be written with upper case letters. If tags themselves
happen to contain upper case AND
, OR
or NOT
, they need to specified
using lower case letters to avoid accidental operator usage:
--include port # Matches tests containing tag 'port', case-insensitively --include PORT # Matches tests containing tag 'P' or 'T', case-insensitively --exclude handoverORportNOTnotification
Environment variables ROBOT_OPTIONS and REBOT_OPTIONS can be used to specify default options for test execution and result post-processing, respectively. The options and their values must be defined as a space separated list and they are placed in front of any explicit options on the command line. The main use case for these environment variables is setting global default values for certain options to avoid the need to repeat them every time tests are run or Rebot used.
export ROBOT_OPTIONS="--outputdir results --tagdoc 'mytag:Example doc with spaces'"
robot tests.robot
export REBOT_OPTIONS="--reportbackground blue:red:yellow"
rebot --name example output.xml
The most visible output from test execution is the output displayed in the command line. All executed test suites and test cases, as well as their statuses, are shown there in real time. The example below shows the output from executing a simple test suite with only two test cases:
============================================================================== Example test suite ============================================================================== First test :: Possible test documentation | PASS | ------------------------------------------------------------------------------ Second test | FAIL | Error message is displayed here ============================================================================== Example test suite | FAIL | 2 tests, 1 passed, 1 failed ============================================================================== Output: /path/to/output.xml Report: /path/to/report.html Log: /path/to/log.html
There is also a notification on the console whenever a top-level keyword in a test case ends. A green dot is used if a keyword passes and a red F if it fails. These markers are written to the end of line and they are overwritten by the test status when the test itself ends. Writing the markers is disabled if console output is redirected to a file.
The command line output is very limited, and separate output files are normally needed for investigating the test results. As the example above shows, three output files are generated by default. The first one is in XML format and contains all the information about test execution. The second is a higher-level report and the third is a more detailed log file. These files and other possible output files are discussed in more detail in the section Different output files.
Runner scripts communicate the overall test execution status to the system running them using return codes. When the execution starts successfully and no tests fail, the return code is zero. All possible return codes are explained in the table below.
RC | Explanation |
---|---|
0 | All tests passed. |
1-249 | Returned number of tests failed. |
250 | 250 or more failures. |
251 | Help or version information printed. |
252 | Invalid test data or command line options. |
253 | Test execution stopped by user. |
255 | Unexpected internal error. |
Return codes should always be easily available after the execution,
which makes it easy to automatically determine the overall execution
status. For example, in bash shell the return code is in special
variable $?
, and in Windows it is in %ERRORLEVEL%
variable. If you use some external tool for running tests, consult its
documentation for how to get the return code.
The return code can be set to 0 even if there are failures using the --NoStatusRC command line option. This might be useful, for example, in continuous integration servers where post-processing of results is needed before the overall status of test execution can be determined.
Note
Same return codes are also used with Rebot.
During the test execution there can be unexpected problems like failing to import a library or a resource file or a keyword being deprecated. Depending on the severity such problems are categorized as errors or warnings and they are written into the console (using the standard error stream), shown on a separate Test Execution Errors section in log files, and also written into Robot Framework's own system log. Normally these errors and warnings are generated by Robot Framework itself, but libraries can also log errors and warnings. Example below illustrates how errors and warnings look like in the log file.
Argument files allow placing all or some command line options and arguments into an external file where they will be read. This avoids the problems with characters that are problematic on the command line. If lot of options or arguments are needed, argument files also prevent the command that is used on the command line growing too long.
Argument files are taken into use with --argumentfile (-A) option along with possible other command line options.
Note
Unlike other long command line options, --argumentfile cannot be given in shortened format like --argumentf.
Argument files can contain both command line options and paths to the test data, one option or data source per line. Both short and long options are supported, but the latter are recommended because they are easier to understand. Argument files can contain any characters without escaping, but spaces in the beginning and end of lines are ignored. Additionally, empty lines and lines starting with a hash mark (#) are ignored:
--doc This is an example (where "special characters" are ok!) --metadata X:Value with spaces --variable VAR:Hello, world! # This is a comment path/to/my/tests
In the above example the separator between options and their values is a single space. It is possible to use either an equal sign (=) or any number of spaces. As an example, the following three lines are identical:
--name An Example --name=An Example --name An Example
If argument files contain non-ASCII characters, they must be saved using UTF-8 encoding.
Argument files can be used either alone so that they contain all the options and paths to the test data, or along with other options and paths. When an argument file is used with other arguments, its contents are placed into the original list of arguments to the same place where the argument file option was. This means that options in argument files can override options before it, and its options can be overridden by options after it. It is possible to use --argumentfile option multiple times or even recursively:
robot --argumentfile all_arguments.robot robot --name Example --argumentfile other_options_and_paths.robot robot --argumentfile default_options.txt --name Example my_tests.robot robot -A first.txt -A second.txt -A third.txt tests.robot
Special argument file name STDIN
can be used to read arguments from the
standard input stream instead of a file. This can be useful when generating
arguments with a script:
generate_arguments.sh | robot --argumentfile STDIN generate_arguments.sh | robot --name Example --argumentfile STDIN tests.robot
Both when executing test cases and when post-processing outputs, it is possible to get command line help with the option --help (-h). These help texts have a short general overview and briefly explain the available command line options.
All runner scripts also support getting the version information with the option --version. This information also contains Python version and the platform type:
$ robot --version Robot Framework 7.0 (Python 3.12.1 on darwin) C:\>rebot --version Rebot 6.1.1 (Python 3.11.0 on win32)
Test cases are often executed automatically by a continuous integration system or some other mechanism. In such cases, there is a need to have a script for starting the test execution, and possibly also for post-processing outputs somehow. Similar scripts are also useful when running tests manually, especially if a large number of command line options are needed or setting up the test environment is complicated.
In UNIX-like environments, shell scripts provide a simple but powerful mechanism for creating custom start-up scripts. Windows batch files can also be used, but they are more limited and often also more complicated. A platform-independent alternative is using Python or some other high-level programming language. Regardless of the language, it is recommended that long option names are used, because they are easier to understand than the short names.
In this example, the same web tests in the login directory are executed with different browsers and the results combined afterwards using Rebot. The script also accepts command line options itself and simply forwards them to the robot command using the handy $* variable:
#!/bin/bash
robot --name Firefox --variable BROWSER:Firefox --output out/fx.xml --log none --report none $* login
robot --name IE --variable BROWSER:IE --output out/ie.xml --log none --report none $* login
rebot --name Login --outputdir out --output login.xml out/fx.xml out/ie.xml
Implementing the above shell script example using batch files is not very complicated either. Notice that arguments to batch files can be forwarded to executed commands using %*:
@echo off
robot --name Firefox --variable BROWSER:Firefox --output out\fx.xml --log none --report none %* login
robot --name IE --variable BROWSER:IE --log none --output out\ie.xml --report none %* login
rebot --name Login --outputdir out --output login.xml out\fx.xml out\ie.xml
Note
Prior to Robot Framework 3.1 robot and rebot commands were implemented as batch files on Windows and using them in another batch file required prefixing the whole command with call.
When start-up scripts gets more complicated, implementing them using shell scripts or batch files is not that convenient. This is especially true if both variants are needed and same logic needs to be implemented twice. In such situations it is often better to switch to Python. It is possible to execute Robot Framework from Python using the subprocess module, but often using Robot Framework's own programmatic API is more convenient. The easiest APIs to use are robot.run_cli and robot.rebot_cli that accept same command line arguments than the robot and rebot commands.
The following example implements the same logic as the earlier shell script and batch file examples. In Python arguments to the script itself are available in sys.argv:
#!/usr/bin/env python
import sys
from robot import run_cli, rebot_cli
common = ['--log', 'none', '--report', 'none'] + sys.argv[1:] + ['login']
run_cli(['--name', 'Firefox', '--variable', 'BROWSER:Firefox', '--output', 'out/fx.xml'] + common, exit=False)
run_cli(['--name', 'IE', '--variable', 'BROWSER:IE', '--output', 'out/ie.xml'] + common, exit=False)
rebot_cli(['--name', 'Login', '--outputdir', 'out', 'out/fx.xml', 'out/ie.xml'])
Note
exit=False is needed because by default run_cli exits to system with the correct return code. rebot_cli does that too, but in the above example that is fine.
On UNIX-like operating systems it is possible to make *.robot files executable by giving them execution permission and adding a shebang like in this example:
#!/usr/bin/env robot
*** Test Cases ***
Example
Log to console Executing!
If the above content would be in a file example.robot and that file would be executable, it could be executed from the command line like below. Starting from Robot Framework 3.2, individually executed files can have any extension, or no extension at all, so the same would work also if the file would be named just example.
./example.robot
This trick does not work when executing a directory but can be handy when executing a single file. It is probably more often useful when automating tasks than when automating tests.
A test case can fail because the system under test does not work correctly, in which case the test has found a bug, or because the test itself is buggy. The error message explaining the failure is shown on the command line output and in the report file, and sometimes the error message alone is enough to pinpoint the problem. More often that not, however, log files are needed because they have also other log messages and they show which keyword actually failed.
When a failure is caused by the tested application, the error message and log messages ought to be enough to understand what caused it. If that is not the case, the test library does not provide enough information and needs to be enhanced. In this situation running the same test manually, if possible, may also reveal more information about the issue.
Failures caused by test cases themselves or by keywords they use can sometimes be hard to debug. If the error message, for example, tells that a keyword is used with wrong number of arguments fixing the problem is obviously easy, but if a keyword is missing or fails in unexpected way finding the root cause can be harder. The first place to look for more information is the execution errors section in the log file. For example, an error about a failed test library import may well explain why a test has failed due to a missing keyword.
If the log file does not provide enough information by default, it is
possible to execute tests with a lower log level. For example
tracebacks showing where in the code the failure occurred are logged
using the DEBUG
level, and this information is invaluable when
the problem is in an individual library keyword.
Logged tracebacks do not contain information about methods inside Robot Framework itself. If you suspect an error is caused by a bug in the framework, you can enable showing internal traces by setting environment variable ROBOT_INTERNAL_TRACES to any non-empty value.
If the log file still does not have enough information, it is a good idea to enable the syslog and see what information it provides. It is also possible to add some keywords to the test cases to see what is going on. Especially BuiltIn keywords Log and Log Variables are useful. If nothing else works, it is always possible to search help from mailing lists or elsewhere.
It is also possible to use the pdb module from the Python standard library to set a break point and interactively debug a running test. The typical way of invoking pdb by inserting:
import pdb; pdb.set_trace()
at the location you want to break into debugger will not work correctly with Robot Framework, as the standard output stream is redirected during keyword execution. Instead, you can use the following:
import sys, pdb; pdb.Pdb(stdout=sys.__stdout__).set_trace()
from within a python library or alternatively:
Evaluate pdb.Pdb(stdout=sys.__stdout__).set_trace() modules=sys, pdb
can be used directly in a test case.
This section describes how the test suite structure created from the parsed test data is executed, how test status is determined, and how to continue executing a test case if there are failures, and how to stop the whole test execution gracefully.
Test cases are always executed within a test suite. A test suite created from a suite file has tests directly, whereas suites created from directories have child test suites which either have tests or their own child suites. By default all the tests in an executed suite are run, but it is possible to select tests using options --test, --suite, --include and --exclude. Suites containing no tests are ignored.
The execution starts from the top-level test suite. If the suite has tests they are executed one-by-one, and if it has suites they are executed recursively in depth-first order. When an individual test case is executed, the keywords it contains are run in a sequence. Normally the execution of the current test ends if any of the keywords fails, but it is also possible to continue after failures. The exact execution order and how possible setups and teardowns affect the execution are discussed in the following sections.
Setups and teardowns can be used on test suite, test case and user keyword levels.
If a test suite has a setup, it is executed before its tests and child suites. If the suite setup passes, test execution continues normally. If it fails, all the test cases the suite and its child suites contain are marked failed. The tests and possible suite setups and teardowns in the child test suites are not executed.
Suite setups are often used for setting up the test environment. Because tests are not run if the suite setup fails, it is easy to use suite setups for verifying that the environment is in state in which the tests can be executed.
If a test suite has a teardown, it is executed after all its test cases and child suites. Suite teardowns are executed regardless of the test status and even if the matching suite setup fails. If the suite teardown fails, all tests in the suite are marked failed afterwards in reports and logs.
Suite teardowns are mostly used for cleaning up the test environment after the execution. To ensure that all these tasks are done, all the keywords used in the teardown are executed even if some of them fail.
Possible test setup is executed before the keywords of the test case. If the setup fails, the keywords are not executed. The main use for test setups is setting up the environment for that particular test case.
Possible test teardown is executed after the test case has been executed. It is executed regardless of the test status and also if test setup has failed.
Similarly as suite teardown, test teardowns are used mainly for cleanup activities. Also they are executed fully even if some of their keywords fail.
User keyword setup is executed before the keyword body. If the setup fails, the body is not executed. There is not much difference between the keyword setup and the first keyword in the body.
Note
User keyword setups are new in Robot Framework 7.0.
User keyword teardown is run after the keyword is executed otherwise, regardless the status. User keyword teardowns are executed fully even if some of their keywords would fail.
Test cases in a test suite are executed in the same order as they are defined in the test case file. Test suites inside a higher level test suite are executed in case-insensitive alphabetical order based on the file or directory name. If multiple files and/or directories are given from the command line, they are executed in the order they are given.
If there is a need to use certain test suite execution order inside a directory, it is possible to add prefixes like 01 and 02 into file and directory names. Such prefixes are not included in the generated test suite name if they are separated from the base name of the suite with two underscores:
01__my_suite.robot -> My Suite 02__another_suite.robot -> Another Suite
If the alphabetical ordering of test suites inside suites is problematic, a good workaround is giving them separately in the required order. This easily leads to overly long start-up commands, but argument files allow listing files nicely one file per line.
It is also possible to randomize the execution order using the --randomize option.
This section explains how tests can get PASS, FAIL or SKIP status and how the suite status is determined based on test statuses.
Note
The SKIP status is new in Robot Framework 4.0.
A test gets the PASS status if it is executed and none of the keywords it contains fails.
Normally all keywords are executed, but it is also possible to use BuiltIn keywords Pass Execution and Pass Execution If to stop execution with the PASS status and not run the remaining keywords.
How Pass Execution and Pass Execution If behave in different situations is explained below:
Passing execution in the middle of a test, setup or teardown should be used with care. In the worst case it leads to tests that skip all the parts that could actually uncover problems in the tested application. In cases where execution cannot continue do to external factors, it is often safer to skip the test.
The most common reason for a test to get the FAIL status is that one of the keywords it contains fails. The keyword itself can fail by raising an exception or the keyword can be called incorrectly. Other reasons for failures include syntax errors and the test being empty.
If a suite setup fails, tests in that suite are marked failed without running them. If a suite teardown fails, tests are marked failed retroactively.
Starting from Robot Framework 4.0, tests can get also SKIP status in addition to PASS and FAIL. There are many different ways to get this status.
The command line option --skip can be used to skip specified tests without
running them at all. It works based on tags and supports tag patterns like
examp??
and tagANDanother
. If it is used multiple times, all tests matching any of
specified tags or tag patterns are skipped:
--skip require-network --skip windowsANDversion9? --skip python2.* --skip python3.[0-6]
Starting from Robot Framework 5.0, a test case can also be skipped by tagging
the test with the reserved tag robot:skip
:
*** Test Cases ***
Example
[Tags] robot:skip
Log This is not executed
The difference between --skip and --exclude is that with the latter tests are omitted from the execution altogether and they will not be shown in logs and reports. With the former they are included, but not actually executed, and they will be visible in logs and reports.
Tests can get the skip status during execution in various ways:
The command line option --skiponfailure can be used to automatically mark failed tests skipped. It works based on tags and supports tag patterns like the --skip option discussed above:
--skiponfailure not-ready --skiponfailure experimentalANDmobile
Starting from RF 5.0, the reserved tag robot:skip-on-failure
can alternatively be used to
achieve the same effect as above:
*** Test Cases ***
Example
[Tags] robot:skip-on-failure
Fail this test will be marked as skipped instead of failed
The motivation for this functionality is allowing execution of tests that are not yet ready or that are testing a functionality that is not yet ready. Instead of such tests failing, they will be marked skipped and their tags can be used to separate them from possible other skipped tests.
Earlier Robot Framework versions supported criticality concept that allowed marking tests critical or non-critical. By default all tests were critical, but the --critical and --noncritical options could be used to configure that. The difference between critical and non-critical tests was that non-critical tests were not included when determining the final status for an executed test suite or for the whole test run. In practice the test status was two dimensional having PASS and FAIL in one axis and criticality on the other.
Non-critical failed tests were in many ways similar to the current skipped tests. Because these features are similar and having both SKIP and criticality would have created strange test statuses like non-critical SKIP, the criticality concept was removed in Robot Framework 4.0 when the SKIP status was introduced. The problems with criticality are explained in more detail in the issue that proposed removing it.
The main use case for the criticality concept was being able to run tests that are not yet ready or that are testing a functionality that is not yet ready. This use case is nowadays covered by the skip-on-failure functionality discussed in the previous section.
To ease migrating from criticality to skipping, the old --noncritical option worked as an alias for the new --skiponfailure in Robot Framework 4.0 and also the old --critical option was preserved. Both old options were deprecated and they were removed in Robot Framework 5.0.
Suite status is determined solely based on statuses of the tests it contains:
Normally test cases are stopped immediately when any of their keywords fail. This behavior shortens test execution time and prevents subsequent keywords hanging or otherwise causing problems if the system under test is in unstable state. This has a drawback that often subsequent keywords would give more information about the state of the system, though, and in some cases those subsequent keywords would actually take care of the needed cleanup activities. Hence Robot Framework offers several features to continue even if there are failures.
To make it sure that all the cleanup activities are taken care of, the continue-on-failure mode is automatically enabled in suite, test and keyword teardowns. In practice this means that in teardowns all the keywords in all levels are always executed.
If this behavior is not desired, the special robot:stop-on-failure
and
robot:recursive-stop-on-failure
tags can be used to disable it.
When using test templates, all the top-level keywords are executed to make it sure that all the different combinations are covered. In this usage continuing is limited to the top-level keywords, and inside them the execution ends normally if there are non-continuable failures.
*** Test Cases ***
Continue with templates
[Template] Should be Equal
this fails
this is run
If this behavior is not desired, the special robot:stop-on-failure
and
robot:recursive-stop-on-failure
tags can be used to disable it.
Library keywords report failures using exceptions, and it is possible to use special exceptions to tell Robot Framework that execution can continue regardless the failure. How these exceptions can be created is explained in the Continuable failures section in the Creating test libraries section.
When a test ends and there have been continuable failures, the test will be marked failed. If there are more than one failure, all of them will be enumerated in the final error message:
Several failures occurred: 1) First error message. 2) Second error message.
Test execution ends also if a normal failure occurs after a continuable failure. Also in that case all the failures will be listed in the final error message.
The return value from failed keywords, possibly assigned to a
variable, is always the Python None
.
BuiltIn keyword Run Keyword And Continue On Failure allows converting any failure into a continuable failure. These failures are handled by the framework exactly the same way as continuable failures originating from library keywords discussed above.
*** Test Cases ***
Example
Run Keyword and Continue on Failure Should be Equal 1 2
Log This is executed but test fails in the end
All keywords executed as part of test cases or user keywords which are
tagged with the robot:continue-on-failure
tag are considered continuable
by default. For example, the following two tests behave identically:
*** Test Cases ***
Test 1
Run Keyword and Continue on Failure Should be Equal 1 2
User Keyword 1
Test 2
[Tags] robot:continue-on-failure
Should be Equal 1 2
User Keyword 2
*** Keywords ***
User Keyword 1
Run Keyword and Continue on Failure Should be Equal 3 4
Log This is executed
User Keyword 2
[Tags] robot:continue-on-failure
Should be Equal 3 4
Log This is executed
These tags also affect the continue-on-failure mode with different control structures. For example, the below test case will execute the Do Something keyword ten times regardless does it succeed or not:
*** Test Cases ***
Example
[Tags] robot:continue-on-failure
FOR ${index} IN RANGE 10
Do Something
END
Setting robot:continue-on-failure
within a test case or a user keyword
will not propagate the continue-on-failure behavior into user keywords
they call. If such recursive behavior is needed, the
robot:recursive-continue-on-failure
tag can be used. For example, all
keywords in the following example are executed:
*** Test Cases ***
Example
[Tags] robot:recursive-continue-on-failure
Should be Equal 1 2
User Keyword 1
Log This is executed
*** Keywords ***
User Keyword 1
Should be Equal 3 4
User Keyword 2
Log This is executed
User Keyword 2
Should be Equal 5 6
Log This is executed
Setting robot:continue-on-failure
or robot:recursive-continue-on-failure
in a
test case does NOT alter the behaviour of a failure in the keyword(s) executed
as part of the [Setup]: The test case is marked as failed and no
test case keywords are executed.
Note
The robot:continue-on-failure
and robot:recursive-continue-on-failure
tags are new in Robot Framework 4.1. They do not work properly with
WHILE
loops prior to Robot Framework 6.0.
Special tags robot:stop-on-failure
and robot:recursive-stop-on-failure
can be used to disable the continue-on-failure mode if needed. They work
when continue-on-failure has been enabled using tags and also with
teardowns and templates:
*** Test Cases ***
Disable continue-in-failure set using tags
[Tags] robot:recursive-continue-on-failure
Keyword
Keyword # This is executed
Disable continue-in-failure in teardown
No Operation
[Teardown] Keyword
Disable continue-in-failure with templates
[Tags] robot:stop-on-failure
[Template] Should be Equal
this fails
this is not run
*** Keywords ***
Keyword
[Tags] robot:stop-on-failure
Should be Equal this fails
Should be Equal this is not run
The robot:stop-on-failure
tag affects only test cases and user keywords
where it is used and does not propagate to user keywords they call nor to
their own teardowns. If recursive behavior affecting all called user keywords
and teardowns is desired, the robot:recursive-stop-on-failure
tag can be
used instead. If there is a need, its effect can again be disabled in lower
level keywords by using robot:continue-on-failure
or
robot:recursive-continue-on-failure
tags.
The robot:stop-on-failure
and robot:recursive-stop-on-failure
tags do not
alter the behavior of continuable failures caused by library keywords or
by Run Keyword And Continue On Failure. For example, both keywords in this
example are run even though robot:stop-on-failure
is used:
*** Test Cases ***
Example
[Tags] robot:stop-on-failure
Run Keyword and Continue on Failure Should be Equal 1 2
Log This is executed regardless the tag
If robot:recursive-stop-on-failure
and robot:continue-on-failure
are used
together in the same test or keyword, execution is stopped in called keywords
if there are failures, but continues in the test or keyword using these tags.
If robot:recursive-continue-on-failure
and robot:stop-on-failure
are used
together in the same test or keyword, execution is continued in called keywords
if there are failures, but stopped in the test or keyword using these tags.
Note
The robot:stop-on-failure
and robot:recursive-stop-on-failure
tags are new in Robot Framework 6.0.
Note
Using recursive and non-recursive tags together in same test or keyword is new in Robot Framework 7.0.
Robot Framework 5.0 introduced native TRY/EXCEPT
syntax that can be used for
handling failures:
*** Test Cases ***
Example
TRY
Some Keyword
EXCEPT Expected error message
Error Handler Keyword
END
For more details see the separate TRY/EXCEPT syntax section.
There are several BuiltIn keywords that can be used to execute other keywords so that execution can continue after possible failures:
TRY/EXCEPT
syntax is
nowadays generally recommended instead.TRY/EXCEPT
syntax generally works better in this case
as well.True
or False
depending on did it pass or fail.Sometimes there is a need to stop the test execution before all the tests have finished, but so that logs and reports are created. Different ways how to accomplish this are explained below. In all these cases the remaining test cases are marked failed.
The tests that are automatically failed get robot:exit
tag and
the generated report will include NOT robot:exit
combined tag pattern
to easily see those tests that were not skipped. Note that the test in which
the exit happened does not get the robot:exit
tag.
Note
Prior to Robot Framework 3.1, the special tag was named robot-exit
.
Ctrl-C
The execution is stopped when Ctrl-C
is pressed in the console
where the tests are running. The execution is stopped immediately,
but reports and logs are still generated.
If Ctrl-C
is pressed again, the execution ends immediately and
reports and logs are not created.
On UNIX-like machines it is possible to terminate test execution
using signals INT
and TERM
. These signals can be sent
from the command line using kill command, and sending signals can
also be easily automated.
The execution can be stopped also by the executed keywords. There is a separate Fatal Error BuiltIn keyword for this purpose, and custom keywords can use fatal exceptions when they fail.
If option --exitonfailure (-X) is used, test execution stops immediately if any test fails. The remaining tests are marked as failed without actually executing them.
Robot Framework separates failures caused by failing keywords from errors caused by, for example, invalid settings or failed test library imports. By default these errors are reported as test execution errors, but errors themselves do not fail tests or affect execution otherwise. If --exitonerror option is used, however, all such errors are considered fatal and execution stopped so that remaining tests are marked failed. With parsing errors encountered before execution even starts, this means that no tests are actually run.
By default teardowns of the tests and suites that have been started are executed even if the test execution is stopped using one of the methods above. This allows clean-up activities to be run regardless how execution ends.
It is also possible to skip teardowns when execution is stopped by using --skipteardownonexit option. This can be useful if, for example, clean-up tasks take a lot of time.
Robot Framework can be used also for other automation purposes than test automation, and starting from Robot Framework 3.1 it is possible to explicitly create and execute tasks. For most parts task execution and test execution work the same way, and this section explains the differences.
When Robot Framework is used execute a file and it notices that the file
has tasks, not tests, it automatically sets itself into the generic automation
mode. This mode does not change the actual execution at all, but when
logs and reports are created, they use term task, not test. They have,
for example, headers like Task Log
and Task Statistics
instead of
Test Log
and Test Statistics
.
The generic automation mode can also be enabled by using the --rpa option. In that case the executed files can have either tests or tasks. Alternatively --norpa can be used to force the test automation mode even if executed files contain tasks. If neither of these options are used, it is an error to execute multiple files so that some have tests and others have tasks.
The execution mode is stored in the generated output file and read by Rebot if outputs are post-processed. The mode can also be set when using Rebot if necessary.
XML output files that are generated during the test execution can be post-processed afterwards by the Rebot tool, which is an integral part of Robot Framework. It is used automatically when test reports and logs are generated during the test execution, and using it separately allows creating custom reports and logs as well as combining and merging results.
rebot [options] outputs python -m robot.rebot [options] outputs python path/to/robot/rebot.py [options] outputs
The most common way to use Rebot is using the rebot command. Alternatively it is possible to execute the installed robot.rebot module or the robot/rebot.py file using the selected Python interpreter.
The basic syntax for using Rebot is exactly the same as when starting test execution and also most of the command line options are identical. The main difference is that arguments to Rebot are XML output files instead of test data files or directories.
Return codes from Rebot are exactly same as when running tests.
Rebot notices have tests or tasks been run, and by default preserves the
execution mode. The mode affects logs and reports so that in the former case
they will use term test like Test Log
and Test Statistics
, and in
the latter case term task like Task Log
and Task Statistics
.
Rebot also supports using --rpa or --norpa options to set the execution mode explicitly. This is necessary if multiple output files are processed and they have conflicting modes.
You can use Rebot for creating the same reports and logs that are created automatically during the test execution. Of course, it is not sensible to create the exactly same files, but, for example, having one report with all test cases and another with only some subset of tests can be useful:
rebot output.xml rebot path/to/output_file.xml rebot --include smoke --name Smoke_Tests c:\results\output.xml
Another common usage is creating only the output file when running tests
(log and report generation can be disabled with --log NONE
--report NONE
) and generating logs and reports later. Tests can,
for example, be executed on different environments, output files collected
to a central place, and reports and logs created there.
Rebot does not create XML output files by default, but it is possible to
create them by using the --output (-o) option. Log and report
are created by default, but they can be disabled by using value NONE
(case-insensitive) if they are not needed:
rebot --include smoke --output smoke.xml --log none --report none original.xml
An important feature in Rebot is its ability to combine outputs from different test execution rounds. This capability allows, for example, running the same test cases on different environments and generating an overall report from all outputs. Combining outputs is extremely easy, all that needs to be done is giving several output files as arguments:
rebot output1.xml output2.xml rebot outputs/*.xml
When outputs are combined, a new top-level test suite is created so that test suites in the given output files are its child suites. This works the same way when multiple test data files or directories are executed, and also in this case the name of the top-level test suite is created by joining child suite names with an ampersand (&) and spaces. These automatically generated names are not that good, and it is often a good idea to use --name to give a more meaningful name:
rebot --name Browser_Compatibility firefox.xml opera.xml safari.xml ie.xml rebot --include smoke --name Smoke_Tests c:\results\*.xml
If same tests are re-executed or a single test suite executed in pieces, combining results like discussed above creates an unnecessary top-level test suite. In these cases it is typically better to merge results instead. Merging is done by using --merge (-R) option which changes the way how Rebot combines two or more output files. This option itself takes no arguments and all other command line options can be used with it normally:
rebot --merge original.xml merged.xml rebot --merge --name Example first.xml second.xml third.xml
When suites are merged, documentation, suite setup and suite teardown are got from the last merged suite. Suite metadata from all merged suites is preserved so that values in latter suites have precedence.
How merging tests works is explained in the following sections discussing the two main merge use cases.
Note
Getting suite documentation and metadata from merged suites is new in Robot Framework 6.0.
There is often a need to re-execute a subset of tests, for example, after fixing a bug in the system under test or in the tests themselves. This can be accomplished by selecting test cases by names (--test and --suite options), tags (--include and --exclude), or by previous status (--rerunfailed or --rerunfailedsuites).
Combining re-execution results with the original results using the default combining outputs approach does not work too well. The main problem is that you get separate test suites and possibly already fixed failures are also shown. In this situation it is better to use --merge (-R) option to tell Rebot to merge the results instead. In practice this means that tests from the latter test runs replace tests in the original. An exception to this rule is that skipped tests in latter runs are ignored and original tests preserved.
This usage is best illustrated by a practical example using --rerunfailed and --merge together:
robot --output original.xml tests # first execute all tests robot --rerunfailed original.xml --output rerun.xml tests # then re-execute failing rebot --merge original.xml rerun.xml # finally merge results
The message of the merged tests contains a note that results have been replaced. The message also shows the old status and message of the test.
Merged results must always have same top-level test suite. Tests and suites in merged outputs that are not found from the original output are added into the resulting output. How this works in practice is discussed in the next section.
Note
Ignoring skipped tests in latter runs is new in Robot Framework 4.1.
Another important use case for the --merge option is merging results got when running a test suite in pieces using, for example, --include and --exclude options:
robot --include smoke --output smoke.xml tests # first run some tests robot --exclude smoke --output others.xml tests # then run others rebot --merge smoke.xml others.xml # finally merge results
When merging outputs like this, the resulting output contains all tests and suites found from all given output files. If some test is found from multiple outputs, latest results replace the earlier ones like explained in the previous section. Also this merging strategy requires the top-level test suites to be same in all outputs.
Rebot can create and process output files also in the JSON format. Creating JSON output files is done using the normal --output option so that the specified file has a .json extension:
rebot --output output.json output.xml
When reading output files, JSON files are automatically recognized by the extension:
rebot output.json rebot output1.json output2.json
When combining or merging results, it is possible to mix JSON and XML files:
rebot output1.xml output2.json rebot --merge original.xml rerun.json
The JSON output file structure is documented in the result.json schema file.
Note
Support for JSON output files is new in Robot Framework 7.0.
This section explains different command line options that can be used for configuring the test execution or post-processing outputs. Options related to generated output files are discussed in the next section.
When executing individual files, Robot Framework tries to parse and run them regardless the name or the file extension. What parser to use depends on the extension:
Examples:
robot example.robot # Standard Robot Framework parser. robot example.tsv # Must be compatible with the standard parser. robot example.rst # reStructuredText parser. robot x.robot y.rst # Parse both files using an appropriate parser.
When executing a directory, files and directories are parsed using the following rules:
When executing a directory, it is possible to parse only certain files based on their name or path by using the --parseinclude (-I) option. This option has slightly different semantics depending on the value it is used with:
example.robot
, files matching
the name in all directories will be parsed.path/to/tests.robot
.Examples:
robot --parseinclude example.robot tests # Parse `example.robot` files anywhere under `tests`. robot -I example_*.robot -I ???.robot tests # Parse files matching `example_*.robot` or `???.robot` under `tests`. robot -I tests/example.robot tests # Parse only `tests/example.robot`. robot --parseinclude tests/example tests # Parse files under `tests/example` directory, recursively.
Values used with --parseinclude are case-insensitive and support
glob patterns like example_*.robot
. There are, however,
two small differences compared to how patterns typically work with Robot Framework:
*
matches only a single path segment. For example, path/*/tests.robot
matches path/to/tests.robot but not path/to/nested/tests.robot.**
can be used to enable recursive matching. For example, path/**/tests.robot
matches both path/to/tests.robot and path/to/nested/tests.robot.If the pattern contains an extension, files with that extension are parsed even if they by default would not be. What parser to use depends on the used extension:
Notice that when you use a pattern like *.robot
and there exists a file that
matches the pattern in the execution directory, the shell may resolve
the pattern before Robot Framework is called and the value passed to
it is the file name, not the original pattern. In such cases you need
to quote or escape the pattern like '*.robot'
or \*.robot
.
Note
--parseinclude
is new in Robot Framework 6.1.
In addition to using the --parseinclude option discussed in the
previous section, it is also possible to enable parsing files that are not
parsed by default by using the --extension (-F) option.
Matching extensions is case insensitive and the leading dot can be omitted.
If there is a need to parse more than one kind of files, it is possible to
use a colon :
to separate extensions:
robot --extension rst path/to/tests # Parse only *.rst files. robot -F robot:rst path/to/tests # Parse *.robot and *.rst files.
The above is equivalent to the following --parseinclude usage:
robot --parseinclude *.rst path/to/tests robot -I *.robot -I *.rst path/to/tests
Because the --parseinclude option is more powerful and covers all same use cases as the --extension option, the latter is likely to be deprecated in the future. Users are recommended to use --parseinclude already now.
External parsers can parse files that Robot Framework does not recognize otherwise. For more information about creating and using such parsers see the Parser interface section.
Robot Framework offers several command line options for selecting which test cases to execute. The same options work also when executing tasks and when post-processing outputs with Rebot.
The easiest way to select only some tests to be run is using the --test (-t) option. As the name implies, it can be used for selecting tests by their names. Given names are case, space and underscore insensitive and they also support simple patterns. The option can be used multiple times to match multiple tests:
--test Example # Match only tests with name 'Example'. --test example* # Match tests starting with 'example'. --test first --test second # Match tests with name 'first' or 'second'.
To pinpoint a test more precisely, it is possible to prefix the test name with a suite name:
--test mysuite.mytest # Match test 'mytest' in suite 'mysuite'. --test root.sub.test # Match test 'test' in suite 'sub' in suite 'root'. --test *.sub.test # Match test 'test' in suite 'sub' anywhere.
Notice that when the given name includes a suite name, it must match the whole suite name starting from the root suite. Using a wildcard as in the last example above allows matching tests with a parent suite anywhere.
Using the --test option is convenient when only a few tests needs to be selected. A common use case is running just the test that is currently being worked on. If a bigger number of tests needs to be selected, it is typically easier to select them by suite names or by tag names.
When executing tasks, it is possible to use the --task option as an alias for --test.
Tests can be selected also by suite names with the --suite (-s) option that selects all tests in matching suites. Similarly as with --test, given names are case, space and underscore insensitive and support simple patterns. To pinpoint a suite more precisely, it is possible to prefix the name with the parent suite name:
--suite Example # Match only suites with name 'Example'. --suite example* # Match suites starting with 'example'. --suite first --suite second # Match suites with name 'first' or 'second'. --suite root.child # Match suite 'child' in root suite 'root'. --suite *.parent.child # Match suite 'child' with parent 'parent' anywhere.
If the name contains a parent suite name, it must match the whole suite name the same way as with --test. Using a wildcard as in the last example above allows matching suites with a parent suite anywhere.
Note
Prior to Robot Framework 7.0, --suite with a parent suite
did not need to match the whole suite name. For example, parent.child
would match suite child
with parent parent
anywhere. The name must
be prefixed with a wildcard if this behavior is desired nowadays.
If both --suite and --test options are used, only the specified tests in specified suites are selected:
--suite mysuite --test mytest # Match test 'mytest' if its inside suite 'mysuite'.
Using the --suite option is more or less the same as executing the appropriate suite file or directory directly. The main difference is that if a file or directory is run directly, possible suite setups and teardowns on higher level are not executed:
# Root suite is 'Tests' and its possible setup and teardown are run. robot --suite example path/to/tests # Root suite is 'Example' and possible higher level setups and teardowns are ignored. robot path/to/tests/example.robot
Prior to Robot Framework 6.1, files not matching the --suite option were not parsed at all for performance reasons. This optimization was not possible anymore after suites got a new Name setting that can override the default suite name that is got from the file or directory name. New --parseinclude option has been added to explicitly select which files are parsed if this kind of parsing optimization is needed.
It is possible to include and exclude test cases by tag names with the --include (-i) and --exclude (-e) options, respectively. If the --include option is used, only test cases having a matching tag are selected, and with the --exclude option test cases having a matching tag are not. If both are used, only tests with a tag matching the former option, and not with a tag matching the latter, are selected:
--include example --exclude not_ready --include regression --exclude long_lasting
Both --include and --exclude can be used several times to match multiple tags. In that case a test is selected if it has a tag that matches any included tags, and also has no tag that matches any excluded tags.
In addition to specifying a tag to match fully, it is possible to use
tag patterns where *
and ?
are wildcards and
AND
, OR
, and NOT
operators can be used for
combining individual tags or patterns together:
--include feature-4? --exclude bug* --include fooANDbar --exclude xxORyyORzz --include fooNOTbar
Starting from RF 5.0, it is also possible to use the reserved
tag robot:exclude
to achieve
the same effect as with using the --exclude
option:
*** Test Cases ***
Example
[Tags] robot:exclude
Fail This is not executed
Selecting test cases by tags is a very flexible mechanism and allows many interesting possibilities:
smoke
and executed with --include smoke
.not_ready
and excluded from the test execution with
--exclude not_ready
.sprint-<num>
, where
<num>
specifies the number of the current sprint, and
after executing all test cases, a separate report containing only
the tests for a certain sprint can be generated (for example, rebot
--include sprint-42 output.xml
).Options --include and --exclude can be used in combination with --suite and --test discussed in the previous section. In that case tests that are selected must match all selection criteria:
--suite example --include tag # Match test if it is in suite 'example' and has tag 'tag'. --suite example --exclude tag # Match test if it is in suite 'example' and does not have tag 'tag'. --test ex* --include tag # Match test if its name starts with 'ex' and it has tag 'tag'. --test ex* --exclude tag # Match test if its name starts with 'ex' and it does not have tag 'tag'.
Note
In Robot Framework 7.0 --include
and --test
were cumulative and
selected tests needed to match only either of these options. That behavior
caused backwards incompatibility problems and it was changed
back to the original already in Robot Framework 7.0.1.
Command line option --rerunfailed (-R) can be used to select all failed tests from an earlier output file for re-execution. This option is useful, for example, if running all tests takes a lot of time and one wants to iteratively fix failing test cases.
robot tests # first execute all tests robot --rerunfailed output.xml tests # then re-execute failing
Behind the scenes this option selects the failed tests as they would have been selected individually using the --test option. It is possible to further fine-tune the list of selected tests by using --test, --suite, --include and --exclude options.
It is an error if the output contains no failed tests, but this behavior can be
changed by using the --runemptysuite option discussed below.
Using an output not originating from executing the same tests that are run
now causes undefined results. Using a special value NONE
as the output is
same as not specifying this option at all.
Tip
Re-execution results and original results can be merged together using the --merge command line option.
Command line option --rerunfailedsuites (-S) can be used to select all failed suites from an earlier output file for re-execution. Like --rerunfailed (-R), this option is useful when full test execution takes a lot of time. Note that all tests from a failed test suite will be re-executed, even passing ones. This option is useful when the tests in a test suite depends on each other.
Behind the scenes this option selects the failed suites as they would have been selected individually with the --suite option. It is possible to further fine-tune the list of selected tests by using --test, --suite, --include and --exclude options.
By default when no tests match the selection criteria test execution fails with an error like:
[ ERROR ] Suite 'Example' contains no tests matching tag 'xxx'.
Because no outputs are generated, this behavior can be problematic if tests are executed and results processed automatically. Luckily a command line option --RunEmptySuite (case-insensitive) can be used to force the suite to be executed also in this case. As a result normal outputs are created but show zero executed tests. The same option can be used also to alter the behavior when an empty directory or a test case file containing no tests is executed.
Similar situation can occur also when processing output files with Rebot. It is possible that no test match the used filtering criteria or that the output file contained no tests to begin with. By default executing Rebot fails in these cases, but it has a separate --ProcessEmptySuite option that can be used to alter the behavior. In practice this option works the same way as --RunEmptySuite when running tests.
Note
Using --RunEmptySuite with --ReRunFailed or --ReRunFailedSuites requires Robot Framework 5.0.1 or newer.
When Robot Framework parses test data, suite names are created from file and directory names. The name of the top-level test suite can, however, be overridden with the command line option --name (-N):
robot --name "Custom name" tests.robot
In addition to defining documentation in the test data, documentation of the top-level suite can be given from the command line with the option --doc (-D). The value can contain simple HTML formatting and must be quoted if it contains spaces.
If the given documentation is a relative or absolute path pointing to an existing file, the actual documentation will be read from that file. This is especially convenient if the externally specified documentation is long or contains multiple lines.
Examples:
robot --doc "Example documentation" tests.robot robot --doc doc.txt tests.robot # Documentation read from doc.txt if it exits.
Note
Reading documentation from an external file is new in Robot Framework 4.1.
Prior to Robot Framework 3.1, underscores in documentation were converted to spaces same way as with the --name option.
Free suite metadata may also be given from the command line with the
option --metadata (-M). The argument must be in the format
name:value
, where name
the name of the metadata to set and
value
is its value. The value can contain simple HTML formatting and
the whole argument must be quoted if it contains spaces.
This option may be used several times to set multiple metadata values.
If the given value is a relative or absolute path pointing to an existing
file, the actual value will be read from that file. This is especially
convenient if the value is long or contains multiple lines.
If the value should be a path to an existing file, not read from that file,
the value must be separated with a space from the name:
part.
Examples:
robot --metadata Name:Value tests.robot robot --metadata "Another Name:Another value, now with spaces" tests.robot robot --metadata "Read From File:meta.txt" tests.robot # Value read from meta.txt if it exists. robot --metadata "Path As Value: meta.txt" tests.robot # Value always used as-is.
Note
Reading metadata value from an external file is new in Robot Framework 4.1.
Prior to Robot Framework 3.1, underscores in the value were converted to spaces same way as with the --name option.
The command line option --settag (-G) can be used to set the given tag to all executed test cases. This option may be used several times to set multiple tags.
When Robot Framework imports a test library, listener, or some other Python based extension, it uses the Python interpreter to import the module containing the extension from the system. The list of locations where modules are looked for is called the module search path, and its contents can be configured using different approaches explained in this section.
Robot Framework uses Python's module search path also when importing resource and variable files if the specified path does not match any file directly.
The module search path being set correctly so that libraries and other extensions are found is a requirement for successful test execution. If you need to customize it using approaches explained below, it is often a good idea to create a custom start-up script.
Python interpreters have their own standard library as well as a directory where third party modules are installed automatically in the module search path. This means that test libraries packaged using Python's own packaging system are automatically installed so that they can be imported without any additional configuration.
Python reads additional locations to be added to
the module search path from PYTHONPATH environment variables.
If you want to specify more than one location in any of them, you
need to separate the locations with a colon on UNIX-like machines (e.g.
/opt/libs:$HOME/testlibs
) and with a semicolon on Windows (e.g.
D:\libs;%HOMEPATH%\testlibs
).
Environment variables can be configured permanently system wide or so that they affect only a certain user. Alternatively they can be set temporarily before running a command, something that works extremely well in custom start-up scripts.
--pythonpath
optionRobot Framework has a separate command line option --pythonpath (-P) for adding locations to the module search path.
Multiple locations can be given by separating them with a colon (:
) or
a semicolon (;
) or by using this option multiple times. If the value
contains both colons and semicolons, it is split from semicolons. Paths
can also be glob patterns matching multiple paths, but they typically
need to be escaped when used on the console.
Examples:
--pythonpath libs --pythonpath /opt/testlibs:mylibs.zip:yourlibs --pythonpath /opt/testlibs --pythonpath mylibs.zip --pythonpath yourlibs --pythonpath c:\temp;d:\resources --pythonpath lib/\*.zip # '*' is escaped
Note
Both colon and semicolon work regardless the operating system. Using semicolon is new in Robot Framework 5.0.
sys.path
programmaticallyPython interpreters store the module search path they use as a list of strings in sys.path attribute. This list can be updated dynamically during execution, and changes are taken into account next time when something is imported.
Variables can be set from the command line either individually using the --variable (-v) option or through variable files with the --variablefile (-V) option. Variables and variable files are explained in separate chapters, but the following examples illustrate how to use these options:
--variable name:value --variable OS:Linux --variable IP:10.0.0.42 --variablefile path/to/variables.py --variablefile myvars.py:possible:arguments:here --variable ENVIRONMENT:Windows --variablefile c:\resources\windows.py
Robot Framework supports so called dry run mode where the tests are run normally otherwise, but the keywords coming from the test libraries are not executed at all. The dry run mode can be used to validate the test data; if the dry run passes, the data should be syntactically correct. This mode is triggered using option --dryrun.
The dry run execution may fail for following reasons:
- Using keywords that are not found.
- Using keywords with wrong number of arguments.
- Using user keywords that have invalid syntax.
In addition to these failures, normal execution errors are shown, for example, when test library or resource file imports cannot be resolved.
It is possible to disable dry run validation of specific user keywords
by adding a special robot:no-dry-run
keyword tag to them. This is useful
if a keyword fails in the dry run mode for some reason, but work fine when
executed normally.
Note
The dry run mode does not validate variables.
The test execution order can be randomized using option
--randomize <what>[:<seed>], where <what>
is one of the following:
tests
suites
all
none
It is possible to give a custom seed
to initialize the random generator. This is useful if you want to re-run tests
using the same order as earlier. The seed is given as part of the value for
--randomize in format <what>:<seed>
and it must be an integer.
If no seed is given, it is generated randomly. The executed top level test
suite automatically gets metadata named Randomized that tells both
what was randomized and what seed was used.
Examples:
robot --randomize tests my_test.robot robot --randomize all:12345 path/to/tests
If the provided built-in features to modify test data before execution are not enough, Robot Framework makes it possible to do custom modifications programmatically. This is accomplished by creating a so called pre-run modifier and activating it using the --prerunmodifier option.
Pre-run modifiers should be implemented as visitors that can traverse through the executable test suite structure and modify it as needed. The visitor interface is explained as part of the Robot Framework API documentation, and it possible to modify executed test suites, test cases and keywords using it. The examples below ought to give an idea of how pre-run modifiers can be used and how powerful this functionality is.
When a pre-run modifier is taken into use on the command line using the
--prerunmodifier option, it can be specified either as a name of
the modifier class or a path to the modifier file. If the modifier is given
as a class name, the module containing the class must be in the module search
path, and if the module name is different than the class name, the given
name must include both like module.ModifierClass
. If the modifier is given
as a path, the class name must be same as the file name. For most parts this
works exactly like when importing a test library.
If a modifier requires arguments, like the examples below do, they can be
specified after the modifier name or path using either a colon (:
) or a
semicolon (;
) as a separator. If both are used in the value, the one used
first is considered to be the actual separator. Starting from Robot Framework
4.0, arguments also support the named argument syntax as well as argument
conversion based on type hints and default values the same way
as keywords do.
If more than one pre-run modifier is needed, they can be specified by using the --prerunmodifier option multiple times. If similar modifying is needed before creating logs and reports, programmatic modification of results can be enabled using the --prerebotmodifier option.
Pre-run modifiers are executed before other configuration affecting the executed test suite and test cases. Most importantly, options related to selecting test cases are processed after modifiers, making it possible to use options like --include also with possible dynamically added tests.
Tip
Modifiers are taken into use from the command line exactly the same way as listeners. See the Registering listeners from command line section for more information and examples.
The first example shows how a pre-run modifier can remove tests from the executed test suite structure. In this example only every Xth tests is preserved, and the X is given from the command line along with an optional start index.
"""Pre-run modifier that selects only every Xth test for execution.
Starts from the first test by default. Tests are selected per suite.
"""
from robot.api import SuiteVisitor
class SelectEveryXthTest(SuiteVisitor):
def __init__(self, x: int, start: int = 0):
self.x = x
self.start = start
def start_suite(self, suite):
"""Modify suite's tests to contain only every Xth."""
suite.tests = suite.tests[self.start::self.x]
def end_suite(self, suite):
"""Remove suites that are empty after removing tests."""
suite.suites = [s for s in suite.suites if s.test_count > 0]
def visit_test(self, test):
"""Avoid visiting tests and their keywords to save a little time."""
pass
If the above pre-run modifier is in a file SelectEveryXthTest.py and the file is in the module search path, it could be used like this:
# Specify the modifier as a path. Run every second test. robot --prerunmodifier path/to/SelectEveryXthTest.py:2 tests.robot # Specify the modifier as a name. Run every third test, starting from the second. robot --prerunmodifier SelectEveryXthTest:3:1 tests.robot
Note
Argument conversion based on type hints like x: int
in the above
example is new in Robot Framework 4.0 and requires Python 3.
Also the second example removes tests, this time based on a given name pattern. In practice it works like a negative version of the built-in --test option.
"""Pre-run modifier that excludes tests by their name.
Tests to exclude are specified by using a pattern that is both case and space
insensitive and supports '*' (match anything) and '?' (match single character)
as wildcards.
"""
from robot.api import SuiteVisitor
from robot.utils import Matcher
class ExcludeTests(SuiteVisitor):
def __init__(self, pattern):
self.matcher = Matcher(pattern)
def start_suite(self, suite):
"""Remove tests that match the given pattern."""
suite.tests = [t for t in suite.tests if not self._is_excluded(t)]
def _is_excluded(self, test):
return self.matcher.match(test.name) or self.matcher.match(test.longname)
def end_suite(self, suite):
"""Remove suites that are empty after removing tests."""
suite.suites = [s for s in suite.suites if s.test_count > 0]
def visit_test(self, test):
"""Avoid visiting tests and their keywords to save a little time."""
pass
Assuming the above modifier is in a file named ExcludeTests.py, it could be used like this:
# Exclude test named 'Example'. robot --prerunmodifier path/to/ExcludeTests.py:Example tests.robot # Exclude all tests ending with 'something'. robot --prerunmodifier path/to/ExcludeTests.py:*something tests.robot
Sometimes when debugging tests it can be useful to disable setups or teardowns. This can be accomplished by editing the test data, but pre-run modifiers make it easy to do that temporarily for a single run:
"""Pre-run modifiers for disabling suite and test setups and teardowns."""
from robot.api import SuiteVisitor
class SuiteSetup(SuiteVisitor):
def start_suite(self, suite):
suite.setup = None
class SuiteTeardown(SuiteVisitor):
def start_suite(self, suite):
suite.teardown = None
class TestSetup(SuiteVisitor):
def start_test(self, test):
test.setup = None
class TestTeardown(SuiteVisitor):
def start_test(self, test):
test.teardown = None
Assuming that the above modifiers are all in a file named disable.py and this file is in the module search path, setups and teardowns could be disabled, for example, as follows:
# Disable suite teardowns. robot --prerunmodifier disable.SuiteTeardown tests.robot # Disable both test setups and teardowns by using '--prerunmodifier' twice. robot --prerunmodifier disable.TestSetup --prerunmodifier disable.TestTeardown tests.robot
Note
Prior to Robot Framework 4.0 setup
and teardown
were accessed via
the intermediate keywords
attribute and, for example, suite setup
was disabled like suite.keywords.setup = None
.
There are various command line options to control how test execution is reported on the console.
The overall console output type is set with the --console option. It supports the following case-insensitive values:
verbose
dotted
.
for passed test, F
for failed tests, s
for skipped
tests and x
for tests which are skipped because
test execution exit. Failed tests are listed separately
after execution. This output type makes it easy to see are there any
failures during execution even if there would be a lot of tests.quiet
none
Separate convenience options --dotted (-.) and --quiet
are shortcuts for --console dotted
and --console quiet
, respectively.
Examples:
robot --console quiet tests.robot robot --dotted tests.robot
The width of the test execution output in the console can be set using the option --consolewidth (-W). The default width is 78 characters.
Tip
On many UNIX-like machines you can use handy $COLUMNS
environment variable like --consolewidth $COLUMNS
.
The --consolecolors (-C) option is used to control whether colors should be used in the console output. Colors are implemented using ANSI escape codes with a backup mechanism for older Windows versions that do not support ANSI codes.
This option supports the following case-insensitive values:
auto
on
ansi
on
but forces ANSI codes to be used unconditionally on Windows.off
Note
Using ANSI codes on Windows by default is new in Robot Framework 7.1.
Result file paths written to the console at the end of the execution are, by default, hyperlinks. This behavior can be controlled with the --consolelinks option that accepts the following case-insensitive values:
auto
off
The hyperlink support depends also on the console that is used, but nowadays the support is pretty good. The commonly used Windows Console does not support links, though, but the newer Windows Terminal does.
Note
Hyperlink support is new in Robot Framework 7.1.
Special markers .
(success) and
F
(failure) are shown on the console when using the verbose output
and top level keywords in test cases end. The markers allow following
the test execution in high level, and they are erased when test cases end.
It is possible to configure when markers are used with --consolemarkers (-K) option. It supports the following case-insensitive values:
auto
on
off
Listeners can be used to monitor the test execution. When they are taken into use from the command line, they are specified using the --listener command line option. The value can either be a path to a listener or a listener name. See the Listener interface section for more details about importing listeners and using them in general.
Several output files are created when tests are executed, and all of them are somehow related to test results. This section discusses what outputs are created, how to configure where they are created, and how to fine-tune their contents.
This section explains what different output files can be created and
how to configure where they are created. Output files are configured
using command line options, which get the path to the output file in
question as an argument. A special value NONE
(case-insensitive) can be used to disable creating a certain output
file.
All output files can be set using an absolute path, in which case they are created to the specified place, but in other cases, the path is considered relative to the output directory. The default output directory is the directory where the execution is started from, but it can be altered with the --outputdir (-d) option. The path set with this option is, again, relative to the execution directory, but can naturally be given also as an absolute path. Regardless of how a path to an individual output file is obtained, its parent directory is created automatically, if it does not exist already.
Output files contain all the test execution results in machine readable XML format. Log, report and xUnit files are typically generated based on them, and they can also be combined and otherwise post-processed with Rebot.
Tip
Generating report and xUnit files as part of test execution does not require processing output files after execution. Disabling log generation when running tests can thus save memory.
The command line option --output (-o) determines the path where the output file is created relative to the output directory. The default name for the output file, when tests are run, is output.xml.
When post-processing outputs with Rebot, new output files are not created unless the --output option is explicitly used.
It is possible to disable creation of the output file when running tests by
giving a special value NONE
to the --output option. If no outputs
are needed, they should all be explicitly disabled using
--output NONE --report NONE --log NONE
.
The XML output file structure is documented in the robot.xsd schema file.
Note
Starting from Robot Framework 7.0, Rebot can read and write JSON output files. The plan is to enhance the support for JSON output files in the future so that they could be created already during execution. For more details see issue #3423.
There were some backwards incompatible changes to the output file format in Robot Framework 7.0. To make it possible to use new Robot Framework versions with external tools that are not yet updated to support the new format, there is a --legacyoutput option that produces output files that are compatible with Robot Framework 6.x and earlier. Robot Framework itself can process output files both in the old and in the new formats.
We hope that external tools are updated soon, but we plan to support this option at least until Robot Framework 8.0. If you encounter tools that are not compatible, please inform the tool developers about changes.
Log files contain details about the executed test cases in HTML format. They have a hierarchical structure showing test suite, test case and keyword details. Log files are needed nearly every time when test results are to be investigated in detail. Even though log files also have statistics, reports are better for getting an higher-level overview.
The command line option --log (-l) determines where log
files are created. Unless the special value NONE
is used,
log files are always created and their default name is
log.html.
Report files contain an overview of the test execution results in HTML format. They have statistics based on tags and executed test suites, as well as a list of all executed test cases. When both reports and logs are generated, the report has links to the log file for easy navigation to more detailed information. It is easy to see the overall test execution status from report, because its background color is green, if all tests pass and bright red if any test fails. Background can also be yellow, which means that all tests were skipped.
The command line option --report (-r) determines where
report files are created. Similarly as log files, reports are always
created unless NONE
is used as a value, and their default
name is report.html.
XUnit result files contain the test execution summary in xUnit compatible XML format. These files can thus be used as an input for external tools that understand xUnit reports. For example, Jenkins continuous integration server supports generating statistics based on xUnit compatible results.
Tip
Jenkins also has a separate Robot Framework plugin.
XUnit output files are not created unless the command line option --xunit (-x) is used explicitly. This option requires a path to the generated xUnit file, relatively to the output directory, as a value.
XUnit output files were changed pretty heavily in Robot Framework 5.0.
They nowadays contain separate <testsuite>
elements for each suite,
<testsuite>
elements have timestamp
attribute, and suite documentation
and metadata is stored as <property>
elements.
Debug files are plain text files that are written during the test execution. All messages got from test libraries are written to them, as well as information about started and ended test suites, test cases and keywords. Debug files can be used for monitoring the test execution. This can be done using, for example, a separate fileviewer.py tool, or in UNIX-like systems, simply with the tail -f command.
Debug files are not created unless the command line option --debugfile (-b) is used explicitly.
All output files generated by Robot Framework itself can be automatically timestamped
with the option --timestampoutputs (-T). When this option is used,
a timestamp in the format YYYYMMDD-hhmmss
is placed between
the extension and the base name of each file. The example below would,
for example, create output files like
output-20080604-163225.xml and mylog-20080604-163225.html:
robot --timestampoutputs --log mylog.html --report NONE tests.robot
The default titles for logs and reports are generated by prefixing the name of the top-level test suite with Test Log or Test Report. Custom titles can be given from the command line using the options --logtitle and --reporttitle, respectively.
Example:
robot --logtitle "Smoke Test Log" --reporttitle "Smoke Test Report" --include smoke my_tests/
Note
Prior to Robot Framework 3.1, underscores in the given titles were converted to spaces. Nowadays spaces need to be escaped or quoted like in the example above.
By default the report file has red background if there are failures, green background if there are passed tests and possibly some skipped ones, and a yellow background if all tests are skipped or no tests have been run. These colors can be customized by using the --reportbackground command line option, which takes two or three colors separated with a colon as an argument:
--reportbackground blue:red --reportbackground blue:red:orange --reportbackground #00E:#E00
If you specify two colors, the first one will be used instead of the default green (pass) color and the second instead of the default red (fail). This allows, for example, using blue instead of green to make backgrounds easier to separate for color blind people.
If you specify three colors, the first two have same semantics as earlier and the last one replaces the default yellow (skip) color.
The specified colors are used as a value for the body
element's background
CSS property. The value is used as-is and
can be a HTML color name (e.g. red
), a hexadecimal value
(e.g. #f00
or #ff0000
), or an RGB value
(e.g. rgb(255,0,0)
). The default green, red and yellow colors are
specified using hexadecimal values #9e9
, #f66
and #fed84f
,
respectively.
Messages in log files can have different log levels. Some of the messages are written by Robot Framework itself, but also executed keywords can log information using different levels. The available log levels are:
FAIL
WARN
INFO
DEBUG
TRACE
By default, log messages below the INFO
level are not logged, but this
threshold can be changed from the command line using the
--loglevel (-L) option. This option takes any of the
available log levels as an argument, and that level becomes the new
threshold level. A special value NONE
can also be used to
disable logging altogether.
It is possible to use the --loglevel option also when
post-processing outputs with Rebot. This allows, for example,
running tests initially with the TRACE
level, and generating smaller
log files for normal viewing later with the INFO
level. By default
all the messages included during execution will be included also with
Rebot. Messages ignored during the execution cannot be recovered.
Another possibility to change the log level is using the BuiltIn keyword Set Log Level in the test data. It takes the same arguments as the --loglevel option, and it also returns the old level so that it can be restored later, for example, in a test teardown.
If the log file contains messages at
DEBUG
or TRACE
levels, a visible log level drop down is shown
in the upper right corner. This allows users to remove messages below chosen
level from the view. This can be useful especially when running test at
TRACE
level.
By default the drop down will be set at the lowest level in the log file, so that all messages are shown. The default visible log level can be changed using --loglevel option by giving the default after the normal log level separated by a colon:
--loglevel DEBUG:INFO
In the above example, tests are run using level DEBUG
, but
the default visible level in the log file is INFO
.
Normally the log file is just a single HTML file. When the amount of the test cases increases, the size of the file can grow so large that opening it into a browser is inconvenient or even impossible. Hence, it is possible to use the --splitlog option to split parts of the log into external files that are loaded transparently into the browser when needed.
The main benefit of splitting logs is that individual log parts are so small that opening and browsing the log file is possible even if the amount of the test data is very large. A small drawback is that the overall size taken by the log file increases.
Technically the test data related to each test case is saved into a JavaScript file in the same folder as the main log file. These files have names such as log-42.js where log is the base name of the main log file and 42 is an incremented index.
Note
When copying the log files, you need to copy also all the log-*.js files or some information will be missing.
There are several command line options that can be used to configure and adjust the contents of the Statistics by Tag, Statistics by Suite and Test Details by Tag tables in different output files. All these options work both when executing test cases and when post-processing outputs.
When a deeper suite structure is executed, showing all the test suite levels in the Statistics by Suite table may make the table somewhat difficult to read. By default all suites are shown, but you can control this with the command line option --suitestatlevel which takes the level of suites to show as an argument:
--suitestatlevel 3
When many tags are used, the Statistics by Tag table can become quite congested. If this happens, the command line options --tagstatinclude and --tagstatexclude can be used to select which tags to display, similarly as --include and --exclude are used to select test cases:
--tagstatinclude some-tag --tagstatinclude another-tag --tagstatexclude owner-* --tagstatinclude prefix-* --tagstatexclude prefix-13
The command line option --tagstatcombine can be used to
generate aggregate tags that combine statistics from multiple
tags. The combined tags are specified using tag patterns where
*
and ?
are supported as wildcards and AND
,
OR
and NOT
operators can be used for combining
individual tags or patterns together.
The following examples illustrate creating combined tag statistics using different patterns, and the figure below shows a snippet of the resulting Statistics by Tag table:
--tagstatcombine owner-* --tagstatcombine smokeANDmytag --tagstatcombine smokeNOTowner-janne*
As the above example illustrates, the name of the added combined statistic
is, by default, just the given pattern. If this is not good enough, it
is possible to give a custom name after the pattern by separating them
with a colon (:
):
--tagstatcombine "prio1ORprio2:High priority tests"
Note
Prior to Robot Framework 3.1, underscores in the custom name were converted to spaces. Nowadays spaces need to be escaped or quoted like in the example above.
You can add external links to the Statistics by Tag table by
using the command line option --tagstatlink. Arguments to this
option are given in the format tag:link:name
, where tag
specifies the tags to assign the link to, link
is the link to
be created, and name
is the name to give to the link.
tag
may be a single tag, but more commonly a simple pattern
where *
matches anything and ?
matches any single
character. When tag
is a pattern, the matches to wildcards may
be used in link
and title
with the syntax %N
,
where "N" is the index of the match starting from 1.
The following examples illustrate the usage of this option, and the figure below shows a snippet of the resulting Statistics by Tag table when example test data is executed with these options:
--tagstatlink mytag:http://www.google.com:Google --tagstatlink example-bug-*:http://example.com --tagstatlink owner-*:mailto:%1@domain.com?subject=Acceptance_Tests:Send_Mail
Tags can be given a documentation with the command line option
--tagdoc, which takes an argument in the format
tag:doc
. tag
is the name of the tag to assign the
documentation to, and it can also be a simple pattern matching
multiple tags. doc
is the assigned documentation. It can contain
simple HTML formatting.
The given documentation is shown with matching tags in the Test Details by Tag table, and as a tool tip for these tags in the Statistics by Tag table. If one tag gets multiple documentations, they are combined together and separated with an ampersand.
Examples:
--tagdoc mytag:Example --tagdoc "regression:*See* http://info.html" --tagdoc "owner-*:Original author"
Note
Prior to Robot Framework 3.1, underscores in the documentation were converted to spaces. Nowadays spaces need to be escaped or quoted like in the examples above.
Most of the content of output files comes from keywords and their log messages. When creating higher level reports, log files are not necessarily needed at all, and in that case keywords and their messages just take space unnecessarily. Log files themselves can also grow overly large, especially if they contain FOR loops or other constructs that repeat certain keywords multiple times.
In these situations, command line options --removekeywords and
--flattenkeywords can be used to dispose or flatten unnecessary keywords.
They can be used both when executing test cases and when post-processing
outputs. When used during execution, they only affect the log file, not
the XML output file. With rebot
they affect both logs and possibly
generated new output XML files.
The --removekeywords option removes keywords and their messages
altogether. It has the following modes of operation, and it can be used
multiple times to enable multiple modes. Keywords that contain errors
or warnings are not removed except when using the ALL
mode.
ALL
PASSED
FOR
WHILE
WUKS
NAME:<pattern>
MyLibrary.Keyword Name
. The pattern is case, space, and underscore
insensitive, and it supports simple patterns with *
, ?
and []
as wildcards.TAG:<pattern>
*
, ?
and []
are supported as wildcards and AND
, OR
and NOT
operators can be used for combining individual tags or patterns together.
Can be used both with library keyword tags and user keyword tags.Examples:
rebot --removekeywords all --output removed.xml output.xml robot --removekeywords passed --removekeywords for tests.robot robot --removekeywords name:HugeKeyword --removekeywords name:resource.* tests.robot robot --removekeywords tag:huge tests.robot
Removing keywords is done after parsing the output file and generating an internal model based on it. Thus it does not reduce memory usage as much as flattening keywords.
The --flattenkeywords option flattens matching keywords. In practice this means that matching keywords get all log messages from their child keywords, recursively, and child keywords are discarded otherwise. Flattening supports the following modes:
FOR
WHILE
ITERATION
FOR
and WHILE
loop iterations.FORITEM
ITERATION
.NAME:<pattern>
NAME:<pattern>
mode.TAG:<pattern>
TAG:<pattern>
mode.Examples:
robot --flattenkeywords name:HugeKeyword --flattenkeywords name:resource.* tests.robot rebot --flattenkeywords foritem --output flattened.xml original.xml
Flattening keywords is done already when the output file is parsed initially. This can save a significant amount of memory especially with deeply nested keyword structures.
Starting from Robot Framework 6.1, it is possible to enable the keyword flattening during
the execution time. This can be done only on an user keyword level by defining the reserved tag
robot:flatten
as a keyword tag. Using this tag will work similarly as the command line
option described in the previous chapter, e.g. all content except for log messages is removed
from under the keyword having the tag. One important difference is that in this case, the removed
content is not written to the output file at all, and thus cannot be accessed at later time.
Some examples
*** Keywords ***
Flattening affects this keyword and all it's children
[Tags] robot:flatten
Log something
FOR ${i} IN RANGE 2
Log The message is preserved but for loop iteration is not
END
*** Settings ***
# Flatten content of all uer keywords
Keyword Tags robot:flatten
Keywords that have passed are closed in the log file by default. Thus information they contain is not visible unless you expand them. If certain keywords have important information that should be visible when the log file is opened, you can use the --expandkeywords option to set keywords automatically expanded in log file similar to failed keywords. Expanding supports the following modes:
NAME:<pattern>
NAME:<pattern>
mode.TAG:<pattern>
TAG:<pattern>
mode.If you need to expand keywords matching different names or patterns, you can use the --expandkeywords multiple times.
Examples:
robot --expandkeywords name:SeleniumLibrary.CapturePageScreenshot tests.robot rebot --expandkeywords tag:example --expandkeywords tag:another output.xml
Note
The --expandkeywords option is new in Robot Framework 3.2.
When combining outputs using Rebot, it is possible to set the start and end time of the combined test suite using the options --starttime and --endtime, respectively. This is convenient, because by default, combined suites do not have these values. When both the start and end time are given, the elapsed time is also calculated based on them. Otherwise the elapsed time is got by adding the elapsed times of the child test suites together.
It is also possible to use the above mentioned options to set start and end times for a single suite when using Rebot. Using these options with a single output always affects the elapsed time of the suite.
Times must be given as timestamps in the format YYYY-MM-DD
hh:mm:ss.mil
, where all separators are optional and the parts from
milliseconds to hours can be omitted. For example, 2008-06-11
17:59:20.495
is equivalent both to 20080611-175920.495
and
20080611175920495
, and also mere 20080611
would work.
Examples:
rebot --starttime 20080611-17:59:20.495 output1.xml output2.xml rebot --starttime 20080611-175920 --endtime 20080611-180242 *.xml rebot --starttime 20110302-1317 --endtime 20110302-11418 myoutput.xml
If a test case fails and has a long error message, the message shown in
reports is automatically cut from the middle to keep reports easier to
read. By default messages longer than 40 lines are cut, but that can be
configured by using the --maxerrorlines command line option.
The minimum value for this option is 10, and it is also possible to use
a special value NONE
to show the full message.
Full error messages are always visible in log files as messages of the failed keywords.
Note
The --maxerrorlines option is new in Robot Framework 3.1.
If the provided built-in features to modify results are not enough, Robot Framework makes it possible to do custom modifications programmatically. This is accomplished by creating a model modifier and activating it using the --prerebotmodifier option.
This functionality works nearly exactly like programmatic modification of test data that can be enabled with the --prerunmodifier option. The obvious difference is that this time modifiers operate with the result model, not the running model. For example, the following modifier marks all passed tests that have taken more time than allowed as failed:
from robot.api import SuiteVisitor
class ExecutionTimeChecker(SuiteVisitor):
def __init__(self, max_seconds: float):
self.max_milliseconds = max_seconds * 1000
def visit_test(self, test):
if test.status == 'PASS' and test.elapsedtime > self.max_milliseconds:
test.status = 'FAIL'
test.message = 'Test execution took too long.'
If the above modifier would be in file ExecutionTimeChecker.py, it could be used, for example, like this:
# Specify modifier as a path when running tests. Maximum time is 42 seconds. robot --prerebotmodifier path/to/ExecutionTimeChecker.py:42 tests.robot # Specify modifier as a name when using Rebot. Maximum time is 3.14 seconds. # ExecutionTimeChecker.py must be in the module search path. rebot --prerebotmodifier ExecutionTimeChecker:3.14 output.xml
If more than one model modifier is needed, they can be specified by using the --prerebotmodifier option multiple times. When executing tests, it is possible to use --prerunmodifier and --prerebotmodifier options together.
Note
Argument conversion based on type hints like max_seconds: float
in
the above example is new in Robot Framework 4.0 and requires Python 3.
Robot Framework has its own plain-text system log where it writes information about
- Processed and skipped test data files
- Imported test libraries, resource files and variable files
- Executed test suites and test cases
- Created outputs
Normally users never need this information, but it can be useful when investigating problems with test libraries or Robot Framework itself. A system log is not created by default, but it can be enabled by setting the environment variable ROBOT_SYSLOG_FILE so that it contains a path to the selected file.
A system log has the same log levels as a normal log file, with the
exception that instead of FAIL
it has the ERROR
level. The threshold level to use can be altered using the
ROBOT_SYSLOG_LEVEL environment variable like shown in the
example below. Possible unexpected errors and warnings are
written into the system log in addition to the console and the normal
log file.
#!/bin/bash
export ROBOT_SYSLOG_FILE=/tmp/syslog.txt
export ROBOT_SYSLOG_LEVEL=DEBUG
robot --name Syslog_example path/to/tests
Robot Framework's actual testing capabilities are provided by test libraries. There are many existing libraries, some of which are even bundled with the core framework, but there is still often a need to create new ones. This task is not too complicated because, as this chapter illustrates, Robot Framework's library API is simple and straightforward.
*varargs
)**kwargs
)@keyword
decorator@not_keyword
decoratorRobot Framework itself is written with Python and naturally test libraries extending it can be implemented using the same language. It is also possible to implement libraries with C using Python C API, although it is often easier to interact with C code from Python libraries using ctypes module.
Libraries implemented using Python can also act as wrappers to functionality implemented using other programming languages. A good example of this approach is the Remote library, and another widely used approaches is running external scripts or tools as separate processes.
Robot Framework has three different test library APIs.
Static API
The simplest approach is having a module or a class
with functions/methods which map directly to
keyword names. Keywords also take the same arguments as
the methods implementing them. Keywords report failures with
exceptions, log by writing to standard output and can return
values using the return
statement.
Dynamic API
Dynamic libraries are classes that implement a method to get the names of the keywords they implement, and another method to execute a named keyword with given arguments. The names of the keywords to implement, as well as how they are executed, can be determined dynamically at runtime, but reporting the status, logging and returning values is done similarly as in the static API.
Hybrid API
This is a hybrid between the static and the dynamic API. Libraries are classes with a method telling what keywords they implement, but those keywords must be available directly. Everything else except discovering what keywords are implemented is similar as in the static API.
All these APIs are described in this chapter. Everything is based on how the static API works, so its functions are discussed first. How the dynamic library API and the hybrid library API differ from it is then discussed in sections of their own.
Test libraries can be implemented as Python modules or classes.
The name of a test library that is used when a library is imported is
the same as the name of the module or class implementing it. For
example, if you have a Python module MyLibrary
(that is,
file MyLibrary.py), it will create a library with name
MyLibrary.
Python classes are always inside a module. If the name of a class
implementing a library is the same as the name of the module, Robot
Framework allows dropping the class name when importing the
library. For example, class MyLib
in MyLib.py
file can be used as a library with just name MyLib. This also
works with submodules so that if, for example, parent.MyLib
module
has class MyLib
, importing it using just parent.MyLib
works. If the module name and class name are different, libraries must be
taken into use using both module and class names, such as
mymodule.MyLibrary or parent.submodule.MyLib.
Tip
If the library name is really long, it is recommended to give
the library a simpler alias by using AS
.
All test libraries implemented as classes can take arguments. These arguments are specified in the Setting section after the library name, and when Robot Framework creates an instance of the imported library, it passes them to its constructor. Libraries implemented as a module cannot take any arguments, so trying to use those results in an error.
The number of arguments needed by the library is the same as the number of arguments accepted by the library's constructor. The default values and variable number of arguments work similarly as with keyword arguments. Arguments passed to the library, as well as the library name itself, can be specified using variables, so it is possible to alter them, for example, from the command line.
*** Settings ***
Library MyLibrary 10.0.0.1 8080
Library AnotherLib ${VAR}
Example implementations for the libraries used in the above example:
from example import Connection
class MyLibrary:
def __init__(self, host, port=80):
self._conn = Connection(host, int(port))
def send_message(self, message):
self._conn.send(message)
class AnotherLib:
def __init__(self, environment):
self.environment = environment
def do_something(self):
if self.environment == 'test':
# do something in test environment
else:
# do something in other environments
Libraries implemented as classes can have an internal state, which can be altered by keywords and with arguments to the constructor of the library. Because the state can affect how keywords actually behave, it is important to make sure that changes in one test case do not accidentally affect other test cases. These kind of dependencies may create hard-to-debug problems, for example, when new test cases are added and they use the library inconsistently.
Robot Framework attempts to keep test cases independent from each other: by default, it creates new instances of test libraries for every test case. However, this behavior is not always desirable, because sometimes test cases should be able to share a common state. Additionally, all libraries do not have a state and creating new instances of them is simply not needed.
Test libraries can control when new libraries are created with a
class attribute ROBOT_LIBRARY_SCOPE
. This attribute must be
a string and it can have the following three values:
TEST
A new instance is created for every test case. A possible suite setup and suite teardown share yet another instance.
Prior to Robot Framework 3.2 this value was TEST CASE
, but nowadays
TEST
is recommended. Because all unrecognized values are considered
same as TEST
, both values work with all versions. For the same reason
it is possible to also use value TASK
if the library is targeted for
RPA usage more than testing. TEST
is also the default value if the
ROBOT_LIBRARY_SCOPE
attribute is not set.
SUITE
A new instance is created for every test suite. The lowest-level test suites, created from test case files and containing test cases, have instances of their own, and higher-level suites all get their own instances for their possible setups and teardowns.
Prior to Robot Framework 3.2 this value was TEST SUITE
. That value still
works, but SUITE
is recommended with libraries targeting Robot Framework
3.2 and newer.
GLOBAL
Note
If a library is imported multiple times with different arguments, a new instance is created every time regardless the scope.
When the SUITE
or GLOBAL
scopes are used with libraries that have a state,
it is recommended that libraries have some
special keyword for cleaning up the state. This keyword can then be
used, for example, in a suite setup or teardown to ensure that test
cases in the next test suites can start from a known state. For example,
SeleniumLibrary uses the GLOBAL
scope to enable
using the same browser in different test cases without having to
reopen it, and it also has the Close All Browsers keyword for
easily closing all opened browsers.
Example library using the SUITE
scope:
class ExampleLibrary:
ROBOT_LIBRARY_SCOPE = 'SUITE'
def __init__(self):
self._counter = 0
def count(self):
self._counter += 1
print(self._counter)
def clear_counter(self):
self._counter = 0
When a test library is taken into use, Robot Framework tries to determine its version. This information is then written into the syslog to provide debugging information. Library documentation tool Libdoc also writes this information into the keyword documentations it generates.
Version information is read from attribute
ROBOT_LIBRARY_VERSION
, similarly as library scope is
read from ROBOT_LIBRARY_SCOPE
. If
ROBOT_LIBRARY_VERSION
does not exist, information is tried to
be read from __version__
attribute. These attributes must be
class or module attributes, depending whether the library is
implemented as a class or a module.
An example module using __version__
:
__version__ = '0.1'
def keyword():
pass
Library documentation tool Libdoc
supports documentation in multiple formats. If you want to use something
else than Robot Framework's own documentation formatting, you can specify
the format in the source code using ROBOT_LIBRARY_DOC_FORMAT
attribute
similarly as scope and version are set with their own
ROBOT_LIBRARY_*
attributes.
The possible case-insensitive values for documentation format are
ROBOT
(default), HTML
, TEXT
(plain text),
and reST
(reStructuredText). Using the reST
format requires
the docutils module to be installed when documentation is generated.
Setting the documentation format is illustrated by the following example that uses reStructuredText format. See Documenting libraries section and Libdoc chapter for more information about documenting test libraries in general.
"""A library for *documentation format* demonstration purposes.
This documentation is created using reStructuredText__. Here is a link
to the only \`Keyword\`.
__ http://docutils.sourceforge.net
"""
ROBOT_LIBRARY_DOC_FORMAT = 'reST'
def keyword():
"""**Nothing** to see here. Not even in the table below.
======= ===== =====
Table here has
nothing to see.
======= ===== =====
"""
pass
Listener interface allows external listeners to get notifications about
test execution. They are called, for example, when suites, tests, and keywords
start and end. Sometimes getting such notifications is also useful for test
libraries, and they can register a custom listener by using
ROBOT_LIBRARY_LISTENER
attribute. The value of this attribute
should be an instance of the listener to use, possibly the library itself.
For more information and examples see Libraries as listeners section.
@library
decoratorAn easy way to configure libraries implemented as classes is using
the robot.api.deco.library
class decorator. It allows configuring library's
scope, version, custom argument converters, documentation format
and listener with optional arguments scope
, version
, converter
,
doc_format
and listener
, respectively. When these arguments are used, they
set the matching ROBOT_LIBRARY_SCOPE
, ROBOT_LIBRARY_VERSION
,
ROBOT_LIBRARY_CONVERTERS
, ROBOT_LIBRARY_DOC_FORMAT
and ROBOT_LIBRARY_LISTENER
attributes automatically:
from robot.api.deco import library
from example import Listener
@library(scope='GLOBAL', version='3.2b1', doc_format='reST', listener=Listener())
class Example:
# ...
The @library
decorator also disables the automatic keyword discovery
by setting the ROBOT_AUTO_KEYWORDS
argument to False
by default. This
means that it is mandatory to decorate methods with the @keyword decorator
to expose them as keywords. If only that behavior is desired and no further
configuration is needed, the decorator can also be used without parenthesis
like:
from robot.api.deco import library
@library
class Example:
# ...
If needed, the automatic keyword discovery can be enabled by using the
auto_keywords
argument:
from robot.api.deco import library
@library(scope='GLOBAL', auto_keywords=True)
class Example:
# ...
The @library
decorator only sets class attributes ROBOT_LIBRARY_SCOPE
,
ROBOT_LIBRARY_VERSION
, ROBOT_LIBRARY_CONVERTERS
, ROBOT_LIBRARY_DOC_FORMAT
and ROBOT_LIBRARY_LISTENER
if the respective arguments scope
, version
,
converters
, doc_format
and listener
are used. The ROBOT_AUTO_KEYWORDS
attribute is set always. When attributes are set, they override possible
existing class attributes.
Note
The @library
decorator is new in Robot Framework 3.2
and converters
argument is new in Robot Framework 5.0.
When the static library API is used, Robot Framework uses introspection to find out what keywords the library class or module implements. By default it excludes methods and functions starting with an underscore. All the methods and functions that are not ignored are considered keywords. For example, the library below implements a single keyword My Keyword.
class MyLibrary:
def my_keyword(self, arg):
return self._helper_method(arg)
def _helper_method(self, arg):
return arg.upper()
Automatically considering all public methods and functions keywords typically works well, but there are cases where it is not desired. There are also situations where keywords are created when not expected. For example, when implementing a library as class, it can be a surprise that also methods in possible base classes are considered keywords. When implementing a library as a module, functions imported into the module namespace becoming keywords is probably even a bigger surprise.
This section explains how to prevent methods and functions becoming keywords.
When a library is implemented as a class, it is possible to tell
Robot Framework not to automatically expose methods as keywords by setting
the ROBOT_AUTO_KEYWORDS
attribute to the class with a false value:
class Example:
ROBOT_AUTO_KEYWORDS = False
When the ROBOT_AUTO_KEYWORDS
attribute is set like this, only methods that
have explicitly been decorated with the @keyword decorator or otherwise
have the robot_name
attribute become keywords. The @keyword
decorator
can also be used for setting a custom name, tags and argument types
to the keyword.
Although the ROBOT_AUTO_KEYWORDS
attribute can be set to the class
explicitly, it is more convenient to use the @library decorator
that sets it to False
by default:
from robot.api.deco import keyword, library
@library
class Example:
@keyword
def this_is_keyword(self):
pass
@keyword('This is keyword with custom name')
def xxx(self):
pass
def this_is_not_keyword(self):
pass
Note
Both limiting what methods become keywords using the
ROBOT_AUTO_KEYWORDS
attribute and the @library
decorator are
new in Robot Framework 3.2.
Another way to explicitly specify what keywords a library implements is using the dynamic or the hybrid library API.
When implementing a library as a module, all functions in the module namespace become keywords. This is true also with imported functions, and that can cause nasty surprises. For example, if the module below would be used as a library, it would contain a keyword Example Keyword, as expected, but also a keyword Current Thread.
from threading import current_thread
def example_keyword():
print('Running in thread "%s".' % current_thread().name)
A simple way to avoid imported functions becoming keywords is to only
import modules (e.g. import threading
) and to use functions via the module
(e.g threading.current_thread()
). Alternatively functions could be
given an alias starting with an underscore at the import time (e.g.
from threading import current_thread as _current_thread
).
A more explicit way to limit what functions become keywords is using
the module level __all__
attribute that Python itself uses for similar
purposes. If it is used, only the listed functions can be keywords.
For example, the library below implements only one keyword
Example Keyword:
from threading import current_thread
__all__ = ['example_keyword']
def example_keyword():
print('Running in thread "%s".' % current_thread().name)
def this_is_not_keyword():
pass
If the library is big, maintaining the __all__
attribute when keywords are
added, removed or renamed can be a somewhat big task. Another way to explicitly
mark what functions are keywords is using the ROBOT_AUTO_KEYWORDS
attribute
similarly as it can be used with class based libraries. When this attribute
is set to a false value, only functions explicitly decorated with the
@keyword decorator become keywords. For example, also this library
implements only one keyword Example Keyword:
from threading import current_thread
from robot.api.deco import keyword
ROBOT_AUTO_KEYWORDS = False
@keyword
def example_keyword():
print('Running in thread "%s".' % current_thread().name)
def this_is_not_keyword():
pass
Note
Limiting what functions become keywords using ROBOT_AUTO_KEYWORDS
is a new feature in Robot Framework 3.2.
@not_keyword
decoratorFunctions in modules and methods in classes can be explicitly marked as
"not keywords" by using the @not_keyword
decorator. When a library is
implemented as a module, this decorator can also be used to avoid imported
functions becoming keywords.
from threading import current_thread
from robot.api.deco import not_keyword
not_keyword(current_thread) # Don't expose `current_thread` as a keyword.
def example_keyword():
print('Running in thread "%s".' % current_thread().name)
@not_keyword
def this_is_not_keyword():
pass
Using the @not_keyword
decorator is pretty much the opposite way to avoid
functions or methods becoming keywords compared to disabling the automatic
keyword discovery with the @library
decorator or by setting the
ROBOT_AUTO_KEYWORDS
to a false value. Which one to use depends on the context.
Note
The @not_keyword
decorator is new in Robot Framework 3.2.
Keyword names used in the test data are compared with method names to
find the method implementing these keywords. Name comparison is
case-insensitive, and also spaces and underscores are ignored. For
example, the method hello
maps to the keyword name
Hello, hello or even h e l l o. Similarly both the
do_nothing
and doNothing
methods can be used as the
Do Nothing keyword in the test data.
Example library implemented as a module in the MyLibrary.py file:
def hello(name):
print("Hello, %s!" % name)
def do_nothing():
pass
The example below illustrates how the example library above can be used. If you want to try this yourself, make sure that the library is in the module search path.
*** Settings ***
Library MyLibrary
*** Test Cases ***
My Test
Do Nothing
Hello world
It is possible to expose a different name for a keyword instead of the
default keyword name which maps to the method name. This can be accomplished
by setting the robot_name
attribute on the method to the desired custom name:
def login(username, password):
# ...
login.robot_name = 'Login via user panel'
*** Test Cases ***
My Test
Login Via User Panel ${username} ${password}
Instead of explicitly setting the robot_name
attribute like in the above
example, it is typically easiest to use the @keyword decorator:
from robot.api.deco import keyword
@keyword('Login via user panel')
def login(username, password):
# ...
Using this decorator without an argument will have no effect on the exposed
keyword name, but will still set the robot_name
attribute. This allows
marking methods to expose as keywords without actually changing keyword
names. Methods that have the robot_name
attribute also create keywords even if the method name itself would start with
an underscore.
Setting a custom keyword name can also enable library keywords to accept arguments using the embedded arguments syntax.
Library keywords and user keywords can have tags. Library keywords can
define them by setting the robot_tags
attribute on the method to a list
of desired tags. Similarly as when setting custom name, it is easiest to
set this attribute by using the @keyword decorator:
from robot.api.deco import keyword
@keyword(tags=['tag1', 'tag2'])
def login(username, password):
# ...
@keyword('Custom name', ['tags', 'here'])
def another_example():
# ...
Another option for setting tags is giving them on the last line of
keyword documentation with Tags:
prefix and separated by a comma. For
example:
def login(username, password):
"""Log user in to SUT.
Tags: tag1, tag2
"""
# ...
With a static and hybrid API, the information on how many arguments a keyword needs is got directly from the method that implements it. Libraries using the dynamic library API have other means for sharing this information, so this section is not relevant to them.
The most common and also the simplest situation is when a keyword needs an exact number of arguments. In this case, the method simply take exactly those arguments. For example, a method implementing a keyword with no arguments takes no arguments either, a method implementing a keyword with one argument also takes one argument, and so on.
Example keywords taking different numbers of arguments:
def no_arguments():
print("Keyword got no arguments.")
def one_argument(arg):
print("Keyword got one argument '%s'." % arg)
def three_arguments(a1, a2, a3):
print("Keyword got three arguments '%s', '%s' and '%s'." % (a1, a2, a3))
It is often useful that some of the arguments that a keyword uses have default values.
In Python a method has always exactly one implementation and possible default values are specified in the method signature. The syntax, which is familiar to all Python programmers, is illustrated below:
def one_default(arg='default'):
print("Argument has value %s" % arg)
def multiple_defaults(arg1, arg2='default 1', arg3='default 2'):
print("Got arguments %s, %s and %s" % (arg1, arg2, arg3))
The first example keyword above can be used either with zero or one
arguments. If no arguments are given, arg
gets the value
default
. If there is one argument, arg
gets that value,
and calling the keyword with more than one argument fails. In the
second example, one argument is always required, but the second and
the third one have default values, so it is possible to use the keyword
with one to three arguments.
*** Test Cases ***
Defaults
One Default
One Default argument
Multiple Defaults required arg
Multiple Defaults required arg optional
Multiple Defaults required arg optional 1 optional 2
*varargs
)Robot Framework supports also keywords that take any number of arguments.
Python supports methods accepting any number of arguments. The same syntax works in libraries and, as the examples below show, it can also be combined with other ways of specifying arguments:
def any_arguments(*args):
print("Got arguments:")
for arg in args:
print(arg)
def one_required(required, *others):
print("Required: %s\nOthers:" % required)
for arg in others:
print(arg)
def also_defaults(req, def1="default 1", def2="default 2", *rest):
print(req, def1, def2, rest)
*** Test Cases ***
Varargs
Any Arguments
Any Arguments argument
Any Arguments arg 1 arg 2 arg 3 arg 4 arg 5
One Required required arg
One Required required arg another arg yet another
Also Defaults required
Also Defaults required these two have defaults
Also Defaults 1 2 3 4 5 6
**kwargs
)Robot Framework supports Python's **kwargs syntax. How to use use keywords that accept free keyword arguments, also known as free named arguments, is discussed under the Creating test cases section. In this section we take a look at how to create such keywords.
If you are already familiar how kwargs work with Python, understanding how they work with Robot Framework test libraries is rather simple. The example below shows the basic functionality:
def example_keyword(**stuff):
for name, value in stuff.items():
print(name, value)
*** Test Cases ***
Keyword Arguments
Example Keyword hello=world # Logs 'hello world'.
Example Keyword foo=1 bar=42 # Logs 'foo 1' and 'bar 42'.
Basically, all arguments at the end of the keyword call that use the
named argument syntax name=value
, and that do not match any
other arguments, are passed to the keyword as kwargs. To avoid using a literal
value like foo=quux
as a free keyword argument, it must be escaped
like foo\=quux
.
The following example illustrates how normal arguments, varargs, and kwargs work together:
def various_args(arg=None, *varargs, **kwargs):
if arg is not None:
print('arg:', arg)
for value in varargs:
print('vararg:', value)
for name, value in sorted(kwargs.items()):
print('kwarg:', name, value)
*** Test Cases ***
Positional
Various Args hello world # Logs 'arg: hello' and 'vararg: world'.
Named
Various Args arg=value # Logs 'arg: value'.
Kwargs
Various Args a=1 b=2 c=3 # Logs 'kwarg: a 1', 'kwarg: b 2' and 'kwarg: c 3'.
Various Args c=3 a=1 b=2 # Same as above. Order does not matter.
Positional and kwargs
Various Args 1 2 kw=3 # Logs 'arg: 1', 'vararg: 2' and 'kwarg: kw 3'.
Named and kwargs
Various Args arg=value hello=world # Logs 'arg: value' and 'kwarg: hello world'.
Various Args hello=world arg=value # Same as above. Order does not matter.
For a real world example of using a signature exactly like in the above example, see Run Process and Start Keyword keywords in the Process library.
Starting from Robot Framework 3.1, it is possible to use named-only arguments
with different keywords. This support
is provided by Python's keyword-only arguments. Keyword-only arguments
are specified after possible *varargs
or after a dedicated *
marker when
*varargs
are not needed. Possible **kwargs
are specified after keyword-only
arguments.
Example:
def sort_words(*words, case_sensitive=False):
key = str.lower if case_sensitive else None
return sorted(words, key=key)
def strip_spaces(word, *, left=True, right=True):
if left:
word = word.lstrip()
if right:
word = word.rstrip()
return word
*** Test Cases ***
Example
Sort Words Foo bar baZ
Sort Words Foo bar baZ case_sensitive=True
Strip Spaces ${word} left=False
Python supports so called positional-only arguments that make it possible to
specify that an argument can only be given as a positional argument, not as
a named argument like name=value
. Positional-only arguments are specified
before normal arguments and a special /
marker must be used after them:
def keyword(posonly, /, normal):
print(f"Got positional-only argument {posonly} and normal argument {normal}.")
The above keyword could be used like this:
*** Test Cases ***
Example
# Positional-only and normal argument used as positional arguments.
Keyword foo bar
# Normal argument can also be named.
Keyword foo normal=bar
If a positional-only argument is used with a value that contains an equal sign
like example=usage
, it is not considered to mean named argument syntax
even if the part before the =
would match the argument name. This rule
only applies if the positional-only argument is used in its correct position
without other arguments using the name argument syntax before it, though.
*** Test Cases ***
Example
# Positional-only argument gets literal value `posonly=foo` in this case.
Keyword posonly=foo normal=bar
# This fails.
Keyword normal=bar posonly=foo
Positional-only arguments are fully supported starting from Robot Framework 4.0. Using them as positional arguments works also with earlier versions, but using them as named arguments causes an error on Python side.
Arguments defined in Robot Framework test data are, by default, passed to keywords as Unicode strings. There are, however, several ways to use non-string values as well:
Automatic argument conversion based on function annotations, types specified
using the @keyword
decorator, and argument default values are all new
features in Robot Framework 3.1. The Supported conversions section
specifies which argument conversion are supported in these cases.
Prior to Robot Framework 4.0, automatic conversion was done only if the given argument was a string. Nowadays it is done regardless the argument type.
If no type information is specified to Robot Framework, all arguments not passed as variables are given to keywords as Unicode strings. This includes cases like this:
*** Test Cases ***
Example
Example Keyword 42 False
It is always possible to convert arguments passed as strings insider keywords.
In simple cases this means using int()
or float()
to convert arguments
to numbers, but other kind of conversion is possible as well. When working
with Boolean values, care must be taken because all non-empty strings,
including string False
, are considered true by Python. Robot Framework's own
robot.utils.is_truthy()
utility handles this nicely as it considers strings
like FALSE
, NO
and NONE
(case-insensitively) to be false:
from robot.utils import is_truthy
def example_keyword(count, case_insensitive):
count = int(count)
if is_truthy(case_insensitive):
# ...
Keywords can also use Robot Framework's argument conversion functionality via
the robot.api.TypeInfo class and its convert
method. This can be useful
if the needed conversion logic is more complicated or the are needs for better
error reporting than what simply using, for example, int()
provides.
from robot.api import TypeInfo
def example_keyword(count, case_insensitive):
count = TypeInfo.from_type(int).convert(count)
if TypeInfo.from_type(bool).convert(case_insensitive):
# ...
Tip
It is generally recommended to specify types using type hints or otherwise and let Robot Framework handle argument conversion automatically. Manual argument conversion should only be needed in special cases.
Note
robot.api.TypeInfo
is new in Robot Framework 7.0.
Starting from Robot Framework 3.1, arguments passed to keywords are automatically converted if argument type information is available and the type is recognized. The most natural way to specify types is using Python function annotations. For example, the keyword in the previous example could be implemented as follows and arguments would be converted automatically:
def example_keyword(count: int, case_insensitive: bool = True):
if case_insensitive:
# ...
See the Supported conversions section below for a list of types that are automatically converted and what values these types accept. It is an error if an argument having one of the supported types is given a value that cannot be converted. Annotating only some of the arguments is fine.
Annotating arguments with other than the supported types is not an error, and it is also possible to use annotations for other than typing purposes. In those cases no conversion is done, but annotations are nevertheless shown in the documentation generated by Libdoc.
Keywords can also have a return type annotation specified using the ->
notation at the end of the signature like def example() -> int:
.
This information is not used for anything during execution, but starting from
Robot Framework 7.0 it is shown by Libdoc for documentation purposes.
@keyword
decoratorAn alternative way to specify explicit argument types is using the
@keyword decorator. Starting from Robot Framework 3.1,
it accepts an optional types
argument that can be used to specify argument
types either as a dictionary mapping argument names to types or as a list
mapping arguments to types based on position. These approaches are shown
below implementing the same keyword as in earlier examples:
from robot.api.deco import keyword
@keyword(types={'count': int, 'case_insensitive': bool})
def example_keyword(count, case_insensitive=True):
if case_insensitive:
# ...
@keyword(types=[int, bool])
def example_keyword(count, case_insensitive=True):
if case_insensitive:
# ...
Regardless of the approach that is used, it is not necessarily to specify
types for all arguments. When specifying types as a list, it is possible
to use None
to mark that a certain argument does not have type information
and arguments at the end can be omitted altogether. For example, both of these
keywords specify the type only for the second argument:
@keyword(types={'second': float})
def example1(first, second, third):
# ...
@keyword(types=[None, float])
def example2(first, second, third):
# ...
Starting from Robot Framework 7.0, it is possible to specify the keyword return
type by using key 'return'
with an appropriate type in the type dictionary.
This information is not used for anything during execution, but it is shown by
Libdoc for documentation purposes.
If any types are specified using the @keyword
decorator, type information
got from annotations is ignored with that keyword. Setting types
to None
like @keyword(types=None)
disables type conversion altogether so that also
type information got from default values is ignored.
If type information is not got explicitly using annotations or the @keyword
decorator, Robot Framework 3.1 and newer tries to get it based on possible
argument default value. In this example count
and case_insensitive
get
types int
and bool
, respectively:
def example_keyword(count=-1, case_insensitive=True):
if case_insensitive:
# ...
When type information is got implicitly based on the default values, argument conversion itself is not as strict as when the information is got explicitly:
If an argument has an explicit type and a default value, conversion is first attempted based on the explicit type. If that fails, then conversion is attempted based on the default value. In this special case conversion based on the default value is strict and a conversion failure causes an error.
If argument conversion based on default values is not desired, the whole
argument conversion can be disabled with the @keyword decorator like
@keyword(types=None)
.
Note
Prior to Robot Framework 4.0 conversion was done based on the default value only if the argument did not have an explict type.
The table below lists the types that Robot Framework 3.1 and newer convert arguments to. These characteristics apply to all conversions:
Note
If an argument has both a type hint and a default value, conversion is
first attempted based on the type hint and then, if that fails, based on
the default value type. This behavior is likely to change in the future
so that conversion based on the default value is done only if the argument
does not have a type hint. That will change conversion behavior in cases
like arg: list = None
where None
conversion will not be attempted
anymore. Library creators are strongly recommended to specify the default
value type explicitly like arg: list | None = None
already now.
The type to use can be specified either using concrete types (e.g. list),
by using abstract base classes (ABC) (e.g. Sequence), or by using sub
classes of these types (e.g. MutableSequence). Also types in in the typing
module that map to the supported concrete types or ABCs (e.g. List
) are
supported. In all these cases the argument is converted to the concrete type.
In addition to using the actual types (e.g. int
), it is possible to specify
the type using type names as a string (e.g. 'int'
) and some types also have
aliases (e.g. 'integer'
). Matching types to names and aliases is
case-insensitive.
The Accepts column specifies which given argument types are converted. If the given argument already has the expected type, no conversion is done. Other types cause conversion failures.
Type | ABC | Aliases | Accepts | Explanation | Examples |
---|---|---|---|---|---|
bool | boolean | str, int, float, None | Strings True and false strings can be localized. See the Translations appendix for supported translations. |
TRUE (converted to True )off (converted to False )example (used as-is) |
|
int | Integral | integer, long | str, float | Conversion is done using the int built-in function. Floats
are accepted only if they can be represented as integers
exactly. For example, Starting from Robot Framework 4.1, it is possible to use
hexadecimal, octal and binary numbers by prefixing values with
Starting from Robot Framework 4.1, spaces and underscores can be used as visual separators for digit grouping purposes. Starting from Robot Framework 7.0, strings representing floats
are accepted as long as their decimal part is zero. This
includes using the scientific notation like |
42 -1 10 000 000 1e100 0xFF 0o777 0b1010 0xBAD_C0FFEE ${1} ${1.0} |
float | Real | double | str, Real | Conversion is done using the float built-in. Starting from Robot Framework 4.1, spaces and underscores can be used as visual separators for digit grouping purposes. |
3.14 2.9979e8 10 000.000 01 10_000.000_01 |
Decimal | str, int, float | Conversion is done using the Decimal class. Decimal is recommended over float when decimal numbers need to be represented exactly. Starting from Robot Framework 4.1, spaces and underscores can be used as visual separators for digit grouping purposes. |
3.14 10 000.000 01 10_000.000_01 |
||
str | string, unicode | Any | All arguments are converted to Unicode strings. New in Robot Framework 4.0. |
||
bytes | str, bytearray | Strings are converted to bytes so that each Unicode code point below 256 is directly mapped to a matching byte. Higher code points are not allowed. | good hyvä (converted to hyv\xe4 )\x00 (the null byte) |
||
bytearray | str, bytes | Same conversion as with bytes, but the result is a bytearray. | |||
datetime | str, int, float | Strings are expected to be timestamps in ISO 8601 like
format Integers and floats are considered to represent seconds since the Unix epoch. |
2022-02-09T16:39:43.632269 2022-02-09 16:39 2022-02-09 ${1644417583.632269} (Epoch time) |
||
date | str | Same string conversion as with datetime, but all time components are expected to be omitted or to be zeros. | 2018-09-12 |
||
timedelta | str, int, float | Strings are expected to represent a time interval in one of the time formats Robot Framework supports: time as number, time as time string or time as "timer" string. Integers and floats are considered to be seconds. | 42 (42 seconds)1 minute 2 seconds 01:02 (same as above) |
||
Path | PathLike | str | Strings are converted to pathlib.Path objects.
On Windows New in Robot Framework 6.0. |
/tmp/absolute/path relative/path/to/file.ext name.txt |
|
Enum | str | The specified type must be an enumeration (a subclass of Enum or Flag) and given arguments must match its member names. Matching member names is case, space, underscore and hyphen insensitive, but exact matches have precedence over normalized matches. Ignoring hyphens is new in Robot Framework 7.0. Enumeration documentation and members are shown in documentation generated by Libdoc automatically. |
class Direction(Enum):
"""Move direction."""
NORTH = auto()
NORTH_WEST = auto()
def kw(arg: Direction):
...
NORTH (Direction.NORTH)north west (Direction.NORTH_WEST) |
||
IntEnum | str, int | The specified type must be an integer based enumeration (a subclass of IntEnum or IntFlag) and given arguments must match its member names or values. Matching member names works the same way as with Enumeration documentation and members are shown in documentation generated by Libdoc automatically. New in Robot Framework 4.1. |
class PowerState(IntEnum):
"""Turn system ON or OFF."""
OFF = 0
ON = 1
def kw(arg: PowerState):
...
OFF (PowerState.OFF)1 (PowerState.ON) |
||
Literal | Any | Only specified values are accepted. Values can be strings,
integers, bytes, Booleans, enums and Strings are case, space, underscore and hyphen insensitive, but exact matches have precedence over normalized matches.
New in Robot Framework 7.0. |
def kw(arg: Literal['ON', 'OFF']):
...
OFF on |
||
None | str | String NONE (case-insensitive) is converted to the Python
None object. Other values cause an error. |
None |
||
Any | Any | Any value is accepted. No conversion is done. New in Robot Framework 6.1. |
|||
list | Sequence | sequence | str, Sequence | Strings must be Python list literals. They are converted
to actual lists using the ast.literal_eval function.
They can contain any values If the used type hint is list (e.g. Alias |
['one', 'two'] [('one', 1), ('two', 2)] |
tuple | str, Sequence | Same as list , but string arguments must tuple literals. |
('one', 'two') |
||
set | Set | str, Container | Same as list , but string arguments must be set literals or
set() to create an empty set. |
{1, 2, 3, 42} set() |
|
frozenset | str, Container | Same as set , but the result is a frozenset. |
{1, 2, 3, 42} frozenset() |
||
dict | Mapping | dictionary, mapping, map | str, Mapping | Same as Alias |
{'a': 1, 'b': 2} {'key': 1, 'nested': {'key': 2}} |
TypedDict | str, Mapping | Same as New in Robot Framework 6.0. Normal |
class Config(TypedDict):
width: int
enabled: bool
{'width': 1600, 'enabled': True} |
Note
Starting from Robot Framework 5.0, types that have a converted are automatically shown in Libdoc outputs.
Note
Prior to Robot Framework 4.0, most types supported converting string NONE
(case-insensitively) to Python
None
. That support has been removed and None
conversion is only done if an argument has None
as an
explicit type or as a default value.
Starting from Robot Framework 4.0, it is possible to specify that an argument has multiple possible types. In this situation argument conversion is attempted based on each type and the whole conversion fails if none of these conversions succeed.
When using function annotations, the natural syntax to specify that an argument has multiple possible types is using Union:
from typing import Union
def example(length: Union[int, float], padding: Union[int, str, None] = None):
...
When using Python 3.10 or newer, it is possible to use the native type1 | type2 syntax instead:
def example(length: int | float, padding: int | str | None = None):
...
Robot Framework 7.0 enhanced the support for the union syntax so that also
"stringly typed" unions like 'type1 | type2'
work. This syntax works also
with older Python versions:
def example(length: 'int | float', padding: 'int | str | None' = None):
...
An alternative is specifying types as a tuple. It is not recommended with annotations,
because that syntax is not supported by other tools, but it works well with
the @keyword
decorator:
from robot.api.deco import keyword
@keyword(types={'length': (int, float), 'padding': (int, str, None)})
def example(length, padding=None):
...
With the above examples the length
argument would first be converted to an
integer and if that fails then to a float. The padding
would be first
converted to an integer, then to a string, and finally to None
.
If the given argument has one of the accepted types, then no conversion is done
and the argument is used as-is. For example, if the length
argument gets
value 1.5
as a float, it would not be converted to an integer. Notice that
using non-string values like floats as an argument requires using variables as
these examples giving different values to the length
argument demonstrate:
*** Test Cases ***
Conversion
Example 10 # Argument is a string. Converted to an integer.
Example 1.5 # Argument is a string. Converted to a float.
Example ${10} # Argument is an integer. Accepted as-is.
Example ${1.5} # Argument is a float. Accepted as-is.
If one of the accepted types is string, then no conversion is done if the given
argument is a string. As the following examples giving different values to the
padding
argument demonstrate, also in these cases passing other types is
possible using variables:
*** Test Cases ***
Conversion
Example 1 big # Argument is a string. Accepted as-is.
Example 1 10 # Argument is a string. Accepted as-is.
Example 1 ${10} # Argument is an integer. Accepted as-is.
Example 1 ${None} # Argument is `None`. Accepted as-is.
Example 1 ${1.5} # Argument is a float. Converted to an integer.
If the given argument does not have any of the accepted types, conversion is attempted in the order types are specified. If any conversion succeeds, the resulting value is used without attempting remaining conversions. If no individual conversion succeeds, the whole conversion fails.
If a specified type is not recognized by Robot Framework, then the original argument value is used as-is. For example, with this keyword conversion would first be attempted to an integer, but if that fails the keyword would get the original argument:
def example(argument: Union[int, Unrecognized]):
...
Starting from Robot Framework 6.1, the above logic works also if an unrecognized
type is listed before a recognized type like Union[Unrecognized, int]
.
Also in this case int
conversion is attempted, and the argument id passed as-is
if it fails. With earlier Robot Framework versions, int
conversion would not be
attempted at all.
With generics also the parameterized syntax like list[int]
or dict[str, int]
works. When this syntax is used, the given value is first converted to the base
type and then individual items are converted to the nested types. Conversion
with different generic types works according to these rules:
list[float]
. All list items are
converted to that type.tuple[int, int]
and
tuple[str, int, bool]
. Tuples used as arguments are expected to have
exactly that amount of items and they are converted to matching types.tuple[int, ...]
. In this case tuple can have any number
of items and they are all converted to the specified type.dict[str, int]
.
Dictionary keys are converted using the former type and values using the latter.set[float]
. Conversion logic
is the same as with lists.Using the native list[int]
syntax requires Python 3.9 or newer. If there
is a need to support also earlier Python versions, it is possible to either use
matching types from the typing module like List[int]
or use the "stringly typed"
syntax like 'list[int]'
.
Note
Support for converting nested types with generics is new in Robot Framework 6.0. Same syntax works also with earlier versions, but arguments are only converted to the base type and nested types are not used for anything.
Note
Support for "stringly typed" parameterized generics is new in Robot Framework 7.0.
In addition to doing argument conversion automatically as explained in the previous sections, Robot Framework supports custom argument conversion. This functionality has two main use cases:
Argument converters are functions or other callables that get arguments used
in data and convert them to desired format before arguments are passed to
keywords. Converters are registered for libraries by setting
ROBOT_LIBRARY_CONVERTERS
attribute (case-sensitive) to a dictionary mapping
desired types to converts. When implementing a library as a module, this
attribute must be set on the module level, and with class based libraries
it must be a class attribute. With libraries implemented as classes, it is
also possible to use the converters
argument with the @library decorator.
Both of these approaches are illustrated by examples in the following sections.
Note
Custom argument converters are new in Robot Framework 5.0.
Let's assume we wanted to create a keyword that accepts date objects for
users in Finland where the commonly used date format is dd.mm.yyyy
.
The usage could look something like this:
*** Test Cases ***
Example
Keyword 25.1.2022
Automatic argument conversion supports dates, but it expects them
to be in yyyy-mm-dd
format so it will not work. A solution is creating
a custom converter and registering it to handle date conversion:
from datetime import date
# Converter function.
def parse_fi_date(value):
day, month, year = value.split('.')
return date(int(year), int(month), int(day))
# Register converter function for the specified type.
ROBOT_LIBRARY_CONVERTERS = {date: parse_fi_date}
# Keyword using custom converter. Converter is resolved based on argument type.
def keyword(arg: date):
print(f'year: {arg.year}, month: {arg.month}, day: {arg.day}')
If we try using the above keyword with invalid argument like invalid
, it
fails with this error:
ValueError: Argument 'arg' got value 'invalid' that cannot be converted to date: not enough values to unpack (expected 3, got 1)
This error is not too informative and does not tell anything about the expected
format. Robot Framework cannot provide more information automatically, but
the converter itself can be enhanced to validate the input. If the input is
invalid, the converter should raise a ValueError
with an appropriate message.
In this particular case there would be several ways to validate the input, but
using regular expressions makes it possible to validate both that the input
has dots (.
) in correct places and that date parts contain correct amount
of digits:
from datetime import date
import re
def parse_fi_date(value):
# Validate input using regular expression and raise ValueError if not valid.
match = re.match(r'(\d{1,2})\.(\d{1,2})\.(\d{4})$', value)
if not match:
raise ValueError(f"Expected date in format 'dd.mm.yyyy', got '{value}'.")
day, month, year = match.groups()
return date(int(year), int(month), int(day))
ROBOT_LIBRARY_CONVERTERS = {date: parse_fi_date}
def keyword(arg: date):
print(f'year: {arg.year}, month: {arg.month}, day: {arg.day}')
With the above converter code, using the keyword with argument invalid
fails
with a lot more helpful error message:
ValueError: Argument 'arg' got value 'invalid' that cannot be converted to date: Expected date in format 'dd.mm.yyyy', got 'invalid'.
By default Robot Framework tries to use converters with all given arguments
regardless their type. This means that if the earlier example keyword would
be used with a variable containing something else than a string, conversion
code would fail in the re.match
call. For example, trying to use it with
argument ${42}
would fail like this:
ValueError: Argument 'arg' got value '42' (integer) that cannot be converted to date: TypeError: expected string or bytes-like object
This error situation could naturally handled in the converter code by checking the value type, but if the converter only accepts certain types, it is typically easier to just restrict the value to that type. Doing it requires only adding appropriate type hint to the converter:
def parse_fi_date(value: str):
# ...
Notice that this type hint is not used for converting the value before calling
the converter, it is used for strictly restricting which types can be used.
With the above addition calling the keyword with ${42}
would fail like this:
ValueError: Argument 'arg' got value '42' (integer) that cannot be converted to date.
If the converter can accept multiple types, it is possible to specify types as a Union. For example, if we wanted to enhance our keyword to accept also integers so that they would be considered seconds since the Unix epoch, we could change the converter like this:
from datetime import date
import re
from typing import Union
# Accept both strings and integers.
def parse_fi_date(value: Union[str, int]):
# Integers are converted separately.
if isinstance(value, int):
return date.fromtimestamp(value)
match = re.match(r'(\d{1,2})\.(\d{1,2})\.(\d{4})$', value)
if not match:
raise ValueError(f"Expected date in format 'dd.mm.yyyy', got '{value}'.")
day, month, year = match.groups()
return date(int(year), int(month), int(day))
ROBOT_LIBRARY_CONVERTERS = {date: parse_fi_date}
def keyword(arg: date):
print(f'year: {arg.year}, month: {arg.month}, day: {arg.day}')
A problem with the earlier example is that date objects could only be given
in dd.mm.yyyy
format. It would not work if there was a need to
support dates in different formats like in this example:
*** Test Cases ***
Example
Finnish 25.1.2022
US 1/25/2022
ISO 8601 2022-01-22
A solution to this problem is creating custom types instead of overriding the default date conversion:
from datetime import date
import re
from typing import Union
from robot.api.deco import keyword, library
# Custom type. Extends an existing type but that is not required.
class FiDate(date):
# Converter function implemented as a classmethod. It could be a normal
# function as well, but this way all code is in the same class.
@classmethod
def from_string(cls, value: str):
match = re.match(r'(\d{1,2})\.(\d{1,2})\.(\d{4})$', value)
if not match:
raise ValueError(f"Expected date in format 'dd.mm.yyyy', got '{value}'.")
day, month, year = match.groups()
return cls(int(year), int(month), int(day))
# Another custom type.
class UsDate(date):
@classmethod
def from_string(cls, value: str):
match = re.match(r'(\d{1,2})/(\d{1,2})/(\d{4})$', value)
if not match:
raise ValueError(f"Expected date in format 'mm/dd/yyyy', got '{value}'.")
month, day, year = match.groups()
return cls(int(year), int(month), int(day))
# Register converters using '@library' decorator.
@library(converters={FiDate: FiDate.from_string, UsDate: UsDate.from_string})
class Library:
# Uses custom converter supporting 'dd.mm.yyyy' format.
@keyword
def finnish(self, arg: FiDate):
print(f'year: {arg.year}, month: {arg.month}, day: {arg.day}')
# Uses custom converter supporting 'mm/dd/yyyy' format.
@keyword
def us(self, arg: UsDate):
print(f'year: {arg.year}, month: {arg.month}, day: {arg.day}')
# Uses IS0-8601 compatible default conversion.
@keyword
def iso_8601(self, arg: date):
print(f'year: {arg.year}, month: {arg.month}, day: {arg.day}')
# Accepts date in different formats.
@keyword
def any(self, arg: Union[FiDate, UsDate, date]):
print(f'year: {arg.year}, month: {arg.month}, day: {arg.day}')
Converters are not used at all if the argument is of the specified type to
begin with. It is thus easy to enable strict type validation with a custom
converter that does not accept any value. For example, the Example
keyword accepts only StrictType
instances:
class StrictType:
pass
def strict_converter(arg):
raise TypeError(f'Only StrictType instances accepted, got {type(arg).__name__}.')
ROBOT_LIBRARY_CONVERTERS = {StrictType: strict_converter}
def example(argument: StrictType):
assert isinstance(argument, StrictType)
As a convenience, Robot Framework allows setting converter to None
to get
the same effect. For example, this code behaves exactly the same way as
the code above:
class StrictType:
pass
ROBOT_LIBRARY_CONVERTERS = {StrictType: None}
def example(argument: StrictType):
assert isinstance(argument, StrictType)
Note
Using None
as a strict converter is new in Robot Framework 6.0.
An explicit converter function needs to be used with earlier versions.
Starting from Robot Framework 6.1, it is possible to access the library instance from a converter function. This allows defining dynamic type conversions that depend on the library state. For example, if the library can be configured to test particular locale, you might use the library state to determine how a date should be parsed like this:
from datetime import date
import re
def parse_date(value, library):
# Validate input using regular expression and raise ValueError if not valid.
# Use locale based from library state to determine parsing format.
if library.locale == 'en_US':
match = re.match(r'(?P<month>\d{1,2})/(?P<day>\d{1,2})/(?P<year>\d{4})$', value)
format = 'mm/dd/yyyy'
else:
match = re.match(r'(?P<day>\d{1,2})\.(?P<month>\d{1,2})\.(?P<year>\d{4})$', value)
format =