Project pages

The number one place to find more information about Robot Framework is the project web site at http://robotframework.org. This User Guide is naturally available there but you can also find more documentation, an issue tracker, downloads, source code and links to other related projects. Robot Framework is hosted on Google Code which provides excellent free services for open source projects.

Mailing lists

There are several Robot Framework mailing lists where to ask and search for more information. The mailing list archives are open for everyone (including the search engines) and everyone can also join these lists freely. Only list members can send mails, though, and to prevent spam new users are moderated which means that it might take a little time before your first message goes through. Do not be afraid to send question to mailing lists but remember How To Ask Questions The Smart Way.

robotframework-users

General discussion about all Robot Framework related issues. Questions and problems can be sent to this list. Used also for information sharing for all users.

robotframework-announce

An announcements-only mailing list where only moderators can send messages. All announcements are sent also to the robotframework-users mailing list so there is no need to join both lists.

robotframework-devel

Discussion about Robot Framework development.

robotframework-commit

Automatically generated mails about commits to the version control system, build results, new and edited issues, and so on. Can be used to follow Robot Framework development.

Python installation

Python 2.6 or 2.5 are recommended, although Python 2.4 and 2.3 are also supported. On most UNIX-like systems, you have Python installed by default. If you are on Windows or otherwise need to install Python yourself, your best place to start is probably the Python homepage. There you can download a suitable installer and get more information about the installation and Python in general.

Jython installation

Using test libraries implemented with Java or using Java tools directly requires running Robot Framework on Jython, which then requires Java Runtime Environment (JRE). The minimum required Java version is 1.4, but newer versions are recommended, as they tend to be faster with dynamic languages, such as Jython. Both Sun and IBM Java versions are supported.

Robot Framework requires Jython version 2.2. The earlier Jython version 2.1 is not compatible with Robot Framework, and Jython 2.2 betas and alphas are not supported either. Unfortunately, also Jython 2.2.1 has certain Unicode problems (for more information, see http://bugs.jython.org/issue1802339 and http://bugs.jython.org/issue1032).

Installing Jython is a fairly easy procedure. First you need to get an installer from the Jython homepage or directly from http://downloads.sourceforge.net/jython/jython_installer-2.2.jar. Note that the installer is an executable JAR package, which you need to run as java -jar jython_installer-2.2.jar. Depending on your system, the installer runs either in the graphical or the textual mode, but in both cases, the actual installation procedure is very easy.

Jython 2.2 does not have Python's distutils module, which is required for automatically installing Robot Framework. The normal procedure is installing Robot Framework using Python but manual installation is also possible. When installing Robot Framework with Python, its installer tries to find the Jython executable on the system to create the jybot runner script correctly. Jython is found if:

1. Jython can be executed in the system directly (i.e. it is in the PATH).

2. An environment variable JYTHON_HOME is set and it points to the Jython installation directory.

3. The installer finds the Jython installation directory from the system. On Windows, it is searched from the C:\ and D:\ drives, and on other systems from the /usr/local and /opt directories. The directory is found if it is under the searched directories mentioned above, or one level deeper. For example, the following Jython installation directories would be found by the installer:

   C:\APPS\Jython2.2
   D:\Jython22
   /usr/local/jython2.2
   /opt/whatever/Jython22
   

If you plan to use Robot Framework only with Jython, you do not necessarily need Python at all. In that case, you need to perform a manual installation or have some custom installer.

Installing from source

You can get Robot Framework source code either directly from version control or as a source distribution package that needs to be extracted somewhere. In both cases, you should have a directory containing the source code, documentation, tools, templates, and so on.

You should be able to install Robot Framework to any environment where Python runs using a source distribution. The installation is done by going to the created directory from the command line, and after that running either of the following commands:

python setup.py install
python install.py install

setup.py is a standard Python installer script. It can take several parameters allowing, for example, installation into non-default locations not requiring administrative rights. It is also be used also for creating distribution packages.

install.py is a custom installation and uninstallation script for Robot Framework. When it is used for installation, it simply uses setup.py, and thus above commands are totally equivalent.

With both of these commands you get a rather long output, and something like the following text should appear at the end:

Creating Robot Framework start-up scripts...
Installation directory: /usr/lib/python2.5/site-packages/robot
Python executable: /usr/bin/python
Jython executable: /cygdrive/c/jython2.2b2/jython.bat (found from system)
Pybot script: /usr/bin/pybot
Jybot script: /usr/bin/jybot
Rebot script: /usr/bin/rebot

Using Windows installer

The installation itself requires only double clicking the provided Windows installer and following instructions. After the installation you probably want to set the environment so that Robot Framework start-up scripts can be executed easily. Notice that the framework is automatically installed under the Python installation and that location cannot be altered at this point.

Using Easy Install

An obvious precondition for using Easy Install is to have it installed, and you can refer to its documentation on how to do that on your operating system. The command to install Robot Framework with Easy Install depends on do you want to install the latest version or some specific version:

easy_install robotframework         # latest version
easy_install robotframework==2.0.4  # specified version (2.0.4)

If you need to use a proxy to access the Internet, you can tell Easy Install to use it by setting the http_proxy environment variable.

A part of the Robot Framework installation is updating pybot, jybot and rebot runner scripts to work on the system where the framework is installed. How this is done is slightly fragile and it does not succeed with Easy Install on all environments. If automatically updating the runner script fails, executing them will not succeed unless they are fixed manually. Two known situations where the runner scripts are not updated correctly are explained below:

• On Windows you always need to run robot_postinstall.py script after the installation to configure the runner scripts. The installation output should tell where the post-install script is located, and you can execute it by double clicking it or running it from command line like python C:\Python25\Scripts\robot_postinstall.py.
• On OS X using Easy Install with virtualenv fails because the installation picks up the system wide Python installation (issue 236). A workaround is updating the runner scripts manually after the installation or installing from source.

Using One Click Installer

The One Click Installer installs Robot Framework and its preconditions Python and Jython (optional). It also automatically sets up environment so that Robot Framework start-up scripts, as well as Python and Jython executables are in PATH.

The One Click Installer requires that you have downloaded all the required component installers and have them in the same directory with it. More detailed instructions and details about the supported installers are available at One Click Installer's documentation.

Manual installation

If you do not want to install Python, or for some other reason do not want to use any automatic way of installing Robot Framework, you can always do it manually following these steps:

1. Get the source code. All the code is in a directory (a module in Python) called robot. If you have a source distribution or a version control checkout, you can find it from the src directory, but you can also get it from an earlier installation.
2. Copy the source code where you want to.
3. Create the needed runner scripts. If you have a source package or a checkout, you can get templates from src/bin directory.

Where files are installed

When an automatic installer is used, the Robot Framework code is copied into a directory containing external Python modules. The actual location is platform-specific, but on computers with a UNIX-like operating system, it is normally something like /usr/lib/[PythonVer]/site-packages, and on Windows it is [PythonInstallationDir]\Lib\site-packages. The actual Robot Framework code is in a directory named robot, or when using Easy Install in directory robotframework[RobotVer].py[PythonVer].egg/robot.

Robot Framework runner scripts (pybot, jybot and rebot) are created and copied into another platform-specific location. On UNIX-like systems, they normally go to /usr/bin and are thus immediately available from the command line. On Windows, the operating system does not provide a similar natural place, and Python copies these scripts into [PythonInstallationDir]\Scripts.

Setting up environment

After the installation, you might want to make Robot Framework's runner scripts easily available from the command line. On UNIX-like systems, that should be the case automatically, but for example on Windows, it is not. In environments where startup scripts are not available, the directory containing them must be set to the PATH environment variable.

Setting the PATH environment variable on Windows:

1. Open Start > Settings > Control Panel > System > Advanced > Environment Variables. There are User variables and System variables, and the difference between them is that User variables affect only the current users, whereas System variables affect all users.
2. To edit the existing PATH, select Edit and add ;[PythonInstallationDir]\Scripts\ at the end of the value. Note that the leading colon (;) is important, as it separates different entries. To add a new value, select New and provide both the name and the value, this time without the colon.
3. Start a new command prompt for the changes to take effect.

Verifying installation

To verify that the installation and environment setup were successful, type:

$ pybot --version
Robot Framework 2.1 (Python 2.5.1 on darwin)

To verify that Robot Framework works also with Jython, type:

$ jybot --version
Robot Framework 2.1 (Jython 2.2 on java1.6.0_03)

In both cases, the exact version and platform information can, of course, differ from these. On Jython, you may also get some notifications from Jython package manager upon the first execution.

HTML format

In HTML files, the test data is defined in separate tables (see the example below). Robot Framework recognizes these test data tables based on the text in their first cell. Everything outside recognized tables is ignored.

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

<?xml version="1.0" encoding="Big5"?>

TSV format

The TSV format can be used in Robot Framework's test data for all the same purposes as HTML. In a TSV file, all the data is in one large table. Test data tables are recognized from one or more asterisks (*), followed by a normal table name and an optional closing asterisks. Everything before the first recognized table is ignored similarly as data outside tables in HTML data.

Plain text format

The plain text format is technically otherwise similar to the TSV format but the separator between cells is different. The TSV format uses tabs, but in the plain text format you can use either two or more spaces or a pipe character surrounded with spaces ( | ). In both cases the test data tables must have one or more asterisk before its name similarly as with the TSV format, and everything before the first table is ignored.

Starting from Robot Framework 2.1.2 tabs in plain text files are automatically converted to two spaces. This allows using a single tab as a separator similarly as in the TSV format. Notice, however, that in the plain text format multiple tabs are considered to be a single separator whereas in the TSV format every tab would be a separator.

***Settings***
Library     OperatingSystem

***Variables***
${MESSAGE}  Hello, world!

***Test Cases***
My Test  [Documentation]  Example test
    Log         ${MESSAGE}
    My Keyword  /tmp

Another Test
    Should Be Equal  ${MESSAGE}  Hello, world!

***Keywords***
My Keyword  [Arguments]  ${path}
    Directory Should Exist  ${path}

| *Setting*  |     *Value*     |
| Library    | OperatingSystem |

| *Variable* |     *Value*     |
| ${MESSAGE} | Hello, world!   |

| *Test Case*  | *Action*        | *Argument*   |
| My Test      | [Documentation] | Example test |
|              | Log             | ${MESSAGE}   |
|              | My Keyword      | /tmp         |
| Another Test | Should Be Equal | ${MESSAGE}   | Hello, world!

| *Keyword*  |
| My Keyword | [Arguments] | ${path}
|            | Directory Should Exist | ${path}

| ${file count} = | Execute Command | ls -1 *.txt \| wc -l |
| Should Be Equal | ${file count}   | 42                   |

reStructuredText format

reStructuredText (reST) is a easy-to-read plaintext markup syntax that is commonly used for documentation of Python projects (including Python itself, as well as this user guide). Using reST with Robot Framework allows you to mix richly formatted documents and tables that specify test data in a concise text format that is easy to work with using simple text editors, diff tools, and source control systems. The support for reStructuredText was added to Robot Framework in version 2.1.

Tools to process reStructuredText are freely available as part of the docutils project, and there is a quick reference guide that shows the most common formatting constructs including the tables used by Robot Framework. Notice that Robot Framework converts test data in reST format internally to HTML before starting to actually parse it. The data must thus follow reST syntax strictly or otherwise processing it will not succeed.

In reST files, test data is defined in tables within the document, similar to the HTML format. Robot Framework identifies test data tables based on the text in the first cell and all content outsided of the recognized table types is ignored.

An example of each of the four types of test data dable is shown below using the reST Simple Tables syntax. Note that \ or .. must be used to indicate an empty cell in the first column of the table:

============  ================  =======  =======
  Setting          Value         Value    Value
============  ================  =======  =======
Library       OperatingSystem
============  ================  =======  =======


============  ================  =======  =======
  Variable         Value         Value    Value
============  ================  =======  =======
${MESSAGE}    Hello, world!
============  ================  =======  =======


============  ===================  ============  =============
 Test Case          Action           Argument      Argument
============  ===================  ============  =============
My Test       [Documentation]      Example test
\             Log                  ${MESSAGE}
\             My Keyword           /tmp
\
Another Test  Should Be Equal      ${MESSAGE}    Hello, world!
============  ===================  ============  =============


============  ======================  ============  ==========
 Keyword            Action             Argument     Argument
============  ======================  ============  ==========
My Keyword    [Arguments]             ${path}
\             Directory Should Exist  ${path}
============  ======================  ============  ==========

Ignored data

When Robot Framework parses the test data, it ignores:

• All tables that do not start with a recognized table name in the first cell
• Everything else on the first row apart from the first cell
• Data outside tables in HTML/reST and data before the first table in TSV
• All empty rows, which means these kinds of rows can be used to make the tables more readable
• All empty cells at the end of rows; you must add a backslash (\) to prevent such cells from being ignored
• All single backslashes (\); they are used as an escape character
• All characters following a hash mark (#), if it is the first character of a cell; this means that hash marks can be used to enter comments in the test data
• All formatting in the HTML/reST test data

When Robot Framework ignores some data, this data is not available in any resulting reports and, additionally, most tools used with Robot Framework also ignore them. To add information that is visible in Robot Framework outputs, or available to, for example, Robot IDE, place it to the documentation or other metadata of test cases or suites, or log with the Log or Comment keywords available from the BuiltIn library.

Escaping

The escape character for the Robot Framework parser is the backslash (\). The escape character can be used as follows:

• To escape special characters so that their literal values are used:
  • \${notvar} means a literal string ${notvar} that looks like a variable
  • \\ means a single backslash (for example, C:\\Temp)
  • \# means a literal hash (#) mark, even at the beginning of a cell
• To affect the parsing of whitespaces
• To prevent ignoring empty cells at the end of a row (this requires \ to be in the appropriate cell). Another possibility is using built-in variable ${EMPTY}.

Handling whitespace

Robot Framework handles whitespace, such as spaces, newlines and tabs, the same way as they are handled in HTML. This means that Robot Framework:

• Removes leading and trailing whitespaces in all cells.
• Changes multiple consecutive spaces into single spaces.
• Converts all newlines and tabs into spaces.

To prevent Robot Framework from parsing data according to these rules, a backslash can be used:

• Before leading spaces, for example \ some text
• Between consecutive spaces, for example text \ \ more text
• After trailing spaces, for example some text \ \
• With n to create a newline, for example first line\n2nd line
• With t to create a tab character, for example text\tmore text
• With r to create a carriage return, for example text\rmore text

Another, and often clearer, possibility for representing leading, trailing, or consecutive spaces is using built-in variable ${SPACE}. The extended variable syntax even allows syntax like ${SPACE * 8} which makes handling consecutive spaces very simple.

Dividing test data to several rows

If a keyword requires more arguments than there are columns available, it is not necessary to add more columns. Instead, it is possible to simply have three dots (...) below the original keyword name and continue arguments there. Arguments presented like this are parsed as if they were all in one row.

The same approach works also with settings and variables taking several values. In these cases, the three dots are, of course, placed under the setting or variable name.

Additionally, values of settings that take only one value (mainly documentations) can be split to several columns. These values will be then catenated together with spaces when the test data is parsed.

All these syntaxes are illustrated in the following examples. The first three tables show tables where test data has not been split, and the following three illustrate how fewer columns are needed after splitting the data to several rows.

===========  ================  ==============  ==========  ==========
 Test Case       Action           Argument      Argument    Argument
===========  ================  ==============  ==========  ==========
Example      [Documentation]   Documentation   for this    test case.
\            ...               This can get    quite       long...
\            [Tags]            t-1             t-2         t-3
\            ...               t-4             t-5
\            Do X              one             two         three
\            ...               four            five        six
\            ${var} =          Get X           1           2
\            ...               3               4           5
\            ...               6
===========  ================  ==============  ==========  ==========

+-----------+-------------------+---------------+------------+------------+
| Test Case |      Action       |   Argument    |  Argument  |  Argument  |
+===========+===================+===============+============+============+
| Example   | [Documentation]   | Documentation | for this   | test case. |
+-----------+-------------------+---------------+------------+------------+
|           | ...               | This can get  | quite      | long...    |
+-----------+-------------------+---------------+------------+------------+
|           | [Tags]            | t-1           | t-2        | t-3        |
+-----------+-------------------+---------------+------------+------------+
|           | ...               | t-4           | t-5        |            |
+-----------+-------------------+---------------+------------+------------+
|           | Do X              | one           | two        | three      |
+-----------+-------------------+---------------+------------+------------+
|           | ...               | four          | five       | six        |
+-----------+-------------------+---------------+------------+------------+
|           | ${var} =          | Get X         | 1          | 2          |
+-----------+-------------------+---------------+------------+------------+
|           | ...               | 3             | 4          | 5          |
+-----------+-------------------+---------------+------------+------------+
|           | ...               | 6             |            |            |
+-----------+-------------------+---------------+------------+------------+

Basic syntax

Test cases are constructed in test case tables from the available keywords. Keywords can be imported from test libraries or resource files, or created in the keyword table of the test case file itself.

The first column in the test case table contains test case names. A test case starts from the row with something in this column and continues to the next test case name or to the end of the table. It is an error to have something between the table headers and the first test.

The second column normally has keyword names. An exception to this rule is setting variables from keyword return values, when the second and possibly also the subsequent columns contain variable names and a keyword name is located after them. In either case, columns after the keyword name contain possible arguments to the specified keyword.

Settings in the Test Case table

Test cases can also have their own settings. Setting names are always in the second column, where keywords normally are, and their values are in the subsequent columns. Setting names have square brackets around them to distinguish them from keywords. The available settings are listed below and explained later in this section.

[Documentation]

Used for specifying a test case documentation.

[Tags]

Used for tagging test cases.

[Setup], [Teardown]

Specify test setup and teardown. Have also synonyms [Precondition] and [Postcondition], respectively.

[Timeout]

Used for setting a test case timeout. Timeouts are discussed in their own section.

Test-case-related settings in the Setting table

The Setting table can have the following test-case-related settings. These settings are mainly default values for the test-case-specific settings listed earlier.

Force Tags, Default Tags

The forced and default values for tags.

Test Setup, Test Teardown

The default values for test setup and teardown. Have also synonyms Test Precondition and Test Postcondition, respectively.

Test Timeout

The default value for test case timeout. Timeouts are discussed in their own section.

Keyword-driven style

Workflow tests, such as the Valid Login test described earlier, are constructed from several keywords. Their normal structure is that first the system is taken into the initial state (Open Login Page in the Valid Login example), then something is done to the system (Input Name, Input Password, Submit Credentials), and finally it is verified that the system behaved as expected (Welcome Page Should Be Open).

Data-driven style

Another style to write test cases is the data-driven approach where test cases contain only one higher-level keyword, normally created as user keywords. These kinds of tests are very useful when there is a need to test the same workflow with different input or output data. This is demonstrated by the example below.

Note that in the example above, column headers have been changed to match the data. This is possible because on the first row other cells except the first one are ignored. Additionally, backslash is used for escaping empty cells at the end of the table.

Behavior-driven style

It is also possible to write test cases as requirements that also non-technical project stakeholders must understand. These Executable Requirements are a corner stone of a process commonly called Acceptance Test Driven Development (ATDD).

One way to write these requirements/tests is Given-When-Then style popularized by Behavior Driven Development (BDD). When writing test cases in this style, the initial state is usually expressed with a keyword starting with word Given, the actions are described with keyword starting with When and the expectations with a keyword starting with Then. Keyword starting with And may be used if a step has more than one action.

Initialization files

A test suite created from a directory can have the same settings (Documentation, Meta: <name>, Suite Setup, Suite Teardown) as a test case created from a file. Because a directory alone cannot have that kind of information, they must be placed into a special initialization file.

Initialization files have the same structure as test case files, except that they cannot have test case tables, and the overall syntax used in them is the same as in other test data files. An initialization file name must always be of the format __init__.extension, where the extension is the same as normally for test data files (for example, __init__.html or __init__.tsv). The name format is borrowed from Python, where files named in this manner denote that a directory is a module.

The main usage of initialization files is setting test-suite-related settings, but setting test-case-related settings is also possible. However, only Force Tags is generally useful, because others are only default values for the same settings in lower-level files, and even they are default values for the settings in the Test Case table.

Note that variables and keywords created in initialization files are not available elsewhere. If there is a need to share them, for example, with lower-level test suites, resource files must be used.

Using Library setting

Test libraries are normally imported using the Library setting in the Setting table and having the library name in the subsequent column. The library name is case-sensitive (it is the name of the module or class implementing the library and must be exactly correct), but any spaces in it are ignored. With Python libraries in modules or Java libraries in packages, the full name including the module or package name must be used. In those cases where the library needs arguments, they are listed in the columns after the library name. Both the library name and arguments can be set using variables.

It is possible to import test libraries in test case files, resource files and test suite initialization files. In all these cases, all the keywords in the imported library are available in that file. With resource files, those keywords are also available in other files using them.

Using Import Library keyword

Another possibility to take a test library into use is using the keyword Import Library from the BuiltIn library. This keyword takes the library name and possible arguments similarly as the Library setting. Keywords from the imported library are available in the test suite where the Import Library keyword was used. This approach is useful in cases where the library is not available when the test execution starts and only some other keywords make it available.

Library search path

The most common way to specify a test library to import is using its name, like it has been done in all the examples in this section. In these cases Robot Framework tries to find the class or module implementing the library from the library search path. Basically, this means that the library code and all its possible dependencies must be in PYTHONPATH or, when running tests on Jython, in a CLASSPATH. Setting the library search path is explained in a section of its own. Libraries can also set the search path automatically or have special instructions on how to do it. All standard libraries, for example, are in the library search path automatically.

The biggest benefit of this approach is that when the library search path has been configured, often using a custom start-up script, normal users do not need to think where libraries actually are installed. The drawback is that getting your own, possible very simple, libraries into the search path may require some additional configuration.

Using physical path to library

Another mechanism for specifying the library to import is using a path to it in the file system. This path is considered relative to the directory where current test data file is situated similarly as paths to resource and variable files. The main benefit of this approach is that there is no need to configure the library search path.

If the library is a file, the path to it must contain extension. For Python libraries the extension is naturally .py and for Java libraries it can either be .class or .java, but the class file must always be available. If Python library is implemented as a directory, the path to it must have a trailing forward slash (/). Following examples demonstrate these different usages.

A limitation of this approach is that libraries implemented as Python classes must be in a module with the same name as the class. Additionally, importing libraries distributed in JAR or ZIP packages is not possible with this mechanism.

BuiltIn library

The BuiltIn library provides a set of generic keywords needed often. It is imported automatically and thus always available. The provided keywords can be used, for example, for verifications (e.g. Should Be Equal, Should Contain), conversions (e.g. Convert To Integer) and for various other purposes (e.g. Log, Sleep, Run Keyword If, Set Global Variable).

The names of the keywords in the BuiltIn library have been renamed in Robot Framework version 1.8. All the old keywords still work, but the long names (the names visible in log files) of the keywords that are deprecated begin with DeprecatedBuiltIn. (for example, DeprecatedBuiltIn.Equals). It is highly recommended to use the new names of the keywords as the old versions will be removed altogether in the future.

For more information, see the BuiltIn library documentation.

OperatingSystem library

The OperatingSystem library enables various operating system related tasks to be performed in the system where Robot Framework is running. It can, among other things, execute commands (e.g. Run), create and remove files and directories (e.g. Create File, Remove Directory), check whether files or directories exists or contain something (e.g. File Should Exist, Directory Should Be Empty) and manipulate environment variables (e.g. Set Environment Variable).

The names of the keywords in the OperatingSystem library have been renamed in Robot Framework 1.8 similarly as the names of the BuiltIn keywords.

For more information, see the OperatingSystem library documentation.

Telnet library

The Telnet library makes it possible to connect to Telnet servers and execute commands on the opened connections.

For more information, see the Telnet library documentation.

Collections library

The Collections library provides a set of keywords for handling Python lists and dictionaries. This library has keywords, for example, for modifying and getting values from lists and dictionaries (e.g. Append To List, Get From Dictionary) and for verifying their contents (e.g. Lists Should Be Equal, Dictionary Should Contain Value).

For more information, see the Collections library documentation.

String library

The String library enables manipulating strings (e.g. Replace String With Regexp, Split To Lines) and verifying their contents (e.g. Should Be String).

For more information, see the String library documentation. This library is new in Robot Framework 2.1.

Dialogs library

The Dialogs library provides means for pausing the test execution and getting input from users. The dialogs are slightly different depending on are tests run on Python or Jython but they provide the same functionality.

For more information, see the Dialogs library documentation. This library is new in Robot Framework 2.1.

Screenshot library

The Screenshot library has keywords to capture and store screenshots of the whole desktop. This library is implemented with Java AWT APIs, so it can be used only when running Robot Framework on Jython.

For more information, see the Screenshot library documentation.

Remote library

The Remote library is totally different than the other standard libraries. It does not have any keywords of its own but it works as a proxy between Robot Framework and actual test library implementations. These libraries can be running on other machines than the core framework and can even be implemented using languages not supported by Robot Framework natively.

See separate Remote library interface section for more information about the concept. This library is new in Robot Framework 2.1.

Scalar variables

When scalar variables are used in the test data, they are replaced with the value they are assigned to. While scalar variables are most commonly used for simple strings, you can assign any objects, including lists, to them. The scalar variable syntax, for example ${NAME}, should be familiar to most users, as it is also used, for example, in shell scripts and Perl programming language.

The example below illustrates the usage of scalar variables. Assuming that the variables ${GREET} and ${NAME} are available and assigned to strings Hello and world, respectively, both the example test cases are equivalent.

When a scalar variable is used as the only value in a test data cell, the scalar variable is replaced with the value it has. The value may be any object. When a scalar variable is used in a test data cell with anything else (constant strings or other variables), its value is first converted into a string and then catenated to whatever is in that cell. Converting the value into a string means that the object's method __str__ (in Python) or toString (in Java) is called.

The example below demonstrates the difference between having a variable in a cell alone or with other content. First, let us assume that we have a variable ${STR} set to a string Hello, world! and ${OBJ} set to an instance of the following Java object:

public class MyObj {

    public String toString() {
        return "Hi, tellus!";
    }
}

With these two variables set, we then have the following test data:

Finally, when this test data is executed, different keywords receive the arguments as explained below:

• KW 1 gets a string Hello, world!
• KW 2 gets an object stored to variable ${OBJ}
• KW 3 gets a string I said "Hello, world!"
• KW 4 gets a string You said "Hi, tellus!"

List variables

List variables are compound variables that can have several values assigned to them. In short, they are always lists and can contain an unlimited number of entries (also empty lists are possible). The main benefit of list variables is that they allow you to assign a name for a larger data set. While list variables normally contain only strings, other content is also possible.

When you use a list variable in test data, then the cell that contains the variable is replaced with the content of the variable. Thus, if the list variable contains two elements, the cell containing the list variable is turned into two cells with the content of the list variable. Note that cells with list variables should not contain other content. The list variable syntax, @{NAME}, is borrowed from Perl.

Assuming that the list variable @{USER} is set to the value ['robot','secret'], the following two test cases are equivalent.

It is also possible to access a certain value from the list variable with the syntax @{NAME}[i], where i is the index of the selected value. Indexes start from zero, and trying to access a value with too large an index causes an error. List items accessed in this manner can be used similarly as scalar variables:

Starting from Robot Framework 2.0.3, it is possible to use list variables as scalar variables containing lists simply by replacing @ with $. This makes it possible to use list variables with list related keywords, for example, from from BuiltIn and Collections libraries. This feature works only if there is no scalar variable with same basename as the list variable has. In these cases the scalar variable has precedence and its value is used instead.

Environment variables

Robot Framework allows using environment variables in the test data using the syntax %{ENV_VAR_NAME}. They are limited to string values.

Environment variables set in the operating system before the test execution are available during it, and it is possible to create new ones with the keyword Set Environment Variable or delete existing ones with the keyword Delete Environment Variable, both available in the OperatingSystem library. Because environment variables are global, environment variables set in one test case can be used in other test cases executed after it. However, changes to environment variables are not effective after the test execution.

Variable table

The most common source for variables are Variable tables in test case files and resource files. Variable tables are convenient, because they allow creating variables in the same place as the rest of the test data, and the needed syntax is very simple. Their main disadvantage is that they only enable assigning variables into strings or a list of strings. If other value types are needed, variable files are probably a better option.

Variable file

Variable files are the most powerful mechanism for creating different kind of variables. It is possible to assign variables to any object using them, and they also enable creating variables dynamically. The variable file syntax and taking variable files into use is explained in section Resource and variable files.

Setting variables in command line

Variables can be set from the command line either individually with the --variable (-v) option or using a variable file with the --variablefile (-V) option. Variables set from the command line are globally available for all executed test data files, and they also override possible variables with the same names in the Variable table and in variable files imported in the test data.

The syntax for setting individual variables is --variable name:value, where name is the name of the variable without ${} and value is its value. Several variables can be set by using this option several times. Only scalar variables can be set using this syntax and they can only get string values. Many special characters are difficult to represent in the command line, but they can be escaped with the --escape option.

--variable EXAMPLE:value
--variable HOST:localhost:7272 --variable USER:robot
--variable ESCAPED:Qquotes_and_spacesQ --escape quot:Q --escape space:_

In the examples above, variables are set so that

• ${EXAMPLE} gets the value value
• ${HOST} and ${USER} get the values localhost:7272 and robot
• ${ESCAPED} gets the value "quotes and spaces"

The basic syntax for taking variable files into use from the command line is --variablefile path/to/variables.py, and Taking variable files into use section has more details. What variables actually are created depends on what variables there are in the referenced variable file.

If both variable files and individual variables are given from the command line, the latter have higher priority. Additionally, if multiple variable files have same variables, the ones in the file specified first are taken into use.

Return values from keywords

Return values from keywords can also be set into variables. This allows communication between different keywords even in different test libraries. The syntax for a simple case is illustrated in the example below:

In the example above, the value returned by the Get X keyword is first set into the variable ${x} and then used by the Log keyword. This syntax works in all cases where a keywords returns something, and the variable is set to whatever value returned by the keyword. Having the equals sign = after the variable name is not obligatory, but recommended, because it makes the assignment more explicit.

If a keyword returns a list, it is also possible to set it into several scalar variables or into a list variable. This is possible with keywords returning Python lists or tuples or, from Robot Framework version 1.8.6 onwards, with Java keywords returning an array. In the future, it is possible to add support also for other iterables, if needed.

Assuming that the keyword Get 3 returns a list [1, 2, 3], the following variables are created:

• ${scalar} with the value [1, 2, 3]
• ${a}, ${b} and ${c} with the values 1, 2, and 3, respectively
• ${first} with the value 1, and @{rest} with the value [2, 3]
• @{list} with the value [1, 2, 3]

Variables set in this manner are otherwise similar to any other variables, but they are available only within the scope of the test case or keyword where they are created. Thus it is not possible, for example, to set a variable in one test case and use it in another. This is because, in general, automated test cases should not depend on each other, and accidentally setting a variable that is used elsewhere could cause hard-to-debug errors. If there is a genuine need for setting a variable in one test case and using it in another, it is possible to use built-in keywords as explained in the next section.

Using built-in Set Test/Suite/Global Variable keywords

The BuiltIn library has keywords Set Test Variable, Set Suite Variable and Set Global Variable which can be used for setting variables dynamically during the test execution. If a variable already exists within the new scope, its value will be overwritten, and otherwise a new variable is created.

Variables set with Set Test Variable keyword are available everywhere within the scope of the currently executed test case. For example, if you set a variable in a user keyword, it is available both in the test case level and also in all other user keywords used in the current test. Other test cases will not see variables set with this keyword.

Variables set with Set Suite Variable keyword are available everywhere within the scope of the currently executed test suite. Setting variables with this keyword thus has the same effect as creating them using the Variable table in the test data file or importing them from variable files. Other test suites, including possible child test suites, will not see variables set with this keyword.

Variables set with Set Global Variable keyword are globally available in all test cases and suites executed after setting them. Setting variables with this keyword thus has the same effect as creating from the command line using the options --variable or --variablefile. Because this keyword can change variables everywhere, it should be used with care.

Operating-system variables

Built-in variables related to the operating system ease making the test data operating-system-agnostic.

Number variables

The variable syntax can be used for creating both integers and floating point numbers, as illustrated in the example below. This is useful when a keyword expects to get an actual number, and not a string that just looks like a number, as an argument.

Boolean and None/null variables

Also Boolean values and Python None and Java null can be created using the variable syntax similarly as numbers.

These variables are case-insensitive, so for example ${True} and ${true} are equivalent. Additionally, ${None} and ${null} are synonyms, because when running tests on the Jython interpreter, Jython automatically converts None and null to the correct format when necessary.

Space and empty variables

Starting from Robot Framework version 2.0.2, it is possible to create spaces and empty strings using variables ${SPACE} and ${EMPTY}, respectively. These variables are useful, for example, when there would otherwise be a need to escape spaces or empty cells with a backslash. If more than one space is needed, it is possible to use the extended variable syntax like ${SPACE * 5}. In the following exmaple, Should Be Equal keyword gets identical arguments but those using variables are easier to understand than those using backslashes.

Automatic variables

Some automatic variables can also be used in the test data. These variables can have different values during the test execution and some of them are not even available all the time.

Variable priorities

Variables from the command line

Variables set in the command line have the highest priority of all variables that can be set before the actual test execution starts. They override possible variables created in Variable tables in test case files, as well as in resource and variable files imported in the test data.

Individually set variables (--variable option) override the variables set using variable files (--variablefile option). Additionally, if multiple variable files have same variables, the ones in the file specified first have the highest priority.

Variable table in a test case file

Variables created using the Variable table in a test case file are available for all the test cases in that file. These variables override possible variables with same names in imported resource and variable files.

Variables created in the variable tables are available in all other tables in the file where they are created. This means that they can be used also in the Setting table, for example, for importing more variables from resource and variable files.

Imported resource and variable files

Variables imported from the resource and variable files have the lowest priority of all variables created in the test data. Variables from resource files and variable files have the same priority. If several resource and/or variable file have same variables, the ones in the file imported first are taken into use.

If a resource file imports resource files or variable files, variables in its own Variable table have a higher priority than variables it imports. All these variables are available for files that import this resource file.

Notice that variables imported from resource and variable files are not available in the Variable table of the file that imports them. This is due to the Variable table being processed before the Setting table where the resource files and variable files are imported.

Variables set during test execution

Variables set during the test execution either using return values from keywords or built-in keywords Set Test/Suite/Global Variable always override possible existing variables in the scope where they are set. In a sense they thus have the highest priority, but on the other hand they do not affect variables outside the scope they are defined.

Built-in variables

Built-in variables like ${TEMPDIR} and ${TEST_NAME} have the highest priority of all variables. They cannot be overridden using Variable table or from command line, but even they can be reset during the test execution. An exception to this rule are number variables, which are resolved dynamically if no variable is found otherwise. They can thus be overridden, but that is generally a bad idea. Additionally ${CURDIR} is special because it is replaced already during the test data processing time.

Variable scopes

Depending on where and how they are created, variables can have a global, test suite, test case or user keyword scope.

Global scope

Global variables are available everywhere in the test data. These variables are normally set from the command line with the --variable and --variablefile options, but it is also possible to create new global variables or change the existing ones with the BuiltIn keyword Set Global Variable anywhere in the test data. Additionally also built-in variables are global.

It is recommended to use capital letters with all global variables.

Test suite scope

Variables with the test suite scope are available anywhere in the test suite where they are defined or imported. They can be created in Variable tables, imported from resource and variable files, or set during the test execution using the BuiltIn keyword Set Suite Variable.

The test suite scope is not recursive, which means that variables available in a higher-level test suite are not available in lower-level suites. If necessary, resource and variable files can be used for sharing variables.

Since these variables can be considered global in the test suite where they are used, it is recommended to use capital letters also with them.

Test case scope

Variables created in test cases from the return values of keywords have a test case scope and they are available only in that test case. Another possibility to create them is using the BuiltIn keyword Set Test Variable anywhere in that particular test case. Test case variables are local and should use lower-case letters.

User keyword scope

User keywords get their own variables from arguments passed to them and return values from the keywords they use. Also these variables are local and should use lower-case letters.

Extended variable syntax

Extended variable syntax can be used with objects set into scalar variables. It allows accessing the attributes of the object (for example, ${obj.name} or ${obj.some_attr}), and even calling its methods (for example, ${obj.get_name()} or ${obj.getSomething('arg')}).

Extended variable syntax is a powerful feature, but it should be used with care. Accessing attributes is normally not a problem, on the contrary, as one variable with an object having several attributes is often better than having several variables. On the other hand, calling methods, especially when they are used with arguments, can make the test data complicated. If that happens, it is recommended to move the code into a test library.

The most common usages of extended variable syntax are illustrated in the example below. First assume that we have the following variable file and test case:

class MyObject:

    def __init__(self, name):
        self.name = name

    def greet(self, who):
        return '%s says hello to %s' % (self.name, who)

    def __str__(self):
        return self.name

OBJECT = MyObject('Robot')
DICTIONARY = { 1: 'one', 2: 'two', 3: 'three'}

When this test data is executed, the keywords get the arguments as explained below:

• KW 1 gets string Robot
• KW 2 gets string Robot says hello to Fit
• KW 3 gets string two

The extended variable syntax is evaluated in the following order:

1. The variable is searched using the full variable name. The extended variable syntax is evaluated only if no matching variable is found.
2. The real name of the base variable is created. The body of the name consists of all the characters after ${ until the first occurrence of a non-alphanumeric character or a space (for example, OBJECT in ${OBJECT.name} and DICTIONARY in ${DICTIONARY[2]}).
3. A variable matching the body is searched. If there is no match, an exception is raised and the test case fails.
4. The expression inside the curly brackets is evaluated as a Python expression, so that the base variable name is replaced with its value. If the evaluation fails because of an invalid syntax or that the queried attribute does not exist, an exception is raised and the test fails.
5. The whole extended variable is replaced with the value returned from the evaluation.

If the object that is used is implemented with Java, the extended variable syntax allows you to access attributes using so-called bean properties. In essence, this means that if you have an object with the getName method set into a variable ${OBJ}, then the syntax ${OBJ.name} is equivalent to, but clearer than ${OBJ.getName()}. Thus the Python object used in the previous example could be replaced with the following Java implementation:

public class MyObject:

    private String name;

    public MyObject(String name) {
        name = name;
    }

    public String getName() {
        return name;
    }

    public String greet(String who) {
        return name + " says hello to " + who;
    }

    public String toString() {
        return name;
    }
}

Many standard Python objects, including strings and numbers, have methods that can be used with the extended variable syntax either explicitly or implicitly. Sometimes this can be really useful and reduce the need for setting temporary variables, but it is also easy to overuse it and create really cryptic test data. Following examples show few pretty good usages.

Note that even though abs(number) is recommended over number.__abs__() in normal Python code, using ${abs(number)} does not work. This is because the variable name must be in the beginning of the extended syntax. Using __xxx__ methods in the test data like this is already a bit questionable, and it is normally better to move this kind of logic into test libraries.

Variables inside variables

Variables are allowed also inside variables, and when this syntax is used, variables are resolved from the inside out. For example, if you have a variable ${var${x}}, then ${x} is resolved first. If it has the value name, the final value is then the value of the variable ${varname}. There can be several nested variables, but resolving the outermost fails, if any of them does not exist.

In the example below, Do X gets the value ${JOHN_HOME} or ${JANE_HOME}, depending on if Get Name returns john or jane. If it returns something else, resolving ${${name}_HOME} fails.

Basic syntax

In many ways, the overall user keyword syntax is identical to the test case syntax. User keywords are created in keyword tables which differ from test case tables only by the name that is used to identify them. User keyword names are in the first column similarly as test cases names. Also user keywords are created from keywords, either from keywords in test libraries or other user keywords. Keyword names are normally in the second column, but when setting variables from keyword return values, they are in the subsequent columns.

Most user keywords take some arguments. This important feature is used already in the second example above, and it is explained in detail later in this section, similarly as user keyword return values.

User keywords can be created in test case files, resource files, and test suite initialization files. Keywords created in resource files are available for files using them, whereas other keywords are only available in the files where they are created.

Settings in the Keyword table

User keywords can have similar settings as test cases, and they have the same square bracket syntax separating them from keyword names. All available settings are listed below and explained later in this section.

[Documentation]

Used for setting a user keyword documentation.

[Arguments]

Specifies user keyword arguments.

[Return]

Specifies user keyword return values.

[Timeout]

Sets the possible user keyword timeout. Timeouts are discussed in a section of their own.

Positional arguments

The simplest way to specify arguments (apart from not having them at all) is using only positional arguments. In most cases, this is all that is needed.

The syntax is such that first the [Arguments] setting is given and then argument names are defined in the subsequent cells. Each argument is in its own cell, using the same syntax as with variables. The keyword must be used with as many arguments as there are argument names in its signature. The actual argument names do not matter to the framework, but from users' perspective they should should be as descriptive as possible. It is recommended to use lower-case letters in variable names, either as ${my_arg}, ${my arg} or ${myArg}.

Default values

Positional arguments are probably sufficient in most situations. However, sometimes it is useful to be able to have a keyword that takes a different number of arguments and has default values for those that are not given. Also user keywords allow this, and the needed new syntax does not add very much to the already discussed basic syntax. In short, default values are added to arguments, so that first there is the equals sign (=) and then the value, for example ${arg}=default. There can be many arguments with defaults, but they all must be given after the normal positional arguments.

As all Pythonistas must have already noticed, the syntax for specifying default arguments is heavily inspired by Python syntax for function default values.

Variable number of arguments

Sometimes, but more and more seldom, even default values are not enough and there is a need for a keyword accepting any number of arguments. User keywords support also this. All that is needed is having list variables such as @{varargs} as the last argument in the keyword signature. This syntax can be combined with the previously described positional arguments and default values, and at the end the list variable gets all the leftover arguments that do not match other arguments. The list variable can thus have any number of items, even zero.

Notice that if the last keyword above is used with more than one argument, the second argument ${opt} always gets the given value instead of the default value. This happens even if the given value is empty. The last example also illustrates how a variable number of arguments accepted by a user keyword can be used in a for loops. This combination of two rather advanced functions can sometimes be very useful.

Again, Pythonistas probably notice that the variable number of arguments syntax is very close to the one in Python.

Embedded arguments matching too much

One tricky part in using embedded arguments is making sure that the values used when calling the keyword match the correct arguments. This is a problem especially if there are multiple arguments and characters separating them may also appear in the given values. For example Select ${city} ${team} leads to unexpected results if used with city containing too parts like Select Los Angeles Lakers. One solution to this problem is quoting the arguments like Select "${city}" "${team}" or separating them from each others otherwise, but sometimes it is easiest to just use positional arguments instead.

The same problem occurs often when trying to create keywords ignoring given/when/then/and prefixes. For example ${name} goes home matches Given Janne goes home so that ${name} gets value Given Janne. Quotes around the variable, like "${name}" goes home resolve also this problem.

Behavior-driven development example

The biggest benefit of having arguments as part of the keyword name is that it makes it easier to use higher-level sentence-like keywords when writing test cases in behavior-driven style. The example below illustrates this. Notice also that prefixes Given, When and Then are left out of the keyword definitions.

Taking resource files into use

Resource files are imported using the Resource setting in the Settings table. The path to the resource file is given in the cell after the setting name.

If the path is given in an absolute format, it is used directly. In other cases, the resource file is first searched relatively to the directory where the importing file is located. If the file is not found there, it is then searched from the directories in PYTHONPATH. The path can contain variables, and it is recommended to use them to make paths system-independent (for example, ${RESOURCES}/login_resources.html or ${RESOURCE_PATH}). Additionally, slashes ("/") in the path are automatically changed to backslashes ("\") on Windows.

The user keywords and variables defined in a resource file are available in the file that takes that resource file into use. Similarly available are also all keywords and variables from the libraries, resource files and variable files imported by the said resource file.

Resource file structure

The higher-level structure of resource files is the same as that of test case files otherwise, but, of course, they cannot contain Test Case tables. Additionally, the Setting table in resource files can contain only import settings (Library, Resource, Variables) and Documentation. The Variable table and Keyword table are used exactly the same way as in test case files.

If several resource files have a user keyword with the same name, they must be used so that the keyword name is prefixed with the resource file name without the extension (for example, myresources.Some Keyword and common.Some Keyword). Moreover, if several resource files contain the same variable, the one that is imported first is taken into use.

Documenting resource files

Keywords created in a resource file can be documented using [Documentation] setting. Starting from Robot Framework 2.1 also the resource file itself can have Documentation in the Setting table similarly as test suites.

Both libdoc.py and Robot IDE use these documentations, and they are naturally available for anyone opening resource files. The first line of the documentation of a keyword is logged when it is run, but otherwise resource file documentations are ignored during the test execution.

Example resource file

Taking variable files into use

--variablefile myvariables.py
--variablefile path/variables.py
--variablefile /absolute/path/common.py
--variablefile taking_arguments.py:arg1:arg2

Creating variables directly

VARIABLE = "An example string"
ANOTHER_VARIABLE = "This is pretty easy!"
INTEGER = 42
STRINGS = ["one", "two", "kolme", "four"]
NUMBERS = [1, INTEGER, 3.14]

LIST__STRINGS = ["list", "of", "strings"]
LIST__MIXED = ["first value", -1.1, None, True]

from java.util import Hashtable

MAPPING = Hashtable()
MAPPING.put("one", 1)
MAPPING.put("two", 2)

MAPPING = { 'one': 1, 'two': 2}

class MyObject:
    def __init__(self, name):
        self.name = name

OBJ1 = MyObject('John')
OBJ2 = MyObject('Jane')

import os
import random
import time

USER = os.getlogin()                # current login name
RANDOM_INT = random.randint(0, 10)  # random integer in range [0,10]
CURRENT_TIME = time.asctime()       # timestamp like 'Thu Apr  6 12:45:21 2006'
if time.localtime()[3] > 12:
    AFTERNOON = True
else:
    AFTERNOON = False

import math

def get_area(diameter):
    radius = diameter / 2
    area = math.pi * radius * radius
    return area

AREA1 = get_area(1)
AREA2 = get_area(2)

import math as _math

def _get_area(diameter):
    radius = diameter / 2.0
    area = _math.pi * radius * radius
    return area

AREA1 = _get_area(1)
AREA2 = _get_area(2)

import math

__all__ = ['AREA1', 'AREA2']

def get_area(diameter):
    radius = diameter / 2.0
    area = math.pi * radius * radius
    return area

AREA1 = get_area(1)
AREA2 = get_area(2)

Getting variables from a special function

An alternative syntax for getting variables is having a special get_variables function (also camelCase syntax getVariables is possible) in the variable file. In this case, Robot Framework calls that function and it returns variables as a Python dictionary (similar to Java maps), with variable names as keys and variable values as values. Variables are considered to be scalars, unless prefixed with LIST__, and values can contain anything. The example below is identical to the first examples of creating variables directly.

def get_variables():
    variables = { "VARIABLE ": "An example string",
                  "ANOTHER_VARIABLE": "This is pretty easy!",
                  "INTEGER": 42,
                  "STRINGS": ["one", "two", "kolme", "four"],
                  "NUMBERS": [1, 42, 3.14],
                  "LIST__STRINGS": ["list", "of", "strings"],
                  "LIST__MIXED": ["first value", -1.1, None, True] }
    return variables

get_variables can also take arguments, which facilitates changing what variables actually are created. Arguments to the function are set just as any other arguments for a Python function. When taking variable files into use in the test data, arguments are specified in cells after the path to the variable file, and in the command line they are separated from the path with a colon.

The dummy example below shows how to use arguments with variable files. In a more realistic example, the argument could be a path to an external text file or database where to read variables from.

variables1 = { 'scalar': 'Scalar variable',
               'LIST__list': ['List','variable'] }
variables2 = { 'scalar' : 'Some other value',
               'LIST__list': ['Some','other','value'],
               'extra': 'variables1 does not have this at all' }

def get_variables(arg):
    if arg == 'one':
        return variables1
    else:
        return variables2

The most notable benefit of using get_variables instead of defining variables directly as global attributes of the variable file is the ability to use arguments. On the other hand, the main drawback is that this approach always requires some actual programming.

Keyword scopes

When only a keyword name is used and there are several keywords with that name, Robot Framework attempts to determine which keyword has the highest priority based on its scope. The keyword's scope is determined on the basis of how the keyword in question is created:

1. Created as a user keyword in the same file where it is used. These keywords have the highest priority and they are always used, even if there are other keywords with the same name elsewhere.
2. Created in a resource file and imported either directly or indirectly from another resource file. This is the second-highest priority.
3. Created in an external test library. These keywords are used, if there are no user keywords with the same name. However, if there is a keyword with the same name in the standard library, a warning is displayed.
4. Created in a standard library. These keywords have the lowest priority.

Specifying a keyword explicitly

Scopes alone are not a sufficient solution, because there can be keywords with the same name in several libraries or resources, and additionally, they provide a mechanism to use only the keyword of the highest priority. In such cases, it is possible to use the full name of the keyword, where the keyword name is prefixed with the name of the resource or library and a dot is a delimiter.

With library keywords, the long format means only using the format LibraryName.Keyword Name. For example, the keyword Run from the OperatingSystem library could be used as OperatingSystem.Run, even if there was another Run keyword somewhere else. If the library is in a module or package, the full module or package name must be used (for example, com.company.Library.Some Keyword). If a custom name is given to a library using the WITH NAME syntax, the specified name must be used also in the full keyword name.

Resource files are specified in the full keyword name, similarly as library names. The name of the resource is derived from the basename of the resource file without the file extension. For example, the keyword Example in a resource file myresources.html can be used as myresources.Example. Note that this syntax does not work, if several resource files have the same basename. In such cases, either the files or the keywords must be renamed. The full name of the keyword is case-, space- and underscore-insensitive, similarly as normal keyword names.

Specifying library search order

Starting from Robot Framework 2.1 it is possible to use the keyword Set Library Search Order from BuiltIn library to specify the priority order between test libraries that are in use. If multiple keywords with the same name are found, libraries implementing them are gone through in the library search order. The first library containing the keyword is selected and the keyword from that library is used. If keyword is not found, test execution fails.

There is no need to use the long format LibraryName.Keyword Name notation when library search order is set. This makes it easier to to use multiple instances of the same library as Set Library Search Order keyword can be used to switch between the libraries. This enables using same user keywords instead of duplicating those for all the library instances.

Library search order is valid in the suite level meaning it is available only in the suite where it was set.

Test case timeout

The test case timeout can be set either by using the Test Timeout setting in the Setting table or the [Timeout] setting in the Test Case table. Test Timeout in the Setting table defines a default test timeout value for all the test cases in the test suite, whereas [Timeout] in the Test Case table applies a timeout to an individual test case and overrides the possible default value.

Regardless of where the test timeout is defined, the first cell after the setting name contains the duration of the timeout. The duration must be given in Robot Framework's time format, that is, either directly in seconds or in a format like 1 minute 30 seconds. It must be noted that there is always some overhead by the framework, and timeouts shorter than one second are thus not recommended.

The default error message displayed when a test timeout occurs is Test timeout <time> exceeded. It is also possible to use custom error messages, and these messages are written into the cells after the timeout duration. The message can be split into multiple cells, similarly as documentations. Both the timeout value and the error message may contain variables.

If there is a timeout, the keyword running is stopped at the expiration of the timeout and the test case fails. However, keywords executed as test teardown are not interrupted if a test timeout occurs, because they are normally engaged in important clean-up activities. If necessary, it is possible to interrupt also these keywords with user keyword timeouts.

User keyword timeout

A timeout can be set for a user keyword using the [Timeout] setting in the Keyword table. The syntax for setting it, including how timeout values and possible custom messages are given, is identical to the syntax used with test case timeouts. If no custom message is provided, the default error message Keyword timeout <time> exceeded is used if a timeout occurs.

A user keyword timeout is applicable during the execution of that user keyword. If the total time of the whole keyword is longer than the timeout value, the currently executed keyword is stopped. User keyword timeouts are applicable also during a test case teardown, whereas test timeouts are not.

If both the test case and some of its keywords (or several nested keywords) have a timeout, the active timeout is the one with the least time left.

Normal for loop

In a normal For loop, one variable is assigned into a list of values, one value per iteration. The syntax starts with :FOR, where colon is required to separate the syntax from normal keywords. The next cell contains the loop variable, the subsequent cell must have IN, and the final cells contain values over which to iterate.

The keywords used in the For loop are on the next rows and they must be indented one cell to the right. The For loop ends when the indentation returns back to normal or the table ends. Having nested For loops directly is not supported, but it is possible to use a user keyword inside a For loop and have another For loop there.

The For loop in Example 1 above is executed twice, so that first the loop variable ${animal} has the value cat and then dog. The loop consists of two Log keywords. In the second example, loop values are split into several rows and the loop is run altogether seven times.

For loops are most useful and also clearest when they are used with list variables. This is illustrated by the example below, where @{ELEMENTS} contains an arbitrary long list of element names and Start Element is used with all of them.

Using several loop variables

It is also possible to use several loop variables. The syntax is the same as with the normal For loop, but all loop variables are listed in the cells between :FOR and IN. There can be any number of loop variables, but their number must match the number of loop values. The number of values must be evenly dividable by the number of variables.

This syntax naturally works both with and without list variables. In the former case, it is often possible to organize loop values below loop variables, as in the first part of the example below:

For in range

Earlier For loops always iterated over a sequence, and this is also the most common use case. Sometimes it is still convenient to have a For loop that is executed a certain number of times, and Robot Framework has a special FOR index IN RANGE limit syntax for this purpose. This syntax is derived from the similar Python idiom.

Similarly as other For loops, the For in range loop starts with :FOR and the loop variable is in the next cell. In this format there can be only one loop variable and it contains the current loop index. The next cell must contain IN RANGE and the subsequent cells loop limits.

In the simplest case, only the upper limit of the loop is specified. In this case, loop indexes start from zero and increase by one until, but excluding, the limit. It is also possible to give both the start and end limits. Then indexes start from the start limit, but increase similarly as in the simple case. Finally, it is possible to give also the step value that specifies the increment to use. If the step is negative, it is used as decrement. All these possibilities are illustrated by the examples below.

Repeating single keyword

For loops can be an overkill in situations where there is only a need to repeat a single keyword. In these cases it is often easier to use BuiltIn keyword Repeat Keyword which was added in Robot Framework 2.0.4. This keyword takes a keyword and how many times to repeat it as arguments. The times to repeat the keyword can have an optional postfix times or x to make the syntax easier to read.

Robot Framework also has an older special syntax for repeating a single keyword. Starting from Robot Framework 2.0.4 this syntax has been deprecated in favor of Repeat Keyword, though, and it will be removed altogether in 2.2 version. The example below shows how easy it is to convert test cases using the old syntax to use Repeat Keyword instead.

Different runner scripts

Test execution is normally started with the pybot or jybot commands. These commands are otherwise identical, but the former executes tests using the Python interpreter and the latter uses Jython. Which one to use depends on the needed test libraries. Some libraries use modules or syntax available only on Python, others use Java-based tools that require Jython, and some work on both. If you can use either pybot or jybot, the former is recommended, as Python is somewhat faster than Jython.

Another possibility for starting the test execution is running the runner.py script under the installed robot module directly. This method allows selecting the interpreter and setting command line options to it freely. The most common use case is altering the options controlling JVM maximum memory consumption.

Regardless of the runner script, a path (or paths) to the test data to be executed is given as an argument. Additionally, different command line options can be used to alter the test execution or generated outputs in some way.

Specifying test data to be executed

Robot Framework test cases are created in files and directories, and they are executed by giving the path to the file or directory in question to the selected runner script. The path can be absolute or, more commonly, relative to the directory where tests are executed from. The given file or directory creates the top-level test suite, which gets its name, unless overridden with the --name option, from the file or directory name. Different execution possibilities are illustrated in the examples below. Note that in these examples, as well as in other examples in this section, only the pybot command is used, but jybot or a custom runner script could be used similarly.

pybot test_cases.html
pybot path/to/my_tests/
pybot /opt/robot/tests.html
pybot c:\robot\tests.html

It is also possible to give paths to several test case files or directories at once, separated with spaces. In this case, Robot Framework creates the top-level test suite automatically, and the specified files and directories become its child test suites. The name of the created test suite is got from child suite names by catenating them together with an ampersand (&) and spaces. For example, the name of the top-level suite in the first example below is My Tests & Your Tests. These automatically created names are seldom very good and they are often quite long. In most cases, it is thus better to use the --name option for overriding it, as in the second example below:

pybot my_tests.html your_tests.html
pybot --name Example path/to/tests/pattern_*.html

Test execution order

Test cases in a test suite are executed in the order they are specified in the test case file where they are created. Test suites inside a higher level test suite are executed in case-insensitive alphabetical order based on the file or directory name. Finally, if multiple files and/or directories are given from the command line, they are executed in the order they are specified.

If there is a need to use certain test suite execution order inside a directory, it is possible to add prefixes like 01 and 02 into file and directory names. Starting from Robot Framework 2.1, such prefixes are automatically removed if they are separated from the base name with two underscores like in the examples below. These prefixes will not be visible anywhere in reports or logs:

01__my_suite.html -> My Suite
02__another_suite.html -> Another Suite

If the alphabetical ordering of test suites inside suites is problematic, a good workaround is giving them separately in the required order. This easily leads to overly long start-up commands, but argument files allow listing files nicely one file per line. It is also possible to randomize the execution order of test cases or test suites using --runmode option.

Using command line options

Robot Framework provides a number of command line options that can be used to control how test cases are executed and what outputs are generated. The syntax for using them is explained in this section. What options actually exist and how they can be used is discussed elsewhere in this chapter.

Generated outputs

The most visible output from test execution is the output displayed in the command line. All executed test suites and test cases, as well as their statuses, are shown there in real time. The example below shows the output from executing a simple test suite with only two test cases:

==============================================================================
Example test suite
==============================================================================
First test :: Possible documentation is here                          | PASS |
------------------------------------------------------------------------------
Second test                                                           | FAIL |
Error message is displayed here
==============================================================================
Example test suite                                                    | FAIL |
2 critical tests, 1 passed, 1 failed
2 tests total, 1 passed, 1 failed
==============================================================================
Output:  /path/to/output.xml
Report:  /path/to/report.html
Log:     /path/to/log.html

The command line output is very limited, and separate output files are normally needed for investigating the test execution. As the example above shows, three output files are generated by default. The first one is in the XML format and contains all the information about test execution. The second is a higher-level report and the third is a more detailed log file. These files and other possible output files are discussed in more detail in the section Different output files.

Return codes

Runner scripts communicate the overall test execution status to the system running them using return codes. The basic rule is that the return code is zero, which is a typical return code for success, when the execution starts successfully and no critical test fail. Possible return codes are explained in the table below.

Return codes should always be easily available after the execution, which makes it easy to automatically determine the overall execution status. For example, in bash shell the return code is in special variable $?, and in Windows it is in %ERRORLEVEL% variable. If you use some external tool for running tests, consult its documentation for how to get the return code.

Errors and warnings during execution

During the test execution there can be unexpected problems like failing to import a library or a resource file or a keyword being deprecated. Depending on the severity such problems are categorized as errors or warnings and they are written into the console (using the standard error stream), shown on a separate Test Execution Errors section in log files, and also written into Robot Framework's own system log. Normally these errors are generated by Robot Framework core, but libraries can use log level WARN to write warnings. Example below illustrates how errors and warnings look like in the log file.

Creating different reports and logs

You can use rebot for creating the same reports and logs that are created automatically during the test execution. Of course, it is not sensible to create the exactly same files, but, for example, having one report with all test cases and another with only some subset of tests can be useful. Another common usage is creating only the output file when running tests (log and report generation can be disabled with options --log NONE and --report NONE) and generating logs and reports later. Tests can, for example, be executed on different environments, output files collected to a central place, and reports and logs created there. If generating reports and logs takes a lot of time when running tests on Jython, it is a good idea to try if using rebot, which always runs on Python, is any faster.

rebot output.xml
rebot path/to/output_file.xml
rebot --include smoke --name Smoke_Tests c:\results\output.xml

Combining outputs

The most important feature of rebot is its ability to combine outputs from different test execution rounds. This capability allows, for example, running the same test cases on different environments and generating an overall report from all outputs. Combining outputs is extremely easy, all that needs to be done is giving several output files as arguments:

rebot output1.xml output2.xml
rebot outputs/*.xml

When outputs are combined, a new top-level test suite is created so that test suites in the given output files are its child suites. This works the same way when multiple test data files or directories are executed, and also in this case the name of the top-level test suite is created by joining child suite names with an ampersand (&) and spaces. These automatically generated names are not that good, and it is often a good idea to use --name to give a more meaningful name:

rebot --name Browser_Compatibility firefox.xml opera.xml safari.xml ie.xml
rebot --include smoke --name Smoke_Tests c:\results\*.xml

Modifying Java startup parameters

Sometimes there is need to alter the Java startup parameters. The most common use case is increasing the JVM maximum memory size as the default value may not be enough for creating reports and logs when outputs are very big. How to give the parameters to Java depends on the Jython version.

With Jython 2.2 you need to modify the Jython start-up script (jython shell script or jython.bat batch file) by adding the needed options (e.g. -Xmx1024m which sets the maximum memory to 1024 megabytes) to the java command. On Windows the final command could look like this:

"C:\Java\jre1.6.0\bin\java.exe" -Xmx1024m -Dpython.home="C:\Jython22" -classpath "C:\Jython22\jython.jar;%CLASSPATH%" org.python.util.jython %ARGS%

With Jython 2.5 it is also possible, and easier, to give the Java parameters with -J option to the Jython interpreter. This means that you can modify the jybot start-up script instead of Jython scripts. The last line of the jybot script can, again in Windows, be edited like below:

%jython% -J-Xmx1024m %runner% %*

By test suite and test case names

Test suites and test cases can be selected by their names with the command line options --suite (-s) and --test (-t), respectively. Both of these options can be used several times to select several test suites or cases. Arguments to these options are case- and space-insensitive, and there can also be simple patterns matching multiple names. If both the --suite and --test options are used, only test cases in matching suites with matching names are selected.

--test Example
--test mytest --test yourtest
--test example*
--suite example-??
--suite mysuite --test mytest --test your*

Using the --suite option is more or less the same as executing only the appropriate test case file or directory. One major benefit is the possibility to select the suite based on its parent suite. The syntax for this is specifying both the parent and child suite names separated with a dot. In this case, the possible setup and teardown of the parent suite are executed.

--suite parent.child
--suite myhouse.myhousemusic --test jack*

Selecting individual test cases with the --test option is very practical when creating test cases, but quite limited when running tests automatically. The --suite option can be useful in that case, but in general, selecting test cases by tag names is more flexible.

By tag names

It is possible to include and exclude test cases by tag names with the --include (-i) and --exclude (-e) options, respectively. When the former is used, only test cases having a matching tag are selected, and with the latter, test cases having a matching tag are not. If both are used, only tests with a tag matching the former option, and not with a tag matching the latter, are selected.

--include example
--exclude not_ready
--include regression --exclude long_lasting

Both --include and --exclude can be used several times to match multiple tags, and their arguments can be simple patterns. In these cases, the rules for selecting test cases apply, so that test cases with a tag matching any include patterns are selected, and tests with a tag matching exclude patterns are not. It is also possible to select only test cases that have two or more specified tags by separating the tags either with & or AND (case-sensitive). Similarly, only tests with a certain tag, but without some others, can be selected by separating these tags with NOT (case-sensitive).

--include req-*
--include regressionANDiter-42
--include tag1&tag2&tag3&tag4
--exclude regressionNOTowner-*

Selecting test cases by tags is a very flexible mechanism and allows many interesting possibilities:

• A subset of tests to be executed before other tests can be tagged with smoke and executed with --include smoke.
• Unfinished test can be committed to version control with the tag not_ready and excluded from the test execution with --exclude not_ready.
• Tests can be tagged with iter-<num>, where <num> specifies the number of the current iteration, and after executing all test cases, a separate report containing only the tests for a certain iteration can be generated (for example, rebot --include iter-42 ouput.xml).

Setting the name

When Robot Framework parses test data, test suite names are created from file and directory names. The name of the top-level test suite can, however, be overridden with the command line option --name (-N). Underscores in the given name are converted to spaces automatically, and words in the name capitalized.

Setting the documentation

In addition to defining documentation in the test data, documentation of the top-level suite can be given from the command line with the option --doc (-D). Underscores in the given documentation are converted to spaces, and it may contain simple HTML formatting.

Setting free metadata

Free test suite metadata may also be given from the command line with the option --metadata (-M). The argument must be in the format name:value, where name the name of the metadata to set and value is its value. Underscores in the former are converted to spaces and words capitalized, and the latter may contain simple HTML formatting. This option may be used several times to set multiple metadata.

Setting tags

The command line option --settag (-G) can be used to set the given tag to all executed test cases. This option may be used several times to set multiple tags.

Locations automatically in PYTHONPATH

Python and Jython installations put their own library directories into PYTHONPATH automatically. This means that test libraries packaged using Python's own packaging system are automatically installed into a location that is in the library search path. Robot Framework also puts the directory containing its standard libraries and the directory where tests are executed from into PYTHONPATH.

Setting PYTHONPATH in system

There are several ways to alter PYTHONPATH in the system, but the most common one is setting an environment variable with the same name before the test execution. Jython actually does not use PYTHONPATH environment variable normally, but Robot Framework ensures that locations listed in it are added into the library search path regardless the interpreter.

Setting CLASSPATH in the system

CLASSPATH is used only with Jython, and the most common way to alter it is setting an environment variable similarly as with PYTHONPATH. Note that instead of CLASSPATH, it is always possible to use PYTHONPATH with Jython, even with libraries and listeners implemented with Java.

Using the --pythonpath option

Robot Framework also has a separate command line option --pythonpath (-P) for adding directories or archives into PYTHONPATH. Multiple paths can be given by separating them with a colon (:) or using this option several times. The given path can also be a glob pattern matching multiple paths, but then it normally must be escaped.

Examples:

--pythonpath libs/
--pythonpath /opt/testlibs:mylibs.zip:yourlibs
--pythonpath mylib.jar --pythonpath lib/STAR.jar --escape star:STAR

Output directory

All output files can be set using an absolute path, in which case they are created to the specified place, but in other cases, the path is considered relative to the output directory. The default output directory is the directory where the execution is started from, but it can be altered with the --outputdir (-d) option. The path set with this option is, again, relative to the execution directory, but can naturally be given also as an absolute path. Regardless of how a path to an individual output file is obtained, its parent directory is created automatically, if it does not exist already.

Output file

Output files contain all the test execution results in the XML format. Log, report, and summary files are generated based on output files, and output files can also be combined and otherwise post-processed after the test execution.

The command line option --output (-o) determines where output files are created. Output files are always created when tests are executed, and the default name for them is output.xml. When post-processing outputs, new output files are not created unless this option is explicitly used.

Log file

Log files contain details about the executed test cases in HTML format. They have a hierarchical structure showing test suite, test case and keyword details. Log files are needed nearly every time when test results are to be investigated in detail. Even though log files also have statistics, report and summary files are better for getting an higher-level overview.

The command line option --log (-l) determines where log files are created. Unless the special value NONE is used, log files are always created and their default name is log.html.

An example of beginning of log file

An example of log file with keyword details visible

Report file

Report files contain an overview of the test execution results in HTML format. They have statistics based on tags and executed test suites, as well as a list of all executed test cases. When both reports and logs are generated, the report has links to the log file for easy navigation to more detailed information. It is easy to see the overall test execution status from reports, because their background color is green, if all critical tests pass, and bright red otherwise.

The command line option --report (-r) determines where report files are created. Similarly as log files, reports are always created unless NONE is used as a value, and their default name is report.html.

An example report file of successful test execution

An example report file of failed test execution

Summary file

Summary files contain the same statistics as reports, and their background color is similarly green or red, depending on the overall test execution status. However, they have no test case details, which makes them much smaller. They are useful when a large number of test cases is executed and a very high-level overview is needed.

Summary files are not generated by default. When they are needed, they can be created using the command line option --summary (-S).

An example summary file

Debug file

Debug files are plain text files that are written during the test execution. All messages got from test libraries are written to them, as well as information about started and ended test suites, test cases and keywords. Debug files can be used for monitoring the test execution. This can be done using, for example, a separate file viewing tool, or in UNIX-like systems, simply with the tail -f command.

Debug files are not created unless the command line option --debugfile (-b) is used explicitly.

Timestamping output files

All output files listed in this section can be automatically timestamped with the option --timestampoutputs (-T), which is one of the rare options taking no value. When this option is used, a timestamp in the format YYYYMMDD-hhmmss is placed between the extension and the basename of each file. The example below would, for example, create such output files as output-20080604-163225.xml and mylog-20080604-163225.html.

pybot --timestampoutputs --log mylog.html --report NONE tests.html

Setting titles

The default titles for log, report and summary files are generated by prefixing the name of the top-level test suite wtih Test Log, Test Report or Summary Report. Custom titles can be given from the command line using the options --logtitle, --reporttitle and --summarytitle, respectively. With all these options, underscores in the given title are converted to spaces automatically.

Examples:

pybot --logtitle Smoke_Test_Log --reporttitle Smoke_Test_Report --include smoke mytests
rebot --summarytitle Overview --summary overview.html --log none --report none *.xml

Available log levels

Messages in log files can have different log levels. Some of the messages are written by Robot Framework itself, but also executed keywords can log information using different levels. The available log levels are:

FAIL

Used when a keyword fails. Can be used only by Robot Framework.

WARN

Used to display warnings. They shown also in the console and in the Test Execution Errors section in log files, but they do not affect the test case status.

INFO

The default level for normal messages. By default, messages below this level are not shown in the log file.

DEBUG

Used for debugging purposes. Useful for example for logging what libraries are doing internally.

TRACE

Another debugging level. Robot Framework itself logs the actual values of keyword arguments and the return values using this level.

Setting log level

By default, log messages below INFO are not logged, but this threshold level can be changed from the command line using the --loglevel (-L) option. This option takes any of the available log levels as an argument, and that level becomes the new threshold level. A special value NONE can also be used to disable logging altogether.

Another possibility to change the log level is using the BuiltIn keyword Set Log Level in the test data. It takes the same arguments as the --loglevel option, and it also returns the old level so that it can be restored later, for example, in a test teardown.

Benefits

When executing a large number of test cases, the size of log files can increase to the extent that opening them into browsers is slow. Additionally, since log files can be created only after output files are ready, they are available only after the test execution. This is not a problem if the test execution time is short, but with long-running tests it is better to be able to investigate the first failures while the rest of the tests are still running.

Splitting outputs provides a solution for both of these problems. First of all, splitting logs means that individual log files are smaller and thus faster to open. If outputs are split while executing test cases, the lower-level log files are also created immediateay when the equivelent output files are ready. Notifications about created files can be received through the listener interface, and automatic variables ${LOG_FILE} and ${OUTPUT_FILE} always contain the path to the current file.

Specifying a split level

Outputs are split from a certain test suite level. Suites below this level will get their own outputs, and everything above the level will be in an index file, which has links to lower-level files. Only output files and log files are split, and they are always split from the same level. Splitting outputs gives the best results when test cases are organized into a relatively balanced hierarchical structure.

Splitting can be activated with the command line option --splitoutputs, which takes the split level as an argument. Index files have the same name as the output and log would have without splitting, and lower-level files get a running counter between the basename and extension.

Examples

Explaining how splitting outputs actually works is easiest with examples, and it is also a good idea to experiment with real test data. In these examples we assume that test cases are organized into test suites hierarchically in the following way:

project
|-- component_a
|   |-- feature_a1
|   |   |-- a11.html
|   |   `-- a12.html
|   `-- feature_a2
|       `-- a21.html
|-- component_b
|   `-- feature_b1
|       |-- b11.html
|       `-- b21.html
`-- component_c
    |-- c1.html
    |-- c2.html
    `-- c3.html

In the first example these test cases are executed so that they are split right below the top-level test suite. The command line to use is as follows, and the following table lists all the created output files:

pybot --splitoutputs 1 project

The fact that output files are split is transparent when post-processing outputs afterwards, and it is enough to point rebot to the index file. Splitting works with rebot regardless of whether outputs are split earlier or not. Outputs created by the first example could thus by re-split, for example, as:

rebot --splitoutputs 2 output.xml

In this example, splitting is done from the second level. The table below again lists created the files, and because rebot does not create new output files by default, they are not listed in the table either.

The last example splits outputs from the third level. It also shows how to configure output file names when splitting:

rebot --splitoutputs 3 --log mylog.html --report none output.xml

If a split level higher than three is used with this test data, all the information ends up to index files. The end result is thus exactly the same as if outputs were not split at all.

Configuring displayed suite statistics

When a deeper suite structure is executed, showing all the test suite levels in the Statistics by Suite table may make the table somewhat difficult to read. You can control the number of levels displayed with the command line option --suitestatlevel. It takes the required level as an integer, and if 0 is used, the whole table is removed.

Including and excluding tag statistics

When many tags are used, the Statistics by Tag table can become quite congested. If this happens, the command line options --tagstatinclude and --tagstatexclude can be used to select which tags to display, similarly as --include and --exclude are used to select test cases:

--tagstatinclude some-tag --tagstatinclude another-tag
--tagstatexclude owner-*
--tagstatinclude prefix-* --tagstatexclude prefix-13

These settings affect also the Test Details by Tag table, so that it has details only by the selected tags. This can make the report considerably smaller, which is why excluding tags that are not interesting can be recommended.

Generating combined tag statistics

The command line option --tagstatcombine can be used to generate aggregate tags that combine statistics from multiple tags. These new combined tags are shown in the Statistics by Tag table, and the matching tests are listed in the Test Details by Tag table. There are three somewhat different ways for giving arguments for this option:

One tag as a simple pattern

All tags matching the given pattern are combined together.

Two or more tags separated by AND or &

The combined statistics contain tests that have all the listed tags. Tags can be given as simple patterns.

Two or more tags separated by NOT

The combined statistics contain tests that have the first tag but not the others. Also in this case tags may be patterns.

The following examples illustrate these usages, and the figure below shows a snippet of the resulting Statistics by Tag table when the example test data is executed with these options:

--tagstatcombine owner-*
--tagstatcombine smokeANDmytag
--tagstatcombine smokeNOTowner-janne*

Examples of combined tag statistics

As the above example shows, the name of the added combined statistic is, by default, generated from the given pattern. In certain situations this name can look pretty cryptic and, starting from Robot Framework 2.0.2, it is possible to specify a more descriptive name. This name is given after the pattern separating it with a colon (:). Example below generates combined tag so that the name shown in reports and logs is Critical Tests:

--tagstatcombine *NOTnon-critical:Critical_Tests

Creating links from tag names

You can add external links to the Statistics by Tag table by using the command line option --tagstatlink. Arguments to this option are given in the format tag:link:name, where tag specifies the tags to assign the link to, link is the link to be created, and name is the name to give to the link.

tag may be a single tag, but more commonly a simple pattern where * matches anything and ? matches any single character. When tag is a pattern, the matches to wildcards may be used in link with the syntax %N, where "N" is the index of the match starting from 1.

The following examples illustrate the usage of this option, and the figure below shows a snippet of the resulting Statistics by Tag table when example test data is executed with these options:

--tagstatlink mytag:http://www.google.com:Google
--tagstatlink jython-bug-*:http://bugs.jython.org/issue_%1:Jython-bugs
--tagstatlink owner-*:mailto:%1@domain.com?subject=Acceptance_Tests:Send_Mail

Examples of links from tag names

Adding documentation to tags

Tags can be given a documentation with the command line option --tagdoc, which takes an argument in the format tag:doc. "tag" is the name of the tag to assign the documentation to, and it can also be a simple pattern matching multiple tags. "doc" is the assigned documentation. Underscores in it are converted to spaces automatically, and it can also contain HTML formatting. The given documentation is shown with matching tags in the Test Details by Tag table, and as a tool tip for these tags in the Statistics by Tag table.

Examples:

--tagdoc mytag:My_documentation
--tagdoc regression:*See*_http://info.html
--tagdoc owner-*:Original_author

Setting times for combined outputs

When combining outputs, it is possible to set the start and end time of the combined test suite using the options --starttime and --endtime, respectively. This is convenient, because by default, combined suites do not have these values. When both the start and end time are given, the elapsed time is also calculated based on them. Otherwise the elapsed time is got by adding the elapsed times of the child test suites together.

Times must be given as timestamps in the format YYYY-MM-DD hh:mm:ss.mil, where all separators are optional and the parts from milliseconds to hours can be omitted. For example, 2008-06-11 17:59:20.495 is equivalent both to 20080611-175920.495 and 20080611175920495, and also mere 20080611 would work.

Examples:

rebot --starttime 20080611-17:59:20.495 output1.xml output2.xml
rebot --starttime 20080611-175920 --endtime 20080611-180242 *.xml

Removing keywords from outputs

Most of the content of output files comes from keywords and especially their log messages. When creating higher level reports, log files are not necessarily needed at all, and then keywords and their messages just take space unnecessarily. In these situations, the command line option --removekeywords can be used to dispose of unnecessary keywords. It has two possible values:

ALL

All keywords are unconditionally removed.

PASSED

Keywords are removed only from the passed test cases. In most cases, log files created after this contain enough information to investigate possible failures.

Removing keywords makes output files considerably smaller and thus faster to process further. Even when keywords are removed, names, arguments and statuses of top-level keywords are preserved, so it is still possible to create log files and see the high-level structure of each test case.

Supported programming languages

Robot Framework itself is written with Python and naturally test libraries extending it can be implemented using the same language. When running the framework on Jython, libraries can also be implemented using Java. Pure Python code works both on Python and Jython, assuming that it does not use syntax or modules that are not available on Jython. When using Python, it is also possible to implement libraries with C using Python C API, although it is often easier to interact with C code from Python libraries using ctypes module.

Libraries implemented using these natively supported languages can also act as wrappers to functionality implemented using other programming languages. A good example of this approach is the Remote library, and another widely used approaches is running external scripts or tools as separate processes.

Different test library APIs

Robot Framework has three different test library APIs.

Static API

The simplest approach is having a module (in Python) or a class (in Python or Java) with methods the names of which map more or less directly to keyword names. Keywords also take the same arguments as the code implementing them. Keywords report failures with exceptions, log by writing to standard output and can return values using the return statement.

Dynamic API

Dynamic libraries are classes that implement a method to get the names of the keywords they implement, and another method to execute a named keyword with given arguments. The names of the keywords to implement, as well as how they are executed, can be determined dynamically at runtime, but reporting the status, logging and returning values is done similarly as in the static API.

Hybrid API

This is a hybrid between the static and the dynamic API. Libraries are classes with a method telling what keywords they implement, but those keywords must be available directly. Everything else except discovering what keywords are implemented is similar as in the static API.

All these APIs are described in this chapter. Everything is based on how the static API works, so its functions are discussed first. How the dynamic library API and the hybrid library API differ from it is then discussed in sections of their own.

The examples in this chapter are mainly about using Python, but they should be easy to understand also for Java-only developers. In those few cases where APIs have differences, both usages are explained with adequate examples.

Test library names

The name of a test library that is used when a library is imported is the same as the name of the module or class implementing it. For example, if you have a Python module MyLibrary (that is, the file MyLibrary.py), it will create a library with a name MyLibrary. Similarly, a Java class YourLibrary, when it is not in any package, creates a library with exactly that name.

Python classes are always inside a module. If the name of a class implementing a library is the same as the name of the module, Robot Framework allows dropping the module name when importing the library. For example, the class MyLib in the MyLib.py file can be used as a library with the name MyLib. If the module name and class name are different, libraries must be taken into use using both module and class names, such as mymodule.MyLibrary.

Java classes in a non-default package must be taken into use with the full name. For example, the class MyLib in the com.mycompany.myproject package must be imported with the name com.mycompany.myproject.MyLib.

Providing arguments to test libraries

All test libraries implemented as classes can take arguments. These arguments are specified in the Setting table after the library name, and when Robot Framework creates an instance of the imported library, it passes them to its constructor. Libraries implemented as a module cannot take any arguments, so trying to use those results in an error.

The number of arguments needed by the library is the same as the number of arguments accepted by the library's constructor. The default values and variable number of arguments work similarly as with keyword arguments, with the exception that there is no variable argument support for Java libraries. Arguments passed to the library, as well as the library name itself, can be specified using variables, so it is possible to alter them, for example, from the command line.

Example implementations, first one in Python and second in Java, for the libraries used in the above example:

from example import Connection

class MyLibrary:

    def __init__(self, host, port=80):
        self._conn = Connection(host, int(port))

    def send_message(self, message):
        self._conn.send(message)

public class AnotherLib {

    private String setting = null;

    public AnotherLib(String setting) {
        setting = setting;
    }

    public void doSomething() {
        if setting.equals("42") {
            // do something ...
        }
    }
}

Test library scope

Libraries implemented as classes can have an internal state, which can be altered by keywords and with arguments to the constructor of the library. Because the state can affect how keywords actually behave, it is important to make sure that changes in one test case do not accidentally affect other test cases. This kind of dependencies may create hard-to-debug problems, for example, when new test cases are added and they use the library inconsistently.

Robot Framework attempts to keep test cases independent from each other: by default, it creates new instances of test libraries for every test case. However, this behavior is not always desirable, because sometimes test cases should be able to share a common state. Additionally, all libraries do not have a state and creating new instances of them is simply not needed.

Test libraries can control when new libraries are created with a class attribute ROBOT_LIBRARY_SCOPE . This attribute must be a string and it can have the following three values:

TEST CASE

A new instance is created for every test case. A possible suite setup and suite teardown share yet another instance. This is the default.

TEST SUITE

A new instance is created for every test suite. The lowest-level test suites, created from test case files and containing test cases, have instances of their own, and higher-level suites all get their own instances for their possible setups and teardowns.

GLOBAL

Only one instance is created during the whole test execution and it is shared by all test cases and test suites. Libraries created from modules are always global.

When the TEST SUITE or GLOBAL scopes are used with test libraries that have a state, it is recommended that libraries have some special keyword for cleaning up the state. This keyword can then be used, for example, in a suite setup or teardown to ensure that test cases in the next test suites can start from a known state. For example, SeleniumLibrary uses the GLOBAL scope to enable using the same browser in different test cases without having to reopen it, and it also has the Close All Browsers keyword for easily closing all open browsers.

Example Python library using the TEST SUITE scope:

class ExampleLibrary:

    ROBOT_LIBRARY_SCOPE = 'TEST SUITE'

    def __init__(self):
        self._counter = 0

    def count(self):
        self._counter += 1
        print self._counter

    def clear_counter(self):
        self._counter = 0

Example Java library using the GLOBAL scope:

public class ExampleLibrary {

    public static final String ROBOT_LIBRARY_SCOPE = "GLOBAL";

    private int counter = 0;

    public void count() {
        counter += 1;
        System.out.println(counter);
    }

    public void clearCounter() {
        counter = 0;
    }
}

Specifying library version

When a test library is taken into use, Robot Framework tries to determine its version. This information is then written into the syslog to provide debugging information. Library documentation tool libdoc.py also writes this information into the keyword documentations it generates.

Version information is read from attribute ROBOT_LIBRARY_VERSION, similarly as test library scope is read from ROBOT_LIBRARY_SCOPE. If ROBOT_LIBRARY_VERSION does not exist, information is tried to be read from __version__ attribute. These attributes must be a class or module attributes, depending whether the library is implemented as a class or a module. For Java libraries, the version attribute must be declared as static final.

An example Python module using __version__:

__version__ = '0.1'

def keyword():
    pass

A Java class using ROBOT_LIBRARY_VERSION:

public class VersionExample {

    public static final String ROBOT_LIBRARY_VERSION = "1.0.2";

    public void keyword() {
    }
}

Keyword names

When the static library API is used, Robot Framework uses reflection to find out what methods the library implements. With dynamic library API and hybrid library API, keyword names are got from the library directly. Naturally, Robot Framework can see only the public methods and it also excludes all methods starting with an underscore. With Java libraries, also methods that are implemented in java.lang.Object and not overridden are ignored.

Keyword names used in the test data are compared with method names to find the method implementing these keywords. Name comparison is case-insensitive, and also spaces and underscores are ignored. For example, the method hello maps to the keyword name Hello, hello or even h e l l o. Similarly both the do_nothing and doNothing methods can be used as the Do Nothing keyword in the test data.

Example Python library implemented as a module in the MyLibrary.py file:

def hello(name):
    print "Hello, %s!" % name

def do_nothing():
    pass