2.3 Executing Robot
Recall the three components of the Robot Framework CLI.
Robot Framework comes with three executables when being installed which are designed to be used via the command-line interface (CLI).
robotis the main executable that is used to execute suites.rebot(not part of this syllabus) is used to post-process execution results and generate reports.libdoc(not part of this syllabus) is used to generate keyword documentation for libraries and resource files. See 2.5 Keyword Interface and Documentation
2.3.1 robot command & help
Understand how to run the robot command and its basic usage.
The robot command is used to run a Robot Framework execution, which will execute suites and their containing tests|tasks.
At a basic level, you can run robot by providing the path to a suite file or suite directory containing suite files as last argument.
robot <path_to_root_suite>
In case of the above given example where a single suite file named TestSuite.robot is stored in a directory robot_files, to execute the example test suite the following command is used, if the current working directory of the terminal is the directory containing the robot_files directory:
> robot robot_files
This command starts the Robot Framework execution by first parsing all files in the given directory tree that have the extension .robot,
then creating an execution model and then executing all suites and test cases in that model.
During execution, the results of each test case are printed to the console and at the end a summary is printed and reports are generated.
Example Console Output:
> robot robot_files
==============================================================================
Robot Files
==============================================================================
Robot Files.TestSuite :: A test suite for valid login.
==============================================================================
Login User With Password | PASS |
------------------------------------------------------------------------------
Denied Login With Wrong Password | PASS |
------------------------------------------------------------------------------
Robot Files.TestSuite :: A test suite for valid login. | PASS |
2 tests, 2 passed, 0 failed
==============================================================================
Robot Files | PASS |
2 tests, 2 passed, 0 failed
==============================================================================
Output: /path/to/output.xml
Log: /path/to/log.html
Report: /path/to/report.html
The robot command can optionally be configured with additional options to control the execution behavior, such as setting output formats, specifying specific tests to run, or controlling logging levels and many more. These options are named arguments that are passed to the robot command BEFORE the path to the suite file or directory. To learn more about these options, you can use the help of the robot command like: robot --help.
2.3.2 Execution Artifacts
Explain the execution artifacts generated by Robot Framework.
After executing a suite, Robot Framework, by default, generates three output files in the output directory. These artifacts provide detailed execution results:
output.xml: A machine-readable file containing ALL logged execution details, limited by the given log-level.log.html: A detailed log file that provides an HTML view of the execution, including keyword-level details.report.html: A summary report that gives an overview of the execution results, including statistics of tests|tasks executed, passed, and failed.
log.html and report.html are generated based on the information stored in output.xml.
A unique feature of Robot Framework is, that it logs each keyword call and its arguments with its log outputs and timestamps, so that it is possible to have a very detailed view of the execution flow and the data that was used during the execution. In case of a failure it is possible to see the exact keyword call that failed and the arguments that were used, which can be very helpful for debugging or reporting. Furthermore you also get all passed keywords and even the non-executed keywords to protocol the whole execution flow.
2.3.3 Status
Recall the four different status labels used by Robot Framework.
Robot Framework uses different status labels to indicate the result of an execution:
On Suite, Test Case, Task and Keyword Level:
PASS: Indicates that the item was successfully executed without unexpected errors.FAIL: Shows that the item encountered an error and did not pass.SKIP: Indicates that the item was intentionally skipped, either by tagging or during execution, typically because some condition was not met.
Additional Keyword Status:
NOT RUN: Refers to keywords that were not executed during execution, i.e. due to previous failure or conditions.
SKIP is explained in more detail in later chapters.
Atomic elements like Library Keywords or Robot Framework language statements do define their own status.
Composite elements like suites (composed of tests|tasks), tests|tasks (composed of keywords) and User Keywords (composed of Library Keywords and Robot Framework statements) do define their status based on the status of their child elements.
2.3.3.1 PASS
Understand when an element is marked as PASS.
This status is used if an element was executed successfully without any errors or exceptions.
Atomic elements are PASS if they were executed successfully without reporting an error by raising an exception.
Composite elements are PASS if all their executed body elements are pass.
In example for User Keywords this means that if all keywords or Robot Framework language statements that were directly called by that User Keyword were PASS the User Keyword itself is considered PASS.
Library Keywords like Run Keyword And Expect Error, from BuiltIn Library, do PASS if the keyword they are internally calling does raise an error with the expected message or type.
That means that a composite element like suite, test|task or User Keyword may be PASS even if some of its deeper child elements are FAIL.
2.3.3.2 FAIL
Understand when an element is marked as FAIL.
This status is used if an element was executed but encountered an error or exception that was not expected.
A failure typically causes the subsequent keywords to be skipped. Exceptions are Teardowns explained in chapter 4 Advanced Structuring and Execution.
Atomic elements are FAIL if they were tried to be executed but raised an exception.
Composite elements are FAIL if at least one of their executed direct body elements are FAIL.
Therefore a failure typically distributes upwards through the hierarchy of elements until it reaches the root suite.
A User Keywords is FAIL if one of its called Library Keywords is FAIL.
A test|task is FAIL if one of its directly called Keywords is FAIL.
A suite (file) is FAIL if one of its test|task is FAIL and
a suite (directory) is FAIL if one of its suites (file) is FAIL.
2.3.4 Logging possibilities (Log vs Console)
Understand the difference between log messages and console output.
There are basically two kinds of logging information in Robot Framework.
- Console Output: The console output is the output that is printed to the terminal where the
robotcommand was executed. It is typically not persistent but can be already seen during execution. - Log Messages: Log messages are written to the
output.xmland therefore alsolog.htmlfile and are persistent. They are typically created by the Library Keywords that are executed and can be used to log information about the execution. Also Robot Framework itself does log information to theoutput.xmllike assigned values of arguments or the return values of keywords.
Log messages can be written with different levels of severity (i.e. INFO, DEBUG, TRACE, WARN or ERROR).
Which levels are written to the log can be controlled by the log level of an execution. Further information in later chapters.