The lifecycle of a test executed using Trade Federation is composed of four separate stages, designed around formally defined interfaces.
Defined interfaces
- Build Provider: Provides a build to test, downloading appropriate files if necessary.
- Target Preparer: Prepares the test environment, possibly including software installation and device configuration.
- Test: Executes test(s) and gathers test results. This may be any JUnit Test, although our IRemoteTest interface is specifically designed to work well in the Trade Federation environment.
- Test Invocation Listener (results reporting): Listens for test results, usually for the purpose of forwarding the test results to a repository or displaying them to the Test Runner.
The fundamental testing entity in TF is a Configuration (config). A config is an XML file that declares the lifecycle components of a test.
This separation of the test's lifecycle is intended to allow for reuse. Using this design, the Developer can create a Test once, and then the Integrator can create different Configurations to run that Test in different environments. For example, they could create a Configuration that will run a test on a local machine and dump the result to stdout. They could then create a second Configuration that would execute that same test, but use a different Test Invocation Listener to store the test results in a database. A third Configuration might be designed run that test continuously from a test lab somewhere.
It's convenient to note here that a Configuration along with its command-line arguments (as
provided by the Test Runner) is known as a Command. When TF pairs a Command with an
ITestDevice and executes it, the subsequent object is known as an Invocation.
In short, an Invocation encompasses a complete TF test execution, across its entire lifecycle.
Additional configuration components
- Device Recovery: mechanism to recover device communication if lost.
- Logger: collects tradefed logging data.
Stage output and errors
Each stage of an invocation executes sequentially and has a specific goal. This section describes the usual outputs and errors of each stage.
Build provider
This stage creates and outputs an
IBuildInfo
object that contains all the required files references to set up and run the tests.
The most common error at this stage is a failure to download or find the requested files.
An error at this stage results in directly reporting the error, and no tests being run.
Target preparation
This stage sets up the necessary states for the target under tests. This stage can alter the device or the host setup as needed for the given test invocation.
Common errors at this stage usually involve failure to setup the device into a given state (for example, failed flashing) and failure to find the required files for the setup.
An error at this stage results in target clean up running, reporting of the error, and no tests being run.
Tests
This stage runs the requested tests on the previously prepared target, and reports all the test execution results.
Common errors at this stage usually involve the target under test being unavailable or some error causing partial execution of the tests. These errors are infrastructure issues that affect the test execution itself as opposed to a failure of a single test case.
An error at this stage results in the test execution stopping, the target clean up running, reporting the error, and getting partial results.
Results reporting
This stage reports the results and errors to the configured services (for example, servers and local files).
Although individual results reporters can have errors, they're isolated from each other (one reporter doesn't see errors from another one). These errors affect only an individual reporter's own results reporting and the errors can be viewed in the logs.