End-to-end TF test example

This tutorial guides you through creating a "hello world" Trade Federation (Tradefed or TF) test configuration and gives you a hands-on introduction to the TF framework. Starting from a development environment, you will create a simple configuration and add features.

The tutorial presents the test development process as a set of exercises, each consisting of several steps, that demonstrate how to build and gradually refine your configuration. All sample code you need to complete the test configuration is provided, and the title of each exercise is annotated with a letter describing the roles involved in that step:

  • D for Developer
  • I for Integrator
  • R for Test Runner

After completing the tutorial, you will have a functioning TF configuration and understand many important concepts in the TF framework.

Set up Trade Federation

For details on setting up the TF development environment, see Machine Setup. The rest of this tutorial assumes you have a shell open that has been initialized to the TF environment.

For simplicity, this tutorial illustrates adding a configuration and its classes to the TF framework core library. This can be extended to developing modules outside the source tree by compiling the tradefed JAR, then compiling your modules against that JAR.

Create a test class (D)

Lets create a hello world test that just dumps a message to stdout. A tradefed test generally implements the IRemoteTest interface. Here's an implementation for the HelloWorldTest:

package com.android.tradefed.example;

import com.android.tradefed.device.DeviceNotAvailableException;
import com.android.tradefed.invoker.TestInformation;
import com.android.tradefed.log.LogUtil.CLog;
import com.android.tradefed.result.ITestInvocationListener;
import com.android.tradefed.testtype.IRemoteTest;

public class HelloWorldTest implements IRemoteTest {
    @Override
    public void run(TestInformation testInfo, ITestInvocationListener listener) throws DeviceNotAvailableException {
        CLog.i("Hello, TF World!");
    }
}

Save this sample code to <tree>/tools/tradefederation/core/src/com/android/tradefed/example/HelloWorldTest.java and rebuild tradefed from your shell:

m -jN

Note that CLog.i in the example above is used to direct output to the console. More information on logging in Trade Federation is described in Logging (D, I, R).

If the build does not succeed, consult Machine Setup to ensure you didn't miss a step.

Create a configuration (I)

Trade Federation tests are made executable by creating a Configuration, an XML file that instructs tradefed on which test (or tests) to run, as well as which other modules to execute and in what order.

Lets create a new Configuration for our HelloWorldTest (note the full class name of the HelloWorldTest):

<configuration description="Runs the hello world test">
    <test class="com.android.tradefed.example.HelloWorldTest" />
</configuration>

Save this data to a helloworld.xml file anywhere on your local filesystem (e.g. /tmp/helloworld.xml). TF will parse the Configuration XML file (aka config), load the specified class using reflection, instantiate it, cast it to a IRemoteTest, and call its run method.

Run the config (R)

From your shell, launch the tradefed console:

tradefed.sh

Ensure a device is connected to the host machine and is visible to tradefed:

tf> list devices
Serial            State      Product   Variant   Build   Battery
004ad9880810a548  Available  mako      mako      JDQ39   100

Configurations can be executed using the run <config> console command. Try:

tf> run /tmp/helloworld.xml
05-12 13:19:36 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World!

You should see "Hello, TF World!" output on the terminal.

You can confirm that a command is done running by using list invocations or l i in the console prompt, and it should print nothing. If commands are currently running, they display as follows:

tf >l i
Command Id  Exec Time  Device       State
10          0m:00      [876X00GNG]  running stub on build(s) 'BuildInfo{bid=0, target=stub, serial=876X00GNG}'

Add the config to the classpath (D, I, R)

For convenience of deployment, you can also bundle configs into the tradefed JARs themselves. Tradefed automatically recognizes all configurations placed in config folders on the classpath.

To illustrate, move the helloworld.xml file into the tradefed core library (<tree>/tools/tradefederation/core/res/config/example/helloworld.xml). Rebuild tradefed, restart the tradefed console, then ask tradefed to display the list of configurations from the classpath:

tf> list configs
[…]
example/helloworld: Runs the hello world test

You can now run the helloworld config using:

tf> run example/helloworld
05-12 13:21:21 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World!

Interact with a device (D, R)

So far, our HelloWorldTest isn't doing anything interesting. Tradefed's specialty is running tests using Android devices, so lets add an Android device to the test.

Tests can get a reference to an Android device by using TestInformation, provided by the framework when the IRemoteTest#run method is called.

Let's modify the HelloWorldTest print message to display the serial number of the device:

@Override
public void run(TestInformation testInfo, ITestInvocationListener listener) throws DeviceNotAvailableException {
    CLog.i("Hello, TF World! I have device " + testInfo.getDevice().getSerialNumber());
}

Now rebuild tradefed and check the list of devices:

tradefed.sh

tf> list devices
Serial            State      Product   Variant   Build   Battery
004ad9880810a548  Available  mako      mako      JDQ39   100

Take note of the serial number listed as Available; that is the device that should be allocated to HelloWorld:

tf> run example/helloworld
05-12 13:26:18 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World! I have device 004ad9880810a548

You should see the new print message displaying the serial number of the device.

Send test results (D)

IRemoteTest reports results by calling methods on the ITestInvocationListener instance provided to the #run method. The TF framework itself is responsible for reporting the start (via ITestInvocationListener#invocationStarted) and end (via ITestInvocationListener#invocationEnded) of each Invocation.

A test run is a logical collection of tests. To report test results, IRemoteTest is responsible for reporting the start of a test run, the start and end of each test, and the end of the test run.

Here's what the HelloWorldTest implementation might look like with a single failed test result.

@Override
public void run(TestInformation testInfo, ITestInvocationListener listener) throws DeviceNotAvailableException {
    CLog.i("Hello, TF World! I have device " + testInfo.getDevice().getSerialNumber());

    TestDescription testId = new TestDescription("com.example.TestClassName", "sampleTest");
    listener.testRunStarted("helloworldrun", 1);
    listener.testStarted(testId);
    listener.testFailed(testId, "oh noes, test failed");
    listener.testEnded(testId, Collections.emptyMap());
    listener.testRunEnded(0, Collections.emptyMap());
}

TF includes several IRemoteTest implementations you can reuse instead of writing your own from scratch. For example, InstrumentationTest can run an Android application's tests remotely on an Android device, parse the results, and forward those results to the ITestInvocationListener). For details, see Test Types.

Store test results (I)

The default test listener implementation for a TF config is TextResultReporter, which dumps the results of an invocation to stdout. To illustrate, run the HelloWorldTest config from the previous section:

./tradefed.sh

tf> run example/helloworld
04-29 18:25:55 I/TestInvocation: Invocation was started with cmd: /tmp/helloworld.xml
04-29 18:25:55 I/TestInvocation: Starting invocation for 'stub' with '[ BuildInfo{bid=0, target=stub, serial=876X00GNG} on device '876X00GNG']
04-29 18:25:55 I/HelloWorldTest: Hello, TF World! I have device 876X00GNG
04-29 18:25:55 I/InvocationToJUnitResultForwarder: Running helloworldrun: 1 tests
04-29 18:25:55 W/InvocationToJUnitResultForwarder:
Test com.example.TestClassName#sampleTest failed with stack:
 oh noes, test failed
04-29 18:25:55 I/InvocationToJUnitResultForwarder: Run ended in 0 ms

To store the results of an invocation elsewhere, such as in a file, specify a custom ITestInvocationListener implementation using the result_reporter tag in your configuration.

TF also includes the XmlResultReporter listener, which writes test results to an XML file in a format similar to that used by the ant JUnit XML writer. To specify the result_reporter in the configuration, edit the …/res/config/example/helloworld.xml config:

<configuration description="Runs the hello world test">
    <test class="com.android.tradefed.example.HelloWorldTest" />
    <result_reporter class="com.android.tradefed.result.XmlResultReporter" />
</configuration>

Now rebuild tradefed and re-run the hello world sample:

tf> run example/helloworld
05-16 21:07:07 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World! I have device 004ad9880810a548
05-16 21:07:07 I/XmlResultReporter: Saved device_logcat log to /tmp/0/inv_2991649128735283633/device_logcat_6999997036887173857.txt
05-16 21:07:07 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_2991649128735283633/host_log_6307746032218561704.txt
05-16 21:07:07 I/XmlResultReporter: XML test result file generated at /tmp/0/inv_2991649128735283633/test_result_536358148261684076.xml. Total tests 1, Failed 1, Error 0

Notice the log message stating that an XML file has been generated; the generated file should look like this:

<?xml version='1.0' encoding='UTF-8' ?>
<testsuite name="stub" tests="1" failures="1" errors="0" time="9" timestamp="2011-05-17T04:07:07" hostname="localhost">
  <properties />
  <testcase name="sampleTest" classname="com.example.TestClassName" time="0">
    <failure>oh noes, test failed
    </failure>
  </testcase>
</testsuite>

You can also write your own custom invocation listeners—they simply need to implement the ITestInvocationListener interface.

Tradefed supports multiple invocation listeners, so you can send test results to multiple independent destinations. To do this, just specify multiple <result_reporter> tags in your config.

Logging facilities (D, I, R)

TF's logging facilities include the ability to:

  1. Capture logs from the device (aka device logcat)
  2. Record logs from the Trade Federation framework running on the host machine (aka host log)

The TF framework automatically captures the logcat from the allocated device and sends it to the invocation listener for processing. XmlResultReporter then saves the captured device logcat as a file.

TF host logs are reported using the CLog wrapper for the ddmlib Log class. Let's convert the previous System.out.println call in HelloWorldTest to a CLog call:

@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
    CLog.i("Hello, TF World! I have device %s", getDevice().getSerialNumber());

CLog handles string interpolation directly, similar to String.format. When you rebuild and rerun TF, you should see the log message on stdout:

tf> run example/helloworld
…
05-16 21:30:46 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548
…

By default, tradefed outputs host log messages to stdout. TF also includes a log implementation that writes messages to a file: FileLogger. To add file logging, add a logger tag to the config, specifying the full class name of FileLogger:

<configuration description="Runs the hello world test">
    <test class="com.android.tradefed.example.HelloWorldTest" />
    <result_reporter class="com.android.tradefed.result.XmlResultReporter" />
    <logger class="com.android.tradefed.log.FileLogger" />
</configuration>

Now, rebuild and run the helloworld example again:

tf >run example/helloworld
…
05-16 21:38:21 I/XmlResultReporter: Saved device_logcat log to /tmp/0/inv_6390011618174565918/device_logcat_1302097394309452308.txt
05-16 21:38:21 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt
…

The log message indicates the path of the host log, which, when viewed, should contain your HelloWorldTest log message:

more /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt

Example output:

…
05-16 21:38:21 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548

Handling options (D, I, R)

Objects loaded from a TF Configuration (aka Configuration objects) can also receive data from command line arguments through the use of the @Option annotation.

To participate, a Configuration object class applies the @Option annotation to a member field and provides it a unique name. This enables that member field value to be populated via a command line option (and also automatically adds that option to the configuration help system).

Note: Not all field types are supported. For a description of supported types, see OptionSetter.

Let's add an @Option to HelloWorldTest:

@Option(name="my_option",
        shortName='m',
        description="this is the option's help text",
        // always display this option in the default help text
        importance=Importance.ALWAYS)
private String mMyOption = "thisisthedefault";

Next, let's add a log message to display the value of the option in HelloWorldTest so we can demonstrate it was received correctly:

@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
    …
    CLog.logAndDisplay(LogLevel.INFO, "I received option '%s'", mMyOption);

Finally, rebuild TF and run helloworld; you should see a log message with the my_option default value:

tf> run example/helloworld
…
05-24 18:30:05 I/HelloWorldTest: I received option 'thisisthedefault'

Pass values from the command line

Pass in a value for my_option; you should see my_option populated with that value:

tf> run example/helloworld --my_option foo
…
05-24 18:33:44 I/HelloWorldTest: I received option 'foo'

TF configurations also include a help system, which automatically displays help text for @Option fields. Try it now, and you should see the help text for my_option:

tf> run example/helloworld --help
Printing help for only the important options. To see help for all options, use the --help-all flag

  cmd_options options:
    --[no-]help          display the help text for the most important/critical options. Default: false.
    --[no-]help-all      display the full help text for all options. Default: false.
    --[no-]loop          keep running continuously. Default: false.

  test options:
    -m, --my_option      this is the option's help text Default: thisisthedefault.

  'file' logger options:
    --log-level-display  the minimum log level to display on stdout. Must be one of verbose, debug, info, warn, error, assert. Default: error.

Note the message about "printing only the important options." To reduce option help clutter, TF uses the Option#importance attribute to determine whether to show a particular @Option field help text when --help is specified. --help-all always shows help for all @Option fields, regardless of importance. For details, see Option.Importance.

Pass values from a configuration

You can also specify an Option value within the config by adding a <option name="" value=""> element. Test it using helloworld.xml:

<test class="com.android.tradefed.example.HelloWorldTest" >
    <option name="my_option" value="fromxml" />
</test>

Re-building and running helloworld should now produce this output:

05-24 20:38:25 I/HelloWorldTest: I received option 'fromxml'

The configuration help should also update to indicate the default value of my_option:

tf> run example/helloworld --help
  test options:
    -m, --my_option      this is the option's help text Default: fromxml.

Other configuration objects included in the helloworld config, such as FileLogger, also accept options. The option --log-level-display is interesting because it filters the logs that show up on stdout. Earlier in the tutorial, you may have noticed the "Hello, TF World! I have device …' log message stopped being displayed on stdout after we switched to using FileLogger. You can increase the verbosity of logging to stdout by passing in the --log-level-display arg.

Try this now, and you should see the 'I have device' log message reappear on stdout, in addition to being logged to a file:

tf> run example/helloworld --log-level-display info
…
05-24 18:53:50 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548

That's all, folks!

As a reminder, if you're stuck on something, the Trade Federation source code has a lot of useful information that isn't exposed in the documentation. If all else fails, try asking on the android-platform Google Group, with "Trade Federation" in the message subject.