CollectingTestListener

public class CollectingTestListener
extends Object implements ITestInvocationListener, ILogSaverListener, IDisableable

java.lang.Object
   ↳ com.android.tradefed.result.CollectingTestListener


A ITestInvocationListener that will collect all test results.

Although the data structures used in this object are thread-safe, the ITestInvocationListener callbacks must be called in the correct order.

Summary

Public constructors

CollectingTestListener()

Public methods

IBuildInfo getBuildInfo()

This method is deprecated. rely on the IBuildInfo from getInvocationContext().

TestRunResult getCurrentRunResults()

Gets the results for the current test run.

int getExpectedTests()

Returns the number of expected tests count.

IInvocationContext getInvocationContext()

Return the invocation context that was reported via invocationStarted(com.android.tradefed.invoker.IInvocationContext)

getMergedTestRunResults()

Return the merged collection of results for all runs across different attempts.

IInvocationContext getModuleContextForRunResult(String testRunName)

Returns the IInvocationContext of the module associated with the results.

MultiMap<String, LogFile> getModuleLogFiles()

Returns a copy of the map containing all the logged file associated with the module

MultiMap<String, LogFile> getNonAssociatedLogFiles()

Returns a copy of the map containing all the logged file not associated with a test run or a module.

int getNumAllFailedTestRuns()

Returns the total number of test runs in a failure state

int getNumAllFailedTests()

Returns the total number of tests in a failure state (only failed, assumption failures do not count toward it).

int getNumTestsInState(TestStatus status)

Returns the number of tests in given state for this run.

int getNumTestsInState(TestResult.TestStatus ddmlibStatus)

For compatibility with older status type

int getNumTotalTests()

Returns the total number of complete tests for all runs.

IBuildInfo getPrimaryBuildInfo()

Return the primary build info that was reported via invocationStarted(com.android.tradefed.invoker.IInvocationContext).

getRunResults()

This method is deprecated. Use getMergedTestRunResults()

TestRunResult getTestRunAtAttempt(String testRunName, int attempt)

Return the TestRunResult for a single attempt.

int getTestRunAttemptCount(String testRunName)

Returns the number of attempts for a given test run name.

getTestRunAttempts(String testRunName)

Gets all the attempts for a TestRunResult of a given test run.

getTestRunForAttempts(int attempt)

Gets all the results for a given attempt.

getTestRunNames()

Return all the names for all the test runs.

boolean hasFailedTests()

Returns if the invocation had any failed or assumption failed tests.

boolean hasTestRunResultsForName(String testRunName)

Returns whether a given test run name has any results.

void invocationEnded(long elapsedTime)

Reports that the invocation has terminated, whether successfully or due to some error condition.

void invocationFailed(Throwable cause)

Reports an incomplete invocation due to some error condition.

void invocationSkipped(SkipReason reason)

Reports an invocation as skipped

void invocationStarted(IInvocationContext context)

Reports the start of the test invocation.

boolean isDisabled()

Returns True if entire object disabled (skip both setup and teardown).

void logAssociation(String dataName, LogFile logFile)

In some cases, log must be strongly associated with a test cases, but the opportunity to do so on the direct testLogSaved(String, com.android.tradefed.result.LogDataType, com.android.tradefed.result.InputStreamSource, com.android.tradefed.result.LogFile) callback is not possible.

void setBuildInfo(IBuildInfo buildInfo)

This method is deprecated. Not necessary for testing anymore.

void setDisable(boolean isDisabled)

Sets whether the object should be disabled.

void setMergeStrategy(MergeStrategy strategy)

Sets the MergeStrategy to use when merging results.

void testAssumptionFailure(TestDescription test, String trace)

Called when an atomic test flags that it assumes a condition that is false

void testAssumptionFailure(TestDescription test, FailureDescription failure)

Called when an atomic test flags that it assumes a condition that is false

void testEnded(TestDescription test, long endTime, testMetrics)

Alternative to ERROR(/#testEnded(com.android.tradefed.result.TestDescription,Map)) where we can specify the end time directly.

void testEnded(TestDescription test, testMetrics)

Reports the execution end of an individual test case.

void testFailed(TestDescription test, FailureDescription failure)

Reports the failure of a individual test case.

void testFailed(TestDescription test, String trace)

Reports the failure of a individual test case.

void testIgnored(TestDescription test)

Called when a test will not be run, generally because a test method is annotated with org.junit.Ignore.

void testModuleEnded()

Reports the end of a module run.

void testModuleStarted(IInvocationContext moduleContext)

Reports the beginning of a module running.

void testRunEnded(long elapsedTime, runMetrics)

Reports end of test run.

void testRunFailed(FailureDescription failure)

Reports test run failed to complete due to a failure described by FailureDescription.

void testRunFailed(String errorMessage)

Reports test run failed to complete due to a fatal error.

void testRunStarted(String name, int numTests, int attemptNumber)

Reports the start of a test run.

void testRunStarted(String name, int numTests)

Reports the start of a test run.

void testRunStarted(String name, int numTests, int attemptNumber, long startTime)

Reports the start of a test run.

void testRunStopped(long elapsedTime)

Reports test run stopped before completion due to a user request.

void testSkipped(TestDescription test, SkipReason reason)

Called when a test is skipped and did not execute for a reason that is not usually expected.

void testStarted(TestDescription test, long startTime)

Alternative to testStarted(com.android.tradefed.result.TestDescription) where we also specify when the test was started, combined with ERROR(/#testEnded(com.android.tradefed.result.TestDescription,long,Map)) for accurate measure.

void testStarted(TestDescription test)

Reports the start of an individual test case.

Protected methods

final void clearModuleLogFiles()

Allows cleaning the module file so we avoid carrying them for too long.

final void clearResultsForName(String testRunName)

Allows to clear the results for a given run name.

void setIsAggregrateMetrics(boolean aggregate)

Toggle the 'aggregate metrics' option

Public constructors

CollectingTestListener

public CollectingTestListener ()

Public methods

getBuildInfo

public IBuildInfo getBuildInfo ()

This method is deprecated.
rely on the IBuildInfo from getInvocationContext().

Returns the build info.

Returns
IBuildInfo

getCurrentRunResults

public TestRunResult getCurrentRunResults ()

Gets the results for the current test run.

Note the results may not be complete. It is recommended to test the value of TestRunResult.isRunComplete() and/or (@link TestRunResult#isRunFailure()} as appropriate before processing the results.

Returns
TestRunResult the TestRunResult representing data collected during last test run

getExpectedTests

public int getExpectedTests ()

Returns the number of expected tests count. Could differ from getNumTotalTests() if some tests did not run.

Returns
int

getInvocationContext

public IInvocationContext getInvocationContext ()

Return the invocation context that was reported via invocationStarted(com.android.tradefed.invoker.IInvocationContext)

Returns
IInvocationContext

getMergedTestRunResults

public  getMergedTestRunResults ()

Return the merged collection of results for all runs across different attempts.

If there are multiple results, each test run is merged, with the latest test result overwriting test results of previous runs. Test runs are ordered by attempt number.

Metrics for the same attempt will be merged based on the preference set by aggregate-metrics. The final metrics will be the metrics of the last attempt.

Returns

getModuleContextForRunResult

public IInvocationContext getModuleContextForRunResult (String testRunName)

Returns the IInvocationContext of the module associated with the results.

Parameters
testRunName String: The name given by {testRunStarted(String, int).

Returns
IInvocationContext The IInvocationContext of the module for a given test run name null if there are no results for that name.

getModuleLogFiles

public MultiMap<String, LogFile> getModuleLogFiles ()

Returns a copy of the map containing all the logged file associated with the module

Returns
MultiMap<String, LogFile>

getNonAssociatedLogFiles

public MultiMap<String, LogFile> getNonAssociatedLogFiles ()

Returns a copy of the map containing all the logged file not associated with a test run or a module.

Returns
MultiMap<String, LogFile>

getNumAllFailedTestRuns

public int getNumAllFailedTestRuns ()

Returns the total number of test runs in a failure state

Returns
int

getNumAllFailedTests

public int getNumAllFailedTests ()

Returns the total number of tests in a failure state (only failed, assumption failures do not count toward it).

Returns
int

getNumTestsInState

public int getNumTestsInState (TestStatus status)

Returns the number of tests in given state for this run.

Parameters
status TestStatus

Returns
int

getNumTestsInState

public int getNumTestsInState (TestResult.TestStatus ddmlibStatus)

For compatibility with older status type

Parameters
ddmlibStatus TestResult.TestStatus

Returns
int

getNumTotalTests

public int getNumTotalTests ()

Returns the total number of complete tests for all runs.

Returns
int

getPrimaryBuildInfo

public IBuildInfo getPrimaryBuildInfo ()

Return the primary build info that was reported via invocationStarted(com.android.tradefed.invoker.IInvocationContext). Primary build is the build returned by the first build provider of the running configuration. Returns null if there is no context (no build to test case).

Returns
IBuildInfo

getRunResults

public  getRunResults ()

This method is deprecated.
Use getMergedTestRunResults()

Returns the results for all test runs.

Returns

getTestRunAtAttempt

public TestRunResult getTestRunAtAttempt (String testRunName, 
                int attempt)

Return the TestRunResult for a single attempt.

Parameters
testRunName String: The name given by {testRunStarted(String, int).

attempt int: The attempt id.

Returns
TestRunResult The TestRunResult for the given name and attempt id or null if it does not exist.

getTestRunAttemptCount

public int getTestRunAttemptCount (String testRunName)

Returns the number of attempts for a given test run name.

Parameters
testRunName String: The name given by {testRunStarted(String, int).

Returns
int

getTestRunAttempts

public  getTestRunAttempts (String testRunName)

Gets all the attempts for a TestRunResult of a given test run.

Parameters
testRunName String: The name given by {testRunStarted(String, int).

Returns
All TestRunResult for a given test run, ordered by attempts.

getTestRunForAttempts

public  getTestRunForAttempts (int attempt)

Gets all the results for a given attempt.

Parameters
attempt int: The attempt we want results for.

Returns
All TestRunResult for a given attempt.

getTestRunNames

public  getTestRunNames ()

Return all the names for all the test runs.

These test runs may have run multiple times with different attempts.

Returns

hasFailedTests

public boolean hasFailedTests ()

Returns if the invocation had any failed or assumption failed tests.

Returns
boolean

hasTestRunResultsForName

public boolean hasTestRunResultsForName (String testRunName)

Returns whether a given test run name has any results.

Parameters
testRunName String: The name given by {testRunStarted(String, int).

Returns
boolean

invocationEnded

public void invocationEnded (long elapsedTime)

Reports that the invocation has terminated, whether successfully or due to some error condition.

Will be automatically called by the TradeFederation framework.

Parameters
elapsedTime long: the elapsed time of the invocation in ms

invocationFailed

public void invocationFailed (Throwable cause)

Reports an incomplete invocation due to some error condition.

Will be automatically called by the TradeFederation framework.

Parameters
cause Throwable: the Throwable cause of the failure

invocationSkipped

public void invocationSkipped (SkipReason reason)

Reports an invocation as skipped

Parameters
reason SkipReason

invocationStarted

public void invocationStarted (IInvocationContext context)

Reports the start of the test invocation.

Will be automatically called by the TradeFederation framework. Reporters need to override this method to support multiple devices reporting.

Parameters
context IInvocationContext: information about the invocation

isDisabled

public boolean isDisabled ()

Returns True if entire object disabled (skip both setup and teardown). False otherwise.

Returns
boolean

logAssociation

public void logAssociation (String dataName, 
                LogFile logFile)

In some cases, log must be strongly associated with a test cases, but the opportunity to do so on the direct testLogSaved(String, com.android.tradefed.result.LogDataType, com.android.tradefed.result.InputStreamSource, com.android.tradefed.result.LogFile) callback is not possible. Thus, this callback allows to provide a strong association explicitly.

Parameters
dataName String: The name of the data

logFile LogFile: the LogFile that was logged before and should be associated with the test case.

setBuildInfo

public void setBuildInfo (IBuildInfo buildInfo)

This method is deprecated.
Not necessary for testing anymore.

Set the build info. Should only be used for testing.

Parameters
buildInfo IBuildInfo

setDisable

public void setDisable (boolean isDisabled)

Sets whether the object should be disabled. Disabled means that both setup and teardown steps should be skipped. Can be use to make an object disabled by default in the default constructor.

Parameters
isDisabled boolean: the state the object should be put in.

setMergeStrategy

public void setMergeStrategy (MergeStrategy strategy)

Sets the MergeStrategy to use when merging results.

Parameters
strategy MergeStrategy

testAssumptionFailure

public void testAssumptionFailure (TestDescription test, 
                String trace)

Called when an atomic test flags that it assumes a condition that is false

Parameters
test TestDescription: identifies the test

trace String: stack trace of failure

testAssumptionFailure

public void testAssumptionFailure (TestDescription test, 
                FailureDescription failure)

Called when an atomic test flags that it assumes a condition that is false

Parameters
test TestDescription: identifies the test

failure FailureDescription: FailureDescription describing the failure and its context.

testEnded

public void testEnded (TestDescription test, 
                long endTime, 
                 testMetrics)

Alternative to ERROR(/#testEnded(com.android.tradefed.result.TestDescription,Map)) where we can specify the end time directly. Combine with testStarted(com.android.tradefed.result.TestDescription, long) for accurate measure.

Parameters
test TestDescription: identifies the test

endTime long: the time the test ended, measured via System.currentTimeMillis()

testMetrics : a ERROR(/Map) of the metrics emitted

testEnded

public void testEnded (TestDescription test, 
                 testMetrics)

Reports the execution end of an individual test case.

If testFailed(TestDescription, FailureDescription) was not invoked, this test passed. Also returns any key/value metrics which may have been emitted during the test case's execution.

Parameters
test TestDescription: identifies the test

testMetrics : a ERROR(/Map) of the metrics emitted

testFailed

public void testFailed (TestDescription test, 
                FailureDescription failure)

Reports the failure of a individual test case.

Will be called between testStarted and testEnded.

Parameters
test TestDescription: identifies the test

failure FailureDescription: FailureDescription describing the failure and its context.

testFailed

public void testFailed (TestDescription test, 
                String trace)

Reports the failure of a individual test case.

Will be called between testStarted and testEnded.

Parameters
test TestDescription: identifies the test

trace String: stack trace of failure

testIgnored

public void testIgnored (TestDescription test)

Called when a test will not be run, generally because a test method is annotated with org.junit.Ignore.

Parameters
test TestDescription: identifies the test

testModuleEnded

public void testModuleEnded ()

Reports the end of a module run.

testModuleStarted

public void testModuleStarted (IInvocationContext moduleContext)

Reports the beginning of a module running. This callback is associated with testModuleEnded() and is optional in the sequence. It is only used during a run that uses modules: suite based runners.

Parameters
moduleContext IInvocationContext: the IInvocationContext of the module.

testRunEnded

public void testRunEnded (long elapsedTime, 
                 runMetrics)

Reports end of test run. FIXME: We cannot have two Map<> interfaces with different type, so we have to use HashMap here.

Parameters
elapsedTime long: device reported elapsed time, in milliseconds

runMetrics : key-value pairs reported at the end of a test run with Metric.

testRunFailed

public void testRunFailed (FailureDescription failure)

Reports test run failed to complete due to a failure described by FailureDescription.

Parameters
failure FailureDescription: FailureDescription describing the failure and its context.

testRunFailed

public void testRunFailed (String errorMessage)

Reports test run failed to complete due to a fatal error.

Parameters
errorMessage String: String describing reason for run failure.

testRunStarted

public void testRunStarted (String name, 
                int numTests, 
                int attemptNumber)

Reports the start of a test run.

Parameters
name String: the test run name

numTests int: total number of tests in test run

attemptNumber int: order number, identifying the different attempts of the same runName that run multiple times. The attemptNumber is 0-indexed and should increment everytime a new run happens. e.g. A test is granular retried 3 times, it should have 4 total runs under the same runName and the attemptNumber is from 0 to 3.

testRunStarted

public void testRunStarted (String name, 
                int numTests)

Reports the start of a test run.

Parameters
name String: the test run name

numTests int: total number of tests in test run

testRunStarted

public void testRunStarted (String name, 
                int numTests, 
                int attemptNumber, 
                long startTime)

Reports the start of a test run.

Parameters
name String: the test run name

numTests int: total number of tests in test run

attemptNumber int: order number, identifying the different attempts of the same runName that run multiple times. The attemptNumber is 0-indexed and should increment everytime a new run happens. e.g. A test is granular retried 3 times, it should have 4 total runs under the same runName and the attemptNumber is from 0 to 3.

startTime long: the time the run started, measured via System.currentTimeMillis()

testRunStopped

public void testRunStopped (long elapsedTime)

Reports test run stopped before completion due to a user request.

TODO: currently unused, consider removing

Parameters
elapsedTime long: device reported elapsed time, in milliseconds

testSkipped

public void testSkipped (TestDescription test, 
                SkipReason reason)

Called when a test is skipped and did not execute for a reason that is not usually expected. These tests will be attempted to be retried to attempt to get a proper execution.

Parameters
test TestDescription: identifies the test

reason SkipReason: SkipReason

testStarted

public void testStarted (TestDescription test, 
                long startTime)

Alternative to testStarted(com.android.tradefed.result.TestDescription) where we also specify when the test was started, combined with ERROR(/#testEnded(com.android.tradefed.result.TestDescription,long,Map)) for accurate measure.

Parameters
test TestDescription: identifies the test

startTime long: the time the test started, measured via System.currentTimeMillis()

testStarted

public void testStarted (TestDescription test)

Reports the start of an individual test case. Older interface, should use testStarted(com.android.tradefed.result.TestDescription) whenever possible.

Parameters
test TestDescription: identifies the test

Protected methods

clearModuleLogFiles

protected final void clearModuleLogFiles ()

Allows cleaning the module file so we avoid carrying them for too long.

clearResultsForName

protected final void clearResultsForName (String testRunName)

Allows to clear the results for a given run name. Should only be used in some cases like the aggregator of results.

Parameters
testRunName String

setIsAggregrateMetrics

protected void setIsAggregrateMetrics (boolean aggregate)

Toggle the 'aggregate metrics' option

Parameters
aggregate boolean