CollectingTestListener
public
class
CollectingTestListener
extends Object
implements
IDisableable,
ILogSaverListener,
ITestInvocationListener
java.lang.Object | |
↳ | com.android.tradefed.result.CollectingTestListener |
A ITestInvocationListener
that will collect all test results.
Although the data structures used in this object are thread-safe, the ITestInvocationListener
callbacks must be called in the correct order.
Summary
Public constructors | |
---|---|
CollectingTestListener()
|
Public methods | |
---|---|
IBuildInfo
|
getBuildInfo()
This method is deprecated.
rely on the |
TestRunResult
|
getCurrentRunResults()
Gets the results for the current test run. |
int
|
getExpectedTests()
Returns the number of expected tests count. |
IInvocationContext
|
getInvocationContext()
Return the invocation context that was reported via |
|
getMergedTestRunResults()
Return the merged collection of results for all runs across different attempts. |
IInvocationContext
|
getModuleContextForRunResult(String testRunName)
Returns the |
MultiMap<String, LogFile>
|
getModuleLogFiles()
Returns a copy of the map containing all the logged file associated with the module |
MultiMap<String, LogFile>
|
getNonAssociatedLogFiles()
Returns a copy of the map containing all the logged file not associated with a test run or a module. |
int
|
getNumAllFailedTestRuns()
Returns the total number of test runs in a failure state |
int
|
getNumAllFailedTests()
Returns the total number of tests in a failure state (only failed, assumption failures do not count toward it). |
int
|
getNumTestsInState(TestStatus status)
Returns the number of tests in given state for this run. |
int
|
getNumTestsInState(TestResult.TestStatus ddmlibStatus)
For compatibility with older status type |
int
|
getNumTotalTests()
Returns the total number of complete tests for all runs. |
IBuildInfo
|
getPrimaryBuildInfo()
Return the primary build info that was reported via |
|
getRunResults()
This method is deprecated.
Use |
TestRunResult
|
getTestRunAtAttempt(String testRunName, int attempt)
Return the |
int
|
getTestRunAttemptCount(String testRunName)
Returns the number of attempts for a given test run name. |
|
getTestRunAttempts(String testRunName)
Gets all the attempts for a |
|
getTestRunForAttempts(int attempt)
Gets all the results for a given attempt. |
|
getTestRunNames()
Return all the names for all the test runs. |
boolean
|
hasFailedTests()
Returns if the invocation had any failed or assumption failed tests. |
boolean
|
hasTestRunResultsForName(String testRunName)
Returns whether a given test run name has any results. |
void
|
invocationEnded(long elapsedTime)
Reports that the invocation has terminated, whether successfully or due to some error condition. |
void
|
invocationFailed(Throwable cause)
Reports an incomplete invocation due to some error condition. |
void
|
invocationSkipped(SkipReason reason)
Reports an invocation as skipped |
void
|
invocationStarted(IInvocationContext context)
Reports the start of the test invocation. |
boolean
|
isDisabled()
Returns True if entire object disabled (skip both setup and teardown). |
void
|
logAssociation(String dataName, LogFile logFile)
In some cases, log must be strongly associated with a test cases, but the opportunity to do
so on the direct |
void
|
setBuildInfo(IBuildInfo buildInfo)
This method is deprecated. Not necessary for testing anymore. |
void
|
setDisable(boolean isDisabled)
Sets whether the object should be disabled. |
void
|
setMergeStrategy(MergeStrategy strategy)
Sets the |
void
|
testAssumptionFailure(TestDescription test, String trace)
Called when an atomic test flags that it assumes a condition that is false |
void
|
testAssumptionFailure(TestDescription test, FailureDescription failure)
Called when an atomic test flags that it assumes a condition that is false |
void
|
testEnded(TestDescription test, long endTime,
Alternative to |
void
|
testEnded(TestDescription test,
Reports the execution end of an individual test case. |
void
|
testFailed(TestDescription test, FailureDescription failure)
Reports the failure of a individual test case. |
void
|
testFailed(TestDescription test, String trace)
Reports the failure of a individual test case. |
void
|
testIgnored(TestDescription test)
Called when a test will not be run, generally because a test method is annotated with org.junit.Ignore. |
void
|
testModuleEnded()
Reports the end of a module run. |
void
|
testModuleStarted(IInvocationContext moduleContext)
Reports the beginning of a module running. |
void
|
testRunEnded(long elapsedTime,
Reports end of test run. |
void
|
testRunFailed(FailureDescription failure)
Reports test run failed to complete due to a failure described by |
void
|
testRunFailed(String errorMessage)
Reports test run failed to complete due to a fatal error. |
void
|
testRunStarted(String name, int numTests, int attemptNumber)
Reports the start of a test run. |
void
|
testRunStarted(String name, int numTests)
Reports the start of a test run. |
void
|
testRunStarted(String name, int numTests, int attemptNumber, long startTime)
Reports the start of a test run. |
void
|
testRunStopped(long elapsedTime)
Reports test run stopped before completion due to a user request. |
void
|
testSkipped(TestDescription test, SkipReason reason)
Called when a test is skipped and did not execute for a reason that is not usually expected. |
void
|
testStarted(TestDescription test, long startTime)
Alternative to |
void
|
testStarted(TestDescription test)
Reports the start of an individual test case. |
Protected methods | |
---|---|
final
void
|
clearModuleLogFiles()
Allows cleaning the module file so we avoid carrying them for too long. |
final
void
|
clearResultsForName(String testRunName)
Allows to clear the results for a given run name. |
void
|
setIsAggregrateMetrics(boolean aggregate)
Toggle the 'aggregate metrics' option |
Public constructors
CollectingTestListener
public CollectingTestListener ()
Public methods
getBuildInfo
public IBuildInfo getBuildInfo ()
This method is deprecated.
rely on the IBuildInfo
from getInvocationContext()
.
Returns the build info.
Returns | |
---|---|
IBuildInfo |
getCurrentRunResults
public TestRunResult getCurrentRunResults ()
Gets the results for the current test run.
Note the results may not be complete. It is recommended to test the value of TestRunResult.isRunComplete()
and/or (@link TestRunResult#isRunFailure()} as appropriate
before processing the results.
Returns | |
---|---|
TestRunResult |
the TestRunResult representing data collected during last test run |
getExpectedTests
public int getExpectedTests ()
Returns the number of expected tests count. Could differ from getNumTotalTests()
if
some tests did not run.
Returns | |
---|---|
int |
getInvocationContext
public IInvocationContext getInvocationContext ()
Return the invocation context that was reported via invocationStarted(com.android.tradefed.invoker.IInvocationContext)
Returns | |
---|---|
IInvocationContext |
getMergedTestRunResults
publicgetMergedTestRunResults ()
Return the merged collection of results for all runs across different attempts.
If there are multiple results, each test run is merged, with the latest test result overwriting test results of previous runs. Test runs are ordered by attempt number.
Metrics for the same attempt will be merged based on the preference set by aggregate-metrics
. The final metrics will be the metrics of the last attempt.
Returns | |
---|---|
|
getModuleContextForRunResult
public IInvocationContext getModuleContextForRunResult (String testRunName)
Returns the IInvocationContext
of the module associated with the results.
Parameters | |
---|---|
testRunName |
String : The name given by {testRunStarted(String, int) . |
Returns | |
---|---|
IInvocationContext |
The IInvocationContext of the module for a given test run name null
if there are no results for that name. |
getModuleLogFiles
public MultiMap<String, LogFile> getModuleLogFiles ()
Returns a copy of the map containing all the logged file associated with the module
Returns | |
---|---|
MultiMap<String, LogFile> |
getNonAssociatedLogFiles
public MultiMap<String, LogFile> getNonAssociatedLogFiles ()
Returns a copy of the map containing all the logged file not associated with a test run or a module.
Returns | |
---|---|
MultiMap<String, LogFile> |
getNumAllFailedTestRuns
public int getNumAllFailedTestRuns ()
Returns the total number of test runs in a failure state
Returns | |
---|---|
int |
getNumAllFailedTests
public int getNumAllFailedTests ()
Returns the total number of tests in a failure state (only failed, assumption failures do not count toward it).
Returns | |
---|---|
int |
getNumTestsInState
public int getNumTestsInState (TestStatus status)
Returns the number of tests in given state for this run.
Parameters | |
---|---|
status |
TestStatus |
Returns | |
---|---|
int |
getNumTestsInState
public int getNumTestsInState (TestResult.TestStatus ddmlibStatus)
For compatibility with older status type
Parameters | |
---|---|
ddmlibStatus |
TestResult.TestStatus |
Returns | |
---|---|
int |
getNumTotalTests
public int getNumTotalTests ()
Returns the total number of complete tests for all runs.
Returns | |
---|---|
int |
getPrimaryBuildInfo
public IBuildInfo getPrimaryBuildInfo ()
Return the primary build info that was reported via invocationStarted(com.android.tradefed.invoker.IInvocationContext)
. Primary build is the build returned by the first
build provider of the running configuration. Returns null if there is no context (no build to
test case).
Returns | |
---|---|
IBuildInfo |
getRunResults
publicgetRunResults ()
This method is deprecated.
Use getMergedTestRunResults()
Returns the results for all test runs.
Returns | |
---|---|
|
getTestRunAtAttempt
public TestRunResult getTestRunAtAttempt (String testRunName, int attempt)
Return the TestRunResult
for a single attempt.
Parameters | |
---|---|
testRunName |
String : The name given by {testRunStarted(String, int) . |
attempt |
int : The attempt id. |
Returns | |
---|---|
TestRunResult |
The TestRunResult for the given name and attempt id or null if it
does not exist. |
getTestRunAttemptCount
public int getTestRunAttemptCount (String testRunName)
Returns the number of attempts for a given test run name.
Parameters | |
---|---|
testRunName |
String : The name given by {testRunStarted(String, int) . |
Returns | |
---|---|
int |
getTestRunAttempts
publicgetTestRunAttempts (String testRunName)
Gets all the attempts for a TestRunResult
of a given test run.
Parameters | |
---|---|
testRunName |
String : The name given by {testRunStarted(String, int) . |
Returns | |
---|---|
|
All TestRunResult for a given test run, ordered by attempts. |
getTestRunForAttempts
publicgetTestRunForAttempts (int attempt)
Gets all the results for a given attempt.
Parameters | |
---|---|
attempt |
int : The attempt we want results for. |
Returns | |
---|---|
|
All TestRunResult for a given attempt. |
getTestRunNames
publicgetTestRunNames ()
Return all the names for all the test runs.
These test runs may have run multiple times with different attempts.
Returns | |
---|---|
|
hasFailedTests
public boolean hasFailedTests ()
Returns if the invocation had any failed or assumption failed tests.
Returns | |
---|---|
boolean |
hasTestRunResultsForName
public boolean hasTestRunResultsForName (String testRunName)
Returns whether a given test run name has any results.
Parameters | |
---|---|
testRunName |
String : The name given by {testRunStarted(String, int) . |
Returns | |
---|---|
boolean |
invocationEnded
public void invocationEnded (long elapsedTime)
Reports that the invocation has terminated, whether successfully or due to some error condition.
Will be automatically called by the TradeFederation framework.Parameters | |
---|---|
elapsedTime |
long : the elapsed time of the invocation in ms |
invocationFailed
public void invocationFailed (Throwable cause)
Reports an incomplete invocation due to some error condition.
Will be automatically called by the TradeFederation framework.Parameters | |
---|---|
cause |
Throwable : the Throwable cause of the failure |
invocationSkipped
public void invocationSkipped (SkipReason reason)
Reports an invocation as skipped
Parameters | |
---|---|
reason |
SkipReason |
invocationStarted
public void invocationStarted (IInvocationContext context)
Reports the start of the test invocation.
Will be automatically called by the TradeFederation framework. Reporters need to override this method to support multiple devices reporting.
Parameters | |
---|---|
context |
IInvocationContext : information about the invocation |
isDisabled
public boolean isDisabled ()
Returns True if entire object disabled (skip both setup and teardown). False otherwise.
Returns | |
---|---|
boolean |
logAssociation
public void logAssociation (String dataName, LogFile logFile)
In some cases, log must be strongly associated with a test cases, but the opportunity to do
so on the direct testLogSaved(String, com.android.tradefed.result.LogDataType, com.android.tradefed.result.InputStreamSource, com.android.tradefed.result.LogFile)
callback is not possible. Thus, this callback allows to provide a strong association
explicitly.
Parameters | |
---|---|
dataName |
String : The name of the data |
logFile |
LogFile : the LogFile that was logged before and should be associated with the
test case. |
setBuildInfo
public void setBuildInfo (IBuildInfo buildInfo)
This method is deprecated.
Not necessary for testing anymore.
Set the build info. Should only be used for testing.
Parameters | |
---|---|
buildInfo |
IBuildInfo |
setDisable
public void setDisable (boolean isDisabled)
Sets whether the object should be disabled. Disabled means that both setup and teardown steps should be skipped. Can be use to make an object disabled by default in the default constructor.
Parameters | |
---|---|
isDisabled |
boolean : the state the object should be put in. |
setMergeStrategy
public void setMergeStrategy (MergeStrategy strategy)
Sets the MergeStrategy
to use when merging results.
Parameters | |
---|---|
strategy |
MergeStrategy |
testAssumptionFailure
public void testAssumptionFailure (TestDescription test, String trace)
Called when an atomic test flags that it assumes a condition that is false
Parameters | |
---|---|
test |
TestDescription : identifies the test |
trace |
String : stack trace of failure |
testAssumptionFailure
public void testAssumptionFailure (TestDescription test, FailureDescription failure)
Called when an atomic test flags that it assumes a condition that is false
Parameters | |
---|---|
test |
TestDescription : identifies the test |
failure |
FailureDescription : FailureDescription describing the failure and its context. |
testEnded
public void testEnded (TestDescription test, long endTime,testMetrics)
Alternative to ERROR(/#testEnded(com.android.tradefed.result.TestDescription,Map))
where we can specify the end time
directly. Combine with testStarted(com.android.tradefed.result.TestDescription, long)
for accurate measure.
Parameters | |
---|---|
test |
TestDescription : identifies the test |
endTime |
long : the time the test ended, measured via System.currentTimeMillis() |
testMetrics |
: a ERROR(/Map) of the metrics emitted |
testEnded
public void testEnded (TestDescription test,testMetrics)
Reports the execution end of an individual test case.
If testFailed(TestDescription, FailureDescription)
was not invoked, this test passed. Also returns any key/value
metrics which may have been emitted during the test case's execution.
Parameters | |
---|---|
test |
TestDescription : identifies the test |
testMetrics |
: a ERROR(/Map) of the metrics emitted |
testFailed
public void testFailed (TestDescription test, FailureDescription failure)
Reports the failure of a individual test case.
Will be called between testStarted and testEnded.
Parameters | |
---|---|
test |
TestDescription : identifies the test |
failure |
FailureDescription : FailureDescription describing the failure and its context. |
testFailed
public void testFailed (TestDescription test, String trace)
Reports the failure of a individual test case.
Will be called between testStarted and testEnded.
Parameters | |
---|---|
test |
TestDescription : identifies the test |
trace |
String : stack trace of failure |
testIgnored
public void testIgnored (TestDescription test)
Called when a test will not be run, generally because a test method is annotated with org.junit.Ignore.
Parameters | |
---|---|
test |
TestDescription : identifies the test |
testModuleEnded
public void testModuleEnded ()
Reports the end of a module run.
testModuleStarted
public void testModuleStarted (IInvocationContext moduleContext)
Reports the beginning of a module running. This callback is associated with testModuleEnded()
and is optional in the sequence. It is only used during a run that uses
modules: suite based runners.
Parameters | |
---|---|
moduleContext |
IInvocationContext : the IInvocationContext of the module. |
testRunEnded
public void testRunEnded (long elapsedTime,runMetrics)
Reports end of test run.
Parameters | |
---|---|
elapsedTime |
long : device reported elapsed time, in milliseconds |
runMetrics |
: key-value pairs reported at the end of a test run |
testRunFailed
public void testRunFailed (FailureDescription failure)
Reports test run failed to complete due to a failure described by FailureDescription
.
Parameters | |
---|---|
failure |
FailureDescription : FailureDescription describing the failure and its context. |
testRunFailed
public void testRunFailed (String errorMessage)
Reports test run failed to complete due to a fatal error.
Parameters | |
---|---|
errorMessage |
String : String describing reason for run failure. |
testRunStarted
public void testRunStarted (String name, int numTests, int attemptNumber)
Reports the start of a test run.
Parameters | |
---|---|
name |
String : the test run name |
numTests |
int : total number of tests in test run |
attemptNumber |
int : order number, identifying the different attempts of the same runName
that run multiple times. The attemptNumber is 0-indexed and should increment everytime
a new run happens. e.g. A test is granular retried 3 times, it should have 4 total
runs under the same runName and the attemptNumber is from 0 to 3. |
testRunStarted
public void testRunStarted (String name, int numTests)
Reports the start of a test run.
Parameters | |
---|---|
name |
String : the test run name |
numTests |
int : total number of tests in test run |
testRunStarted
public void testRunStarted (String name, int numTests, int attemptNumber, long startTime)
Reports the start of a test run.
Parameters | |
---|---|
name |
String : the test run name |
numTests |
int : total number of tests in test run |
attemptNumber |
int : order number, identifying the different attempts of the same runName
that run multiple times. The attemptNumber is 0-indexed and should increment everytime a
new run happens. e.g. A test is granular retried 3 times, it should have 4 total runs
under the same runName and the attemptNumber is from 0 to 3. |
startTime |
long : the time the run started, measured via System.currentTimeMillis() |
testRunStopped
public void testRunStopped (long elapsedTime)
Reports test run stopped before completion due to a user request.
TODO: currently unused, consider removing
Parameters | |
---|---|
elapsedTime |
long : device reported elapsed time, in milliseconds |
testSkipped
public void testSkipped (TestDescription test, SkipReason reason)
Called when a test is skipped and did not execute for a reason that is not usually expected. These tests will be attempted to be retried to attempt to get a proper execution.
Parameters | |
---|---|
test |
TestDescription : identifies the test |
reason |
SkipReason : SkipReason |
testStarted
public void testStarted (TestDescription test, long startTime)
Alternative to testStarted(com.android.tradefed.result.TestDescription)
where we also specify when the test was
started, combined with ERROR(/#testEnded(com.android.tradefed.result.TestDescription,long,Map))
for accurate measure.
Parameters | |
---|---|
test |
TestDescription : identifies the test |
startTime |
long : the time the test started, measured via System.currentTimeMillis() |
testStarted
public void testStarted (TestDescription test)
Reports the start of an individual test case. Older interface, should use testStarted(com.android.tradefed.result.TestDescription)
whenever possible.
Parameters | |
---|---|
test |
TestDescription : identifies the test |
Protected methods
clearModuleLogFiles
protected final void clearModuleLogFiles ()
Allows cleaning the module file so we avoid carrying them for too long.
clearResultsForName
protected final void clearResultsForName (String testRunName)
Allows to clear the results for a given run name. Should only be used in some cases like the aggregator of results.
Parameters | |
---|---|
testRunName |
String |
setIsAggregrateMetrics
protected void setIsAggregrateMetrics (boolean aggregate)
Toggle the 'aggregate metrics' option
Parameters | |
---|---|
aggregate |
boolean |