We use TestNG to run our tests. In one particular build we currently have 7714 different tests. But some of those tests are run multiple times with different parameters using a TestNG DataProvider (
http://testng.org/doc/documentation-main.html#parameters-dataproviders)
This means that the XML results can have multiple rows for the same test, where perhaps one run of the test with one set of parameters passes, but the same test run with different parameters fails.
When you add up all the runs of all the tests for this build there are 7928, and in a recent run one of those failed. But Pulse only reports 7714 tests run and all successful, because it seems to ignore multiple runs of the same test.
Here is an example of part of the XML output from TestNG for this build
<testcase name="testUpdateDisplay" time="0.0050" classname="com.teradata.trm.segmentation.web.struts.action.test.SelectionBaseActionTests"/>
<testcase name="testUpdateDisplay" time="0.0" classname="com.teradata.trm.segmentation.web.struts.action.test.SelectionBaseActionTests"/>
<testcase name="testUpdateDisplay" time="0.0" classname="com.teradata.trm.segmentation.web.struts.action.test.SelectionBaseActionTests"/>
<testcase name="testUpdateDisplay" time="0.0020" classname="com.teradata.trm.segmentation.web.struts.action.test.SelectionBaseActionTests"/>
<testcase name="testUpdateDisplay" time="0.0030" classname="com.teradata.trm.segmentation.web.struts.action.test.SelectionBaseActionTests"/>
<testcase name="testUpdateDisplay" time="0.0020" classname="com.teradata.trm.segmentation.web.struts.action.test.SelectionBaseActionTests"/>
<testcase name="testUpdateDisplay" time="0.0030" classname="com.teradata.trm.segmentation.web.struts.action.test.SelectionBaseActionTests"/>
<testcase name="testUpdateDisplay" time="0.0030" classname="com.teradata.trm.segmentation.web.struts.action.test.SelectionBaseActionTests">
<failure type="java.lang.AssertionError" message="expected:<1> but was:<2>">
<![CDATA[java.lang.AssertionError: expected:<1> but was:<2>
at com.teradata.trm.segmentation.web.struts.action.test.SelectionBaseActionTests.testUpdateDisplay(SelectionBaseActionTests.java:188)
... Removed 27 stack frames]]>
</failure>
</testcase>
<testcase name="createNewSelectionRangeValue" time="0.0" classname="com.teradata.trm.segmentation.web.struts.action.test.SelectionBaseActionTests"/>
Bug
Resolved
Major
Pulse ignores multiple runs of the same test
This is actually the intended behaviour of the post-processor. In some cases builds can pick up test cases multiple times and the default behaviour is to collapse them into one case. What is more, there is currently no support for having two tests with the exact same name/path.
I can see that in your situation the default behaviour is not desirable. I think perhaps this can be solved by allowing the behaviour to be customised. In fact, another processor (RegexTestPostProcessor) already supports customisable conflict resolution. By default conflicting cases are combined, but you can also tell Pulse to append (or prepend) a number to make the case name unique. So in your case you Pulse test reports would end up with several cases with names:
testUpdateDisplay
testUpdateDisplay2
testUpdateDisplay3
testUpdateDisplay4
and so on. Would this solve your problem?