History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: CIB-624
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: jason
Reporter: Binyan
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
Pulse

Incorrect test results reported when using checkout scheme: incremental update

Created: 14/Sep/06 12:03 PM   Updated: 05/Nov/06 01:39 PM
Component/s: Web UI
Affects Version/s: 1.1.10
Fix Version/s: 1.1.19, 1.2.2


 Description  « Hide
If the build fails before the test sare run in an incremental update scenario, then the UI reports that all tests ran even thought they didn't. Instead pulse is picking up the previous test run's results. Suggest using time markers in the build. If the test results' time stamps are before the build started then they should be disregarded.

 All   Comments   Change History      Sort Order:
jason - 14/Sep/06 03:25 PM
Hi,

We would prefer not to have Pulse second-guessing artifacts based on timestamps. In this case, our preferred solution is to clean out test reports at the start of the build. These test results are stale in any case: they do not correspond to the newly-built code. This has the potential to confuse users as well as Pulse!

Is it difficult to modify the build to clean the test results first?

Cheers,
Jason

Binyan - 14/Sep/06 07:58 PM
Sorry for the late response, but when you say "modify the build to clean out the results", what do you mean? I ask because we run "mvn clean install" and this _would_ normally clean out the results, but if maven has a problem starting (missing plugin, badly configured pom, etc) then the build is properly flagged with an error. However, the UI will show that the tests from the previous run all passed. It is merely a UI detail which is easily diagnosed if you look at the build log and see that maven didn't start. However, PHB's may not be "sophisticated" enough to grasp this. Sorry, been reading Dilbert lately.

jason - 15/Sep/06 12:13 AM
I suppose what I was really asking is if I was missing something, which I obviously was! I hadn't considered this case (the build failing before it had a chance to clean). That certainly makes life more interesting. I will need to think more about this, but now I understand why you are suggesting timestamp checks.

Current workarounds for this include:
- using either the clean checkout or clean update scheme (preferred if your build is quick anyhow)
- using a custom/versioned build and splitting the build into multiple commands, where the artifact is captured in the last (test) command: if the test command doesn't run, the artifact won't be captured

However, we will also review options for making this work with built-in projects and incremental builds.

Binyan - 15/Sep/06 02:15 AM
Glad I finally started making sense to you. :)

Actually my builds are clean checkouts and take less than 30 seconds. It's another of the project's I'm trying Pulse with that sadly checks out hundreds of MB of data (top level project, with a lot of sub projects). This project is simply too slow to do agile, clean checkouts with scm triggers.

jason - 15/Sep/06 03:40 AM
What we will probably end up doing is adding an option to ignore "stale" artifacts. This will be off by default, but at least it gives a way to handle this situation.

Re: long builds: we added incremental building for exactly this reason. Short builds are great, but life isn't always that easy :).

Binyan - 15/Sep/06 08:02 AM
Sounds good.

jason - 05/Nov/06 01:31 PM
Added an ignore-stale option to artifacts which is false by default. Refer to change 2506.

jason - 05/Nov/06 01:39 PM
Merged to trunk in change 2507.