Hi!
We are using EMMA as code coverage tool in our build.
We noticed that the coverage results are really unsteady. Sometimes, classes are covered, but in the next build, they aren't covered (without changing anything in the code).
Is there anything we can do in order to get stable results from EMMA?
Regards,
Markus Bischof
Coverage can be unstable because some code is not always working the same way. For example, there can be code which tries to allocate some resource and if allocation fails, tries again. In some build you may have successful resource allocation from the first attempt, while in other build - it is not. In these cases coverage can differ.
Also coverage can differ if there is some platform specific code.
Hi Pavel,
thanks for your reply.
I understand that code coverage differs in some cases. But that's not really my problem.
I can give you an example: TeamCity tells me, that a testcase XY was successful. This testcase uses an instance of class A. But the EMMA coverage report tells me, that class A is not covered (class-coverage). And some buils later, the EMMA coverage report tells me, that the class is covered...that's really confusing...
Markus
Probably this is a bug in EMMA. In fact we are moving to use IDEA coverage engine, because EMMA is not supported anymore by its authors. IDEA coverage engine is available starting with TeamCity 5.0. So if possible, I would recommend to upgrade to TC 5.0.
Right now I'm running into OutOfMemoryErros when using EMMA...let's switch to IDEA coverage! ![]()
Actually, I tried to use IDEA code coverage, but I get the following log error:
Publishing artifacts
Do I have to install additional packages/plugins to get that to work? Or do I miss some configuration parameters? Perhaps I have to put coverage.ic into my artifacs?
(I'm using TeamCity 5.0.2)
Regards,
Markus
No, you don't need to install anything. Could you please check is there a coverage-error.log file in the working directory on the agent? Also please provide your build log. If it contains sensitive information you can send it to teamcity-feedback@jetbrains.com
Hi Pavel,
hm, I would prefer not to send my build log (it really contains sensitive data). But I can provide the part of my ANT-Buildfile which is used in the run-configuration (I've attachted it).
In my build log, I see the following line right when the runTests-target is called:
[runTests] IDEA Coverage: Java agent added to JUnit task
But after the tests did run, I still get the error:
...
[07:29:26]: [antcall] clean
What include / exclude patterns do you use?
First, I used the default "*" as include pattern and none as exclude pattern.
Then I tried to use one of my package names as include pattern, which really contains classes: "company.project.*"
As exclude pattern, I used the package name which contained the tests: "company.project.unitTests.*"
Unfortunately, I had the same error result.
Do you run your tests in separate JVM (is fork enabled)? If not please try to enable fork.
Great! ![]()
Using junit with fork="yes" solved the issue!
Thank you very much for your help.
Congratulations! I think we need to add some note on the web UI about fork requirement.