On refactoring an hidden technical dept

Can you accumulate technical dept, even if you regularly clean up your sources meticulously? A short while ago I would have said that this is possible but unlikely. That was before I started taking on the migration of all of OpenFastTrace‘s unit tests from JUnit4 to JUnit5.

Like most non-trivial projects OFT accumulated quite a number of unit tests over the years and although it was always the plan to move to JUnit5 some day, they were all written for JUnit4.  With the arrival of OFT 2.0.0 — which also was a major refactoring endeavor — I felt now would be the perfect time to make that migration happen. After all I was in the refactoring flow.

To rule them all…

Shouldn’t take long, right? Boy was I mistaken. The one thing I completely underestimated was how dependent many of our integration tests were on JUnit rules like Stefan Birkner’s “JUnit System Rules“. While getting the dependencies right in the Maven POM was only slightly harder than anticipated, migrating all parts where rules were used took way longer. I quickly found a replacement for the “Temporary Folder API“: JUnit Pioneer’s “TempDirectory Extension“.

Homebrew extensions

I did not find a suitable replacement of the JUnit System Rules though, so I decided to take matters into my own hands and create the “JUnit5 System Extensions“. At the time of this writing they cover System.out, System.err and System.exit assertions.

It was a good if also painful practice for writing JUnit5 extensions. Only later I realized that Stefan seems to have started a port of his rules, but it did not look finished to me. Anyway this gave me the opportunity to dive deeper into the JUnit 5 life cycle.

Run baby, run baby, run baby, run!

Christoph was the one who pointed out to me, that the tests I wrote only ran in Eclipse but were silently skipped when triggered by Maven. The culprit was an older version of the Surefire plugin for Maven. Turns out that you need 2.22.1.

Simply beautiful

Even though the migration turned out to be much more of a pain than anticipated, especially integration tests and test that check for exceptions looks much nicer thanks to JUnit5. It’s about time for this upgrade and I am sure we won’t regret this when writing new tests.

Improving test coverage in log message

Log message test coverage for the java.util.logging.Logger depends on the log level by default. As an optimization the lambda functions that constitute log messages are only executed if the configured log level is higher or equal the log message level.

In effect this means that for optimum test coverage you would have to set the log level to FINEST for your unit tests. But that will spam your console or log files.

Thankfully the designers thought of that issue and provided a means to use multiple log consumers in parallel. Christoph created a NoOpLoggingHandler and added it to the list of log consumers in the logging.properties of the JUnit tests.

This makes sure all lambdas are called and we get maximum test coverage.

If you want to learn more check out the documentation of the LogManager class.

Test coverage pitfalls

When did I test enough?

While code coverage is a good indicator, you still have to know how code coverage works. Today I was hunting a bug in a Markdown importer I wrote for OpenFastTrace. The class where the bug sits had a whooping 93.1% code coverage and part of that missing coverage was owed to the fact that the JaCoCo coverage tracer can’t mark code that throws exceptions as covered – even though there is a test case for it.

So I wrote a new issue ticket, created a fix branch and wrote a three line test case that reproduced the problem. While the test now failed as expected, the code coverage didn’t go up.

I used the Mockito mocking framework to verify that an interaction with a mocked observer happended or in my case due to the bug didn’t.

The error was hiding in a regular expression which read

"Needs:\\s*(\\w+(?:,\\s*\\w+)+)"

instead of

"Needs:\\s*(\\w+(?:,\\s*\\w+)*)"

Notice the asterisk in the end.

So while there are a lot test cases waiting to be written for this regular expression, code coverage won’t tell you.

You get what you measure, so be sure you understand what it is you are measuring.