TLDR; Unit Testing and Code Coverage can lead to emotive discussions. These are personal lessons learned around Unit Testing and TDD. Keep things simple. Keep refactoring. View Test and Main as symbiotic.
I’ve been doing a lot of programming work recently and wrote down some of my lessons learned around unit testing.
Some general concepts in the below description:
- ‘execution’ coverage – coverage achieved when a line of code is used in passing e.g. setup, but not directly related to the Assertions or concerns of the Test class.
Should I have one Test class per Main class?
- This can make the Unit tests easy to find.
- If they are class focussed tests and the class is simple, leading to few tests, then this can make reviewing the coverage easier as their are fewer Test classes to review.
- For coverage reporting, the ‘package’ is more important than ‘one Test class per Main class’
Should I only have one assertion per Test?
- For very simple classes that can work fairly well.
- For more complex classes that rule might mean having more Test classes, with a common set of
@Beforesetup, and then small
@Testmethods which only
- I don’t use that as a guideline, it feels artificial.
- A guideline I do use is to avoid asserting in the middle of tests. When I review the tests that I have created with Assertions in the middle it is often because I am unaware that that condition has been asserted in another
@Testor because I have missing
- Summary: I do use multiple assertions. I try to avoid them in the middle of the @Test class.
Note: this is a widely debated topic and people have strong feelings about it. Some people will enforce and mandate one Assertion per
@Testmethod. That feels too restrictive for me.
Should my package structure for Unit tests match my main code?
- When you ‘run with coverage’, then the coverage reports for the tests report on the coverage of the Main classes which are in the same package as the Test classes run.
- if you put your Test classes in a package unrelated to the Main class they are testing then the coverage report may not show any classes covered – by default.
- the “Edit Run Configuration Dialog” shows the ‘packages and classes to include in coverage data’, by default this is the package of The tests you are running. You can amend this to include more classes. If this has nothing selected, then all packages are in scope and you can see all classes that are exercised during the test execution.
- I use this as my default because by reviewing the package structures I can see if I have any major gaps in my testing.
- Writing tests at the same level of the Main code means that your Test code is going to be tightly focussed on the responsibilities of a small number fo Main classes, making the Test creation and maintenance simpler.
- Sometimes I find that reviewing tests, and coverage causes me to restructure my Main code to make coverage easier to assess and understand.
It is too easy to view Main code as ‘gospel’ which Test code must ‘work around’ because Main code can exist without Test code. But when taken together as ‘Production’ code, the Main code and Test code have a more helpful relationship. When I find Test code hard to write, then changing the Main can result in ‘better’ Main code (easier to understand, maintain, extend and reliably build on).
Should I Unit Test My Test Code?
- Yes. When we write ‘support’ code for our Test code, e.g. a DSL, Data Generation, Reporting Classes.
- Why? Because it can save a lot of time debugging test failures e.g. Random data generation might cause a test failure if it is not generated correctly. Comparison routines for complex data might case a test failure if not Unit tested effectively.
- No. When we are talking about abstraction layers for ‘Page Objects’ or ‘API calls’ – avoid testing ‘external interfacing’ Test code.
- Why? Because then we would end up mocking Selenium interacting with our application or Rest Assured interacting with our API. That is too much to maintain. Rely on the results of execution for these classes.
- No. When we are talking about
@Testmethods or classes.
Are code coverage reports useful?
- When we control the coverage scope, to include only a few classes we are interested in. Then run the tests that are targeted at the functionality of those classes. Then they are very useful because we can see any non-covered code.
- There is a risk that future changes impact this code, and we don’t have any tests that will report any unexpected changes to this code.
- If this code is exercised by other tests, and you consider the code simple enough that ‘execution’ rather than ‘test’ coverage is a good enough safety net then you may decide not to add additional tests. (I prefer not to take this approach and instead write Test code to directly cover the methods in the class rather than rely on ‘execution’ coverage.)
- If you leave the code uncovered by tests, then there is a risk that you have to perform low level code coverage reviews more regularly to ensure the uncovered risk status is the same. Sometimes I push my code coverage in low level tests to 100% so that they are no longer ‘flagged’ during my review processes and focus on the important non-covered areas.
Can I have too many tests?
- Yes. If all your tests are covering the same thing then this can increase maintenance effort.
- If you have many tests that are very similar, consider adding some data driven or parameterised testing.
Are there any common code smells?
- Too much setup code in tests can mean they haven’t been reviewed and refactored enough.
- Consider moving setup code to
Before...annotation classes e.g.
- For complicated setup consider creating Test Objects which can be re-used by many tests.
- Warning. Keep these ‘Test Objects’ at the lowest level of a package hierarchy as much as possible. If you are sharing the same ‘Test Object’ in many Test classes in many different packages then you may not have architected your classes to be testable enough.
- Consider moving setup code to
e.g. when I started working on a hobby project I had ‘case study’ tests. These use a common set of data – the Case Study – and cover a lot of ground. This allowed me to move fast and explore different implementation approaches in my prototype. When making the code suitable for production, these Case Study tests were useful in supporting refactoring to better architected code. But I needed to avoid the temptation of using the Case Study Test Objects in the new Unit Tests, because this makes the Unit Tests hard to maintain, and tightly couples the low level Units, with the wider system concerns. I wrote new Unit Tests for the new code, and expanded code coverage at a Unit Level, without using the Test Objects.
Is 100% code coverage useful?
- If 100% code coverage is achieved through ‘execution coverage’, where high level operations are performed in the code so that a lot of code is executed in passing, but few results are checked, then no, it probably isn’t useful.
- You don’t need 100% code coverage to gain confidence that your tests will pick up many issues.
- If you decide not to have 100% code coverage from low level class based tests then consider if your class really needs those methods and lines of code that you are not testing.
- If you decide not to have 100% code coverage from low level class based tests because the code conditions are hard to test, then consider rewriting your class to make it easier to test.
- Considering the usefulness of 100% coverage, for a specific set of code, is useful because we have to to think about the risks associated with both ‘hitting it’ and ‘not hitting it’. It is possible to achieve 100% coverage, and obtain no useful change detection information when the tests are executed. The 100% number, by itself, is not useful.
How much code coverage do I need?
- No generic number as a target will help you.
- Numbers do not provide confidence, understanding what is behind that number, and how that number is achieved will help you.
- A low number tells you that there are a lot of gaps, and that you should have less confidence in the ongoing stability of your code.
- A high number, without understanding, should not fill you with confidence, it should fill you with questions: What does that number mean? What are we asserting on? Is that ‘tested’ or ‘executed’ coverage?
e.g. I have a set of 15 Test Classes in one package. When executed they achieve 65% line coverage of the core domain classes of my project. This does not fill me with confidence. I know that much of this is ‘execution’ coverage, rather than ‘test’ coverage, because of the style of testes in that package. But… I don’t rely on that one package. The number is not relevant, the understanding and decision making behind the number is important.
How can I use code coverage to help me review my tests?
- When you run a set of tests, edit the ‘run configuration’ to only cover the code you want to review coverage for.
- Review the tests for gaps, i.e. I don’t have a test that cover invalid condition X, if you already have a high % coverage, then you might find you haven’t coded for this condition. (you don’t actually need code coverage to help with this, but it can be a useful extra step)
- Review the coverage for gaps. For any gaps consider:
- Is this code hard to test?
- Perhaps the code should be simplified and refactored?
- Is the uncovered code, so simple that it can’t ever go wrong?
- Sometimes I add a test anyway: to ensure that it stays simple, and so that I don’t get distracted by it in future reviews.
- Is this code hard to test?
What are the benefits of TDD?
- You create coverage as you create code. This saves time in reviews looking for gaps.
- It saves future debugging time.
- Your confidence about the code grows with the coverage.
What are the risks of TDD?
- The risk of creating code that is only used by the Test.
- I primarily encounter this situation when I have made my Main code too complicated, and hard to test. I find that splitting my Main code into smaller classed to make them easier to test often removes this situation. This is why the refactoring stage of TDD is important, and why ongoing code reviews and refactoring from different levels of abstraction is important.
- The risk of adding methods into ‘exercised’ code, which are not covered by tests.
- This more likely happens if you are not mocking classes. I tend not to mock classes so I have to be wary when adding code to other classes which support the class I’m TDD’ing. When this happens, I try to go back and review all the code created, and revisit the code coverage for @Test classes in the surrounding classes.
Edit Run Configuration dialog in IntelliJ
- ‘Packages and classes to include/exclude’ are very useful when reviewing Test Coverage. Only including the classes that I’m interested makes my review more accurate and I can more quickly identify gaps in coverage.
- When I have ‘test’ abstraction classes. I often tick ‘Enable coverage in test folders’ to help me gauge the ‘risky’ elements of my abstraction layers. Because I want to write Unit tests for any Abstraction layers that are not directly engaging with external systems. e.g. I don’t write Unit Tests for ‘Page Objects’ or classes that are composed of calls to Selenium or Rest Assured. But I do want to have Unit Tests for my domain abstractions, logging, and data generation.
<strong><a href="https://www.eviltester.com/ebooktop6">Read our free ebook</a> on the Software Testing books you Must Read (and Why)</strong>
Unit Testing FAQs and Lessons Learned