Sunday, November 22, 2015

An opinionated definition of a unit test

During the same C# code reviews that triggered last week's blog post about writing great unit tests, another discussion tends to pop-up, in particularly with new joiners (both experienced and junior):

"What's the difference between a unit test and an integration test?"

Well, in my opinion (which is heavily influenced by The Art of Unit Testing), a unit test has the following characteristics:

  • Fast (as in a couple of milliseconds)
  • Runs in-memory
  • Has no dependencies on external I/O-bound resources such as networks, configuration files, databases and such
  • Can be run an infinite number times without failing
  • Can be run in any order with the other tests
  • Doesn't cause any side-effects that might affect other tests or require manual clean-up
  • Cover one or more closely related classes

If a unit test does not meet ALL of these characteristics, it's an integration test.

But what about subcutaneous tests, automated tests that cover everything from just below the user interface layer (which is usually technologically specific) all the way to the database? The goal of such a test is to cover as much as possible without relying on front-end technologies. So the question if it's a unit test is kind of irrelevant. It's more useful to discuss whether or not to include them in a CI build of some sort. And that all depends on how much they meet the earlier mentioned criteria.

And what about tests that employ temporary in-memory databases such as Sqlite or RavenDB? Some would argue that the mere fact they rely on a database disqualifies them as a unit test. However, in our current project, we have hundreds of these tests. They run fast, cause no side-effects and don't have any dependencies. So yes, I usually treat these as unit tests as well.

Another example in the grey area of unit tests is a test that uses OWIN to test the entire pipeline from the HTTP API all the way to the database level. In most cases, we'll use the same approach as subcutaneous tests and use an embedded Sqlite or RavenDB database. Technically they meet most of the characteristics of a unit test and we always run them as part of the CI build. However, whether I call them unit or integration tests depends highly on the scope of the test. If the purpose is to just test an HTTP API without involving the other layers, I tend to call them unit tests. Which brings me to the topic of the 'unit' in unit test.

Of all the things you need to think of while writing automated tests or practicing Test Driven Development, defining the scope of the unit you're testing is the hardest one. I still think Jeremy D. Miller's Third Law of TDD, in which he advices to test small before you test big, make sense. But small doesn’t mean a single class though. Instead, I recommend you to evaluate the collaboration of a couple of closely related classes and consider which of these change together when requirements change. Those might be a good candidate for the unit in a unit test. At the same time, having a couple of end-to-end integration tests is just as useful. Just don't follow any na├»ve advice in which you need to choose between one or the other. Focusing on unit tests only might bring you in a situation were it’s difficult to refactor or redesign the internals of your codebase without breaking your test. Similarly, focusing on integration tests only will most definitely bring you into the debugger hell. Always choose the right tool for the right issue.

So what do you think? Does this definition make sense? If not, I'm very interested in hearing your thoughts. Let me know by participating in the Disqussions below. Oh and follow me at @ddoomen to get regular updates on my everlasting quest for better solutions.

Wednesday, November 11, 2015

12 tips to write unit tests that don't cripple your codebase

Over the last months, I've been involved in more and more code reviews, mostly because we've increased the level of quality required for code changes to our code base. While doing that, I started to track my most frequently used review comments intended to improve .

  1. If the dependencies of your subject are not important for a particular test, use a Test Data Builder to build the subject-under-test (SUT). However, always have at least one test that clearly show those dependencies. Hiding them increases the change your subject ends up with more dependencies then what's good for  you.

  2. Don't repeat the action your executing against the SUT in the name of the class. For instance, tests for a query handler should not start with When_executing_…. That's superfluous information that does not help to keep the purpose of the test apart.

  3. Be very conservative in applying the typical Don't Repeat Yourself (DRY) principles to unit tests. They tend to obscure the cause and effect, especially if you're moving stuff into some kind of base-class (or worse, a hierarchy of base-classes).

  4. Don't introduce constant variables and such at the top of the test, unless you're dealing with a magic number. For example, I often use literals like "TheIdentifier" and "OtherIdentifier" to set them apart and communicate intend. Even numeric values like 1123 don't need constants, even if you use them multiple times. It's quite obvious not just a number. But, numbers like 1, 0 and such, don't say much, and usually benefit from a well-named constant.

  5. Hide everything that is not important to see in that particular test. In other words, if it's not important for the test, it's very important NOT to show it. Test Data Builders or a simple With extension method can help hide the umimportant. But again, be careful to hide design errors by adding intelligence into the test data builders. That's the software equivalence of sweeping the dirt under the carpet.

  6. Ensure that the cause and effect of the test is crystal clear. Nothing is more annoying than a failing test that makes no sense at all, neither by looking at the name, nor the implementation. If you're practicing Test Driven Development well, the tests can serve as documentation for your API. So make sure your tests read like a book.

  7. If the test fails, ensure that the failure message is so clear that no debugging sessions are necessary. Using a proper assertion library is not a luxury anymore. One note though. If you love FakeItEasy like I do, don't use constructions like this:

    A.CallTo(() => commandService.Execute(A.That.Matches(cmd =>
        (cmd.Site == new SiteName(equipment.Owner.ToString())) &
        (cmd.Identity == equipment.Code))))
        .MustHaveHappened(); 


    If the Site and Identity properties of the command contain anything but the expected values, you'll need a debugger to figure out what happened.

  8. Don't stick to a single style. I generally make a distinction between tests that do simple state testing and those that are more orchastration oriented. For the former, I prefer the AAA syntax, whereas the latter is more suitable for a BDD-style test, e.g. using Chill or something else. But be consistent. Don't mix more than two styles in a single code-base (or git repository).

  9. Whether to test small or test big is a topic that is a source for heated debates. I see value in both of them, so depending on the thing I'm building you might see me start of with an end-to-end integration test and then drill down into the code base to add more fine grained unit tests. Reverting this process is perfectly fine as well. However, if you don't start off with such an end-to-end test, you might discover that you didn't design for it, making it diffuclt to add it later on.

  10. If you need to verify that a certain string value or exception message matches the expectation, never verify the exact message. Use wildcards to verify the specific parts that are relevant to verify that the behavior meets the expectation. It will save from unexpectingly failing tests if you decide to refine the text involved.

  11. Don't refer to technical names such as the names of methods, variables and UI elements in the name of a test. In my opinion, the name of a test should explain the functional scenario, whereas the cause and effect should clearly illustrate the technicalities of that scenario. If some class or member is renamed, it shouldn't affect the functional name of the test.

  12. Be careful with libraries like AutoFixture. They have value, but using those for everything increases the chance your test starts to fail on something that is irrelevant for that specific test. Moreover, they usually have a negative impact on performance. And again, if the value of certain object is important to understand the test scenario, it is VERY important to show that.
As usual, these opinions are mine, so feel free to comment below. Oh and follow me at @ddoomen to get regular updates on my everlasting quest for better solutions.