Previous: Design of PUB/SUB subsystem in ØMQ
I hear that prople feel an uncontrollable urge to write unit tests nowaydays.
If you are one of those affected, spare few minutes and consider these reasons for NOT writing unit tests:
1.
Return on investment of unit tests is an order(s) of magnitude lower than that of end-to-end tests.
If you write a single end-to-end test the execution of the test will probably cover a substantial part of the codebase:
If you write a single unit test, you cover just a small piece of the codebase:
If you are painting a house, you want to start with a biggest brush at hand and spare the tiny brush for the end to deal with fine details.
If you begin your QA work with unit tests, you are essentially trying to paint entire house using the finest chinese calligraphy brush.
2.
End-to-end tests test the critical path. Unit test do not.
End-to-end tests typically simulate real-world usage scenarios. Thus, after running end-to-end test you have reasonable level of confidence that the product is going to work for the actual user.
If all you have are unit tests, you are pretty sure that all the individual gears inside the project work as expected. However, you have no idea whether the project as a whole works or not. It may well be that the user won't be able to performs a single task.
Yes, unit tests are rigorous and make sure that the component will work even in corner cases. However, user wants the product to work in common cases in the first place. If it fails in common cases it's not a product. It's a failure.
On the other hand, if the product fails in exotic cases that happen rarely or never at all, the defect can be tolerated and possibly fixed later on.
3.
Unit tests ossify the internal architecture.
Imagine you have three components, A, B and C. You have written extensive unit test suite to test them.
Later on you decide to refactor the architecture so that functionality of B will be split among A and C. you now have two new components with diferent interfaces.
All the unit tests are suddenly rendered useless. Some test code may be reused but all in all the entire test suite has to be rewritten.
This way the unit test suite makes the product resistent to internal change. A programmer with limited time to allocate on tasks will consider the refactoring, then consider the cost of rewriting the test suite and place to whole endeavour into the "not worth it" mental slot.
I have seen a nice definition of "architecture" somewhere. It said that architecture is that what makes software able to change. If we embrace this point of view, unit tests can be considered to be a strictly an anti-architecture device.
4.
There are things that can't be unit-tested.
Consider a protocol decoder. The protocol says "9th byte in the packet is called TLL".
The implementation does this:
int ttl = packet [8];
How are you supposed to unit test that? You can create a fake packet that has value 123 as 9th byte and then check that the decoder extracts TTL of 123. How is that different from testing that 1 == 1 though?
Protocol layout is a definition, not an algorithm and there's little to test there. What works though is interoperability testing: Take two implementations of the same protocol and check whether they speak each to another. And once again, we end up with end-to-end tests instead of unit tests.
5.
Some stuff has no rigorous acceptance criteria.
Some of the code we write can be described in almost mathematical rigour: lists, hashmaps, 3D graphics et c.
Other code is not meant to be defined so rigorously. It's meant to feel good to the user. The most obvious example is GUI.
In such cases there's no right or wrong behaviour, there's only good or bad behaviour. And good or bad is much harder to define and test.
Admittedly, end-to-end tests and unit tests face the same problem here.
However, it's likely that the project has a high-level description of how the end user experience should look like. You can use that as basis for the tests.
Not so with individual components. After all, what does "end user experience" even mean for an internal component? In such case you should think of components as malleable pieces of code whose sole purpose it to enable globally (end-to-end) defined user experience. If the end-to-end experience is OK, the component is OK as well.
6.
Unit tests are easy to measure and you should fear that.
If you are working in a hierarchically structured company, beware!
Progress on project has to be reported up the command chain and with a mainly spiritual activity such as software development it's pretty hard to find any hard metrics. How the hell are we supposed to measure QA?
And there's an easy solution to that: Report number and/or code coverage of unit tests!
But that leads into a trap: Once you start reporting code coverage of unit tests, you'll be immediately pressured to improve the metric, with the goal of achieving 100% coverage, despite all the problems with unit testing outlined above. The problems, after all, are not quantifiable, but the code coverage is.
It's hard to say how to fight the scenario above. Maybe keeping the unit test coverage so embarassingly low that nobody even thinks of reporting it to their superiors would help…
7.
All that being said, unit tests are great for testing complex algorithmical tasks with strictly defined behaviour and lot of corner cases. Don't even think about implementing a red/black tree without an extensive suite of unit tests.
However, be frank: How often do you implement red/black trees?
And more generally: Is there really a rational justification for all the unit tests you are writing?
Think about it for a minute and you may spare yourself a lot of useless work.
EDIT: There seems to be some confusion about what is "unit test" and what is "end-to-end test". In the context of this article, "end-to-end" test means test that uses external interface of the product (one that is visible to the end user). "Unit test", on the other hand, means test that uses internal interfaces within the product (those which are not visible to the end user).
Martin Sústrik, Jun 4th, 2014
Previous: Design of PUB/SUB subsystem in ØMQ
This is a false dichotomy. You don't choose between unit tests and e2e tests, you write them both. And they serve rather different purposes. Unit tests are most helpful while you are still developing your components, and can not test the system as a whole. By definition e2e tests can be run only after the system is near completion, and waiting until that point is a recipe for disaster.
I think the most important value derived from unit tests is having a quick feedback cycle. You write code, build it, and immediately get results. If your have nothing but e2e tests, it often takes much longer to verify that the piece you are working on works correctly.
Also, most people overlook this benefit of having good unit tests - easier refactoring. If your unit tests verify the interface of your component, you can confidently refactor and move stuff around - as long as all your unit tests passed, you can be reasonably confident that you have not broken anything. Of course, e2e test will still have to be run, unit tests are not meant to replace them. They are a tool in your arsenal that helps you write better code quickly and efficiently.
Of course it's a false dichotomy :)
It just seems that people today are overemphasising unit testing and ignoring e2e tests. I felt like a reminder of importance of e2e tests would be worth it.
As for refactoring, I don't agree. You can change the internal implementation of the component, but that's not what I call refactoring. On the other hand, you can't change the component's interaface (refactoring proper) because that would break all the unit tests.
I have a library in charge of writing video to disk. The entire internal mechanism of the library changed (from fopen() to mmap()) but the API stayed the same. I suspect 90% of the LOC were changed in the whole library…
The unit tests were still valid because the API didn't change. Having the tests in place made me feel confident to replace the guts in a weekend. This was a huge benefit.
"The unit tests were still valid because the API didn't change. Having the tests in place made me feel confident to replace the guts in a weekend. This was a huge benefit."
See, the thing is that you're in a situation where the API is already ossified—you can't change its API very much because you'd break existing clients. Unit tests work great in that situation.
Where they don't work so great is when you still don't have clients and you're trying to come up with a good API. You spend a lot of effort writing a bunch of unit tests… and then you realize your API, which still doesn't have any code using it, is poorly organized. Now you have to throw out all of the existing unit tests.
@Tony: It's external API (the one exposed to the user) that stayed the same, right? That's what end-to-end tests are supposed to test. Unit tests test internal APIs (invisible to the end user).
@Luis: There's in interesting paradox involved here: Unit tests work great once the project is nearing the maintenance phase. There's no much change going on, so the test are never broken by rafactoring or such. At the same time, however, the product in the maintenance phase is already pretty stable and there's not much incentive to write unit tests.
Thanks for the good thoughtful blog post. I like your analogies with the paint brushes and the gears. I've seen the most value come out of what we call automated functional tests, which is another name for end-to-end tests.
Thanks for the kind words!
Arguably e2e tests are more behavior driven. What you’re after is the result, not necessarily good maintainable code. E2e does not necessitate “better code” by enforcing modularity etc. but it could potentially foster much quicker development for a skunkworks dev studio where often fast churn-out of quick creative/marketing apps is praised over long term maintainability.
It also questionable whether there's a real need to have maintainable test code. The application code should be maintainable. Yes. But tests? I would rather keep the test as simple possible so that they can be thrown away without much regret when the application is refactored for better maintainability.
The problem that I frequently run into with teams that have a large suite of e2e tests is that they are brittle and are frequently broken and become untrusted. These teams are out of balance.
Please remember that your tests form a portfolio. This portfolio of test boundaries needs to balanced based on multiple factors; engineer experience & professionalism level, technology and economic impact of a defect to name a few. Also remember to rebalance as the factors change.
Nicely put!
I spent some time trying to address points 3 and 4: http://akkartik.name/post/tracing-tests
You are making an incorrect assumption about the scope of a unit test. The slew of this type of post recently would seem to suggest that many people are defining a unit as too small. Unit tests should be written from the public API of the context that you are working in. If this is not a critical path through your system, then you have got the public API wrong. For example, there is nothing "wrong" with a unit test hitting the database, as long as you have suitable isolation.
If unit test uses public interface of the product how does it differ from e2e test then?
e2e tests would be across contexts. A better debate might be what constitutes a context. This can vary by project. In a typical web app, there would probably be contexts for the UI and the server, and it might be split by functionality or bounded context (DDD). UI unit tests would mock calls to the server as they are not part of the context. Server tests would be at the http level. If you have separate contexts on the server you would want to test the contracts between them. Also, architectural influences may introduce contexts, for example, if you use hexagonal architecture/ports and adapters/DDD, the domain may be considered a context. This is typically because you desire isolation from 3rd party components that may change over time, so your public API would be at the port (ports and adapters) and you would have test implementations of your adapter layers. e2e tests would then be using real implementations. So, in short, public API is influenced by product, technical and architectural concerns.
How is the server API part of the public interface? Public interface is what the end user is using, i.e. the GUI in the case of web app.
I'm talking about the public API of the particular defined context. The end user of a context might not be a customer/user, that is usually only the case with the UI. There are many influences that introduce contexts, I listed some above. DDD bounded contexts are a good example, the public API's are typically event contracts.
So, afaiu, if i have an app that uses a library, the library API is one context, app's API is another context.
So we can have one unit test for the library and another unit test for the app.
End-to-end test is a test that test both the library API and the app's API in parallel.
If that's the definition of end-to-end, I don't see much point in writing e2e tests at all.
Contexts are a business concern. A library would usually be an implementation detail of a particular context, and changing implementation details will not affect tests. In the case of hexagonal architecture/ports and adapters you are making an explicit business decision that you want to de-couple that component from your core domain.
A more concrete example would be a UI, Product, Order and Billing context. Unit tests would be:
UI - Given known response from Product context, does UI display products correctly
Product context - Given a request for products, are the correct products returned
UI - When clicking submit, does the UI send the correct request to the Order context
Order context - Given an order request, is the correct order placed event raised
Billing - Given order placed event is received, is the correct amount invoiced
e2e test would be the whole flow through these.
This is just an example of how you might choose to define your contexts/public interfaces. You could decide that the app is one context, but then you should test everything through the UI. Also, you shouldn't be constrained into thinking that the public API is what the "customer" sees. Architectural quality factors play an important part and testability is an important factor that may lead you to split contexts. A common example of this is splitting a processing pipeline into intermediary outputs. It's all a trade off.
Also, the idea of testing at the public interface is an ideal. Sometimes you might need to be pragmatic and place tests around implementation details, but you should always be aware these tests are brittle and you should be prepared to throw them away.
Yep, I was ranting against testing private interfaces. I think we are in accord here.
TDD is fascism.
While I usually write unit tests (for parts where it’s worth to), point 3 is exactly the reason why I don’t subscribe to TDD’s “tests first” approach at all.
During the creative process I frequently change the interfaces (simply because I’m a mere mortal and I occasionaly overlook some dependencies, or other issues). If I had to write test cases first and rewrite them again and again any time I change my mind about the code structure, I’d fall into despair :) It makes much more sense to write tests once the APIs settle down a bit.
Perhaps it very much depends on one’s personal coding style? Or maybe TDD works best for simple pieces of code, where you can decide upon the interface upfront?
Frankly, I don't know. I've never tried TDD myself, so it's hard to say. My feeling though is that TDD may be useful in the cases where the problem is well understood (i.e. when you are writing your 5th system doing basically the same task). In such cases it may be possible to get the unit boundaries right immediately and write the unit tests accordingly.
TDD is religion… or bible, to be exact
It's intent is to provide guide when times are trying, e.g. "I’d fall into despair :)"…
Bibles are collections of fun stories to read, BTW.
:)
Excellent point 3 ("Unit tests ossify the internal architecture").
I work in research and I always avoided them to let the software easier/faster
to modify in the future (but I do very rigorous end to end tests at some point of the lifetime of a project).
I even avoid to cast interfaces in stone for the same reason.
Also, I know of some languages where the need for unit tests is quite low: OCaml and Haskell
are good examples.
Post preview:
Close preview