Should unit tests check for error conditions?

Should a unit test incorporate tests for error conditions?  For example, what if the database connection fails?  Remember that the line by line debug testing should have tested all exception and error handling code at least once.  So, if the developer does line by line testing for all of the code, including error handlers, then there should be at least some confidence that the error handling code has been tested and works.  The problem with that approach is that it is not verifiable and the word of the developer would have to be taken.  For most organizations, this is usually adequate. 

For organizations that require 100% verified unit test coverage, then developers will have to come up with ways to make the software fail in order to prove that the error handling code works properly.  I question the cost / benefit ratio of including unit tests that test error handlers.  The problem is that it is at least difficult and sometimes close to impossible to test out error conditions without modifying the code or the environment.  If the code or environment is changed, then it can be argued that the actual production code is no longer being tested.  The only value of testing error handlers is that the actual error handling code can be verified as correct.

For organizations that insist on 100% code coverage, the easiest approach to testing the error handling code might be to include a test function within the class that when called throws the appropriate error.  The problem with this is that the code has to be modified to call this function.  Perhaps a better way would be to have the developer think of a way to make the code fail without having to modify any code or the environment.  For example, if an exception is thrown when opening a file that does not exist, then it would be fairly easy to pass in a file name that does not exist or is locked for exclusive access.  But, there are other cases that are not so easy and coming up with a real test to cause the software to fail is problematical, time-consuming, and costly.

Prev          Next          Back to write the unit tests

Copyright 2011 by Robert G. Bryan

Advertisements

3 Responses to Should unit tests check for error conditions?

  1. Robert J. Engdahl says:

    You must design for testability, otherwise you can be certain that unit testing becomes too expensive. Both when it comes to normal control flow and exceptional control flow. Unit-tests are not supposed to test a unit of code in its full-blown environment, but just the unit in isolation. Unit tests should show that the unit will behave as you expect if the environment does so.

    When I speak of design for testability, you should make it possible to insert test shunts for provoking the desired control flow.

    I have yet to come across a programming problem that only has solutions which cannot be unit-tested. That being said, I have seen plenty of code that was not written to be tested.

    • Bob Bryan says:

      You said “Unit-tests are not supposed to test a unit of code in its full-blown environment, but just the unit in isolation. Unit tests should show that the unit will behave as you expect if the environment does so.” So, I believe that one of the top priorities of a unit test is to maximize its usefulness as a regression test. My experience has shown that writing the test to maximize its usefulness as a regression test is very valuable. The problem with doing stand alone unit testing with mocked objects is that it tests the code from a point of view of how the tester believes the system should work and not necessarily how the system actually works. It also will generate many false positives later on when changes are made to the code and is not a very reliable regression test. In some complex environments, your options are more limited and stand alone unit testing with mocked objects might be one of the few ways to do testing. This is especially true if quite a bit of effort has already been spent into testing with this approach as you probably don’t want to change course midway through a project. But, for other environments this is simply not the case. If a developer has written and tested a library that I am calling into with my code, then I am going to include his actual library in my unit test if reasonably possible. If it is going to fail, then I want it to fail early. I guess the question is – why not do early integration testing? Of course, there are some exceptions – libraries of code using specialized hardware, version issues, takes 5 hours to complete, etc. would not be suitable candidates for early unit integration testing.

      • Robert J. Engdahl says:

        What I meant was that it is worth it to use mocks for enabling the unit testing of otherwise unreachable control flows. And it becomes affordable when you design for it.

        The top priority of any unit test is to maximize its own usefulness. So if you can test anything in addition to the unit itself with little or no overhead, go ahead. We should always be pragmatic.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s