Standalone tests or integrated tests?

Background

Sometime back in the 1980s the C programming language started to become more popular and widespread.  Some issues with how it was used also started to appear in print.  One issue that most agreed was a bad practice constituted using global variables.  A global variable is a variable that can be accessed anywhere within a program.  One function can set a global variable and any other function inside that program can read it and/or also update it.

Recommended testing procedures started to appear in print, including unit testing.  Along with never using global variables, it was recommended that each function in a program be tested independently or as a standalone test.  That was all fine and well back then and is still applicable today for developers working with C (not C++) as long as they are not using global variables.

Some of the benefits of using a standalone test is that if it fails, it is easy to identify where it failed, and is a very logical place to place tests for detailed function behavior.  Another terrific benefit is that the standalone test can act as a regression test which can identify any defects introduced in the future by a developer making changes to that function.

Standalone unit tests in object-oriented environment

However, today many if not most developers are using an object-oriented language like C++, C#, and Java to name a few.  An OO language provides the capability to use variables that are global to the entire class, including all methods of a class.  What that means is that OO methods are not self-contained like they were under C.  Instead, OO methods can reference and update class variables that have usually been set by other methods of the class.  So, if a standalone test is written to test an OO method, then it has little to no value as a regression test.  Why?  Lets look at an example.

  class ThisClass
  {
    ...
    public void HighLevel1()
    {
      ...

      LowLevel10();
    }
    public void HighLevel2()
    {
      ...

      LowLevel10();
    }
    public void LowLevel10()
    {
      ...
    }
  }

In the above code, HighLevel1 and HighLevel2 are both defined to call LowLevel10 at some point. LowLevel10 is a method that has 10 subfunctions and both the inputs and outputs to this method use class variables.  HighLevel1 only calls LowLevel10 for subfunctions 1 – 5.  HighLevel2 only calls LowLevel10 for subfunctions 6 – 10. 

A standalone unit test is written to test out LowLevel10’s 10 subfunctions.  Standalone unit tests are also written to test out HighLevel1 and HighLevel2.  Since LowLevel10’s 10 subfunctions have already been tested in its own unit test, only subfunctions 1 and 6 are tested by the unit tests for HighLevel1 and HighLevel2, respectively.

So, assume that the developer initially writes all of the unit test code, tests it, and it passes all of the unit tests and he checks the code into the repository.  A few weeks or months later his manager asks him to make a few changes to support a new feature.  He decides that no changes will need to be made to LowLevel10, but some code changes will be required in both HighLevel1 and HighLevel2.  He makes the changes, but without realizing it, introduces a defect by using a class variable used within LowLevel10 for subfunctions 2 – 4 and 7 – 9.  He runs the unit tests again to make sure that all the tests pass.  What do you think – do the tests pass, or do they fail at this point?

The tests pass!  Why?  Recall that all of the subfunctions are tested by the unit test for LowLevel10.  That unit test is responsible for setting up the test for each of those subfunctions and since no changes were made in LowLevel10, then no changes were required in that unit test.  In HighLevel1 and HighLevel2, only subfunctions 1 and 6, respectively, are tested.  Thus, the code passes the tests since those subfunctions are unaffected by the changes.  The code is checked in and no one is the wiser that a defect has been introduced into the system.  Thus, standalone unit tests become almost useless as regression tests in an OO environment.

Integrated unit tests in object-oriented environment

What happens if an integrated unit test is added for HighLevel1 and HighLevel2?  HighLevel1, which uses subfunctions 1 – 5 of LowLevel10 would now incorporate tests for all 5 subfunctions.  Likewise, the integrated unit test for HighLevel2 would incorporate tests for subfunctions 6 – 10.  So, all of LowLevel10’s functionality is now completely tested by HighLevel1 and HighLevel2.  So, what about the standalone unit tests for LowLevel10?  Should they also be written?  Clearly, the answer is no.  The unit tests for LowLevel10 do not test for anything different than the integrated unit tests for HighLevel1 and HighLevel2, so they do not add any value and are thus wasteful and redundant.

So, with integrated unit tests, when the developer goes to add some changes to HighLevel1 and HighLevel2 in the future, and introduces that same defect, what happens when he runs the integrated unit tests?  Do the tests pass, or do they fail?  The tests fail this time!  They did their job and identified a defect in the code that was just changed because the test was in context of how it will be used in the production environment, and not some canned test that makes some assumptions on how it is “supposed to be used.”

If a method that is tested at a higher level only has a portion of its functionality tested by the higher level tests, then it should have its own separate test to exercise the functionality not tested by the higher level tests.  This is especially true if the method is declared public and can be called outside of the class.

Conclusion

Standalone unit tests are great for C developers that are not using global variables.  For object-oriented developers or C programmers using global variables, integrated unit tests should be used in place of standalone unit tests in order to test the code in context of how it will be used in the production environment.  If an integrated test thoroughly tests out a lower level method, then a separate standalone test for that method should not be required since it offers nothing different from the higher level tests and is thus wasteful and redundant.

Prev          Next          Back to write the unit tests

Copyright 2011 by Robert G. Bryan

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s