Almost never run all of the acceptance tests

Acceptance tests are tests written by the QA team or by the customer and usually automated to independently show that the software works as intended.  There can be anywhere between 50 to 1000+ individual acceptance tests, depending on the size of the project and dedication of the QA team.  An individual acceptance test can take anywhere from 3 times to 1000+ times longer to run than an individual unit test.

In many agile environments, the developers are supposed to run all of the acceptance tests before checking the code in.  The idea behind this is that running all of the acceptance tests before checking in has a very good chance of identifying any new bugs introduced by the changes that were last submitted since the acceptance tests can be thought of as the ultimate regression test.

This argument sounds very nice in theory, but in practice what happens is that as the number and complexity of the acceptance tests grow, the longer it takes to run them.  The Extreme Programming project that I worked on in Chicago at one point had so many acceptance tests that it was taking over an hour to run.

The reason why developers should usually not run the acceptance tests before checking in is because it impacts too negatively on developer productivity.  For example, if the acceptance tests take 1 hour to run; there are 10 developers on a team; the developers typically check code out in the morning and do a build which involves running the tests; and then check the code in at the end of the day, then they will run the tests twice a day.  So, the total number of man hours to run the acceptance tests can be calculated as 10 * 2 * 1 hour = 20 man hours per day.

Requiring developers to spend 25% of their day to run acceptance tests is very inefficient and unproductive.  It also would not make much sense to run all of the acceptance tests if a one-line cosmetic change is made to a user interface component that has nothing to do with any of the other components in the system.

A developer’s most valuable utilization is in designing and coding of the software.  Some testing also needs to be done.  Line by line testing, unit code testing, and interface testing should be the extent of the developer’s testing.  This strikes a nice balance between design/coding chores and testing.  Making the developers also run the acceptance tests is counterproductive in that it turns developers more into testers and reduces the time they have available to design and code.  Requiring the developers to run the acceptance tests will cause the project to be much less productive than it could otherwise be.

A very important concept for developer productivity is how quickly can a component be compiled, linked, and tested?  If this can be done in 5 minutes or less, then that is a fairly fast turn around time that can yield highly productive developers.  Compare this to an environment that requires the developers to run the acceptance tests as well.  If it takes an hour to run these tests, then the turn around time has gone from 5 minutes to 1 hour and five minutes – a 13 fold increase.  Which environment do you think developers would be more productive in – the 5 minute build and test or the 65 minute build and test?

So, only the unit tests and the interface tests, at the developer’s discretion, should be run.  The code should only be checked in after the tests passed with no errors.

Prev          Next          Back to efficient software development methodology practices

Copyright 2011 by Robert G. Bryan

One Response to Almost never run all of the acceptance tests

  1. all three says:

    Easily, the post is actually the greatest on this deserving topic. I agree with your conclusions and will thirstily look forward to your coming updates. . . . .

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s