.Hibri

The major highlight at XP Day 2009, was Mark Striebeck’s talk on unit testing practices at Google.  What makes a good test depends on experience , skill and school of thought. I had to agree when he said that developers can be almost religious when it comes to the topic of what makes a good test. This made them solve this problem the Google way, by gathering data. Let the data speak.

He went on to describe metrics that they were collecting on tests and test code.  A test that has never failed is likely to be a bad test. If the test was fixed to make the test pass, then this is also an attribute of a bad test. A test can be a good test if the code was fixed to make the test pass.

This got me thinking. Generally I haven’t gathered metrics on test code. We have a pretty good metrics dashboard for production code. What metrics can I gather on test code ?

Metrics on test code should also focus on the readability of the code. Having large test methods is ok, but not too big. My opinion is that a test method with more than 20 lines is too big.

Tests should be concise, the assert should be obvious. Some code duplication is fine to make the test readable. This is all fine, but how can I get these as metrics  ? Only way to judge this is to eyeball the tests, and there are differences of opinion.

However, there are ways to measure what a test should not be. These are test smells. Test smells are described in xUnit Test Patterns

I’ve listed a few test smells and NDepend CQL queries find these smells. These can be automated in the build process and flagged up.

Large Test Methods

These can be a chore to read. Tests should be written as simply as possible. These also  point to too many responsibilities and dependencies in the code being tested, as most of the test code is used to do setup for the test.

SELECT METHODS WHERE HasAttribute "NUnit.Framework.TestAttribute" AND  NbLinesOfCode > 20

Large setup methods

Usually when unit testing the same code, we tend to have a common setup method, in order to make the test more readable. What happens is, more and more code is moved into the common setup method. We get blind to this after a while, and all the dependencies for the test are hidden away. If you do have [Setup] methods, keep them small.

SELECT METHODS WHERE HasAttribute "NUnit.Framework.SetUpAttribute" AND NbLinesOfCode  > 10

Deep inheritance trees in test fixtures

Again, common test code moved up to a base class and the base class is used in many tests. Then more base classes are created. This creates more tight coupling between test classes. Which makes tests harder to change. Low coupling and high cohesion applies to test code as well. Make each unit test class as independent as possible.

SELECT TYPES WHERE HasAttribute "NUnit.Framework.TestFixtureAttribute"  AND DepthOfInheritance >2

Test fixture setup

TestFixtureSetup is bad. The TestFixtureSetup is run once before all tests. This leads to fragile tests and inadvertently leads to using some shared state. Use Setup instead

SELECT METHODS WHERE HasAttribute "NUnit.Framework.TestFixtureSetUpAttribute"

Tests that fail when they are run in a different order

The xUnit test runner helps with this, by randomizing the order tests are run.

Ignored tests

Ignored tests are like comments. Dead code that doesn’t do anything. Either fix them or delete them.

I have yet to find some way of detecting duplicated tests, shared state in tests and multiple asserts in tests. What other ways can I find test smells ?

When you are in the zone, and want to add a reference to Rhino, NUnit or any other common reference, adding it via the Add Reference dialog can be painfully slow. Fortunately VS has a good automation interface, which lets you to write macros.

I wrote a simple macro to add a NUnit reference to the current project. Add this to your Macros project in Visual Studio, map a button or a shortcut key to it. This way you can add those common references pretty quick.

This is a cleaned up version of the sample here http://msdn.microsoft.com/en-us/library/vslangproj80.reference3%28VS.80%29.aspx

   1:      Sub AddNUnitReference()

   2:          AddNewReference(DTE, "C:\Tools\NUnit\nunit.framework.dll")

   3:      End Sub

   4:   

   5:      Sub AddNewReference(ByVal dte As DTE2, ByVal referencePath As String)

   6:          Dim aProject As Project

   7:          Dim aVSProject As VSProject

   8:   

   9:          aProject = dte.ActiveDocument.ProjectItem.ContainingProject

  10:   

  11:          aVSProject = CType(dte.ActiveDocument.ProjectItem.ContainingProject.Object, VSProject)

  12:          ' Add an Assembly reference and display its type and additional

  13:          ' information.

  14:          Dim newRef As Reference

  15:          newRef = aVSProject.References.Add(referencePath)

  16:   

  17:      End Sub

I like the idea of running unit tests continuously as I type and save code.  I read

about Autobuild .Net recently. This is easy to setup and use.

Grab the  code from http://code.google.com/p/autobuildtool/  build it by running the default.build script.

When this is built successfully, the build process creates the tool in the output directory.

Copy the contents of this output directory to the project where you want autobuild to work on.

Autobuild .Net runs a nant build script each time a file is saved in a specific path being watched.

Change the autobuild.build script with the path to the solution file and the path to the unit test assembly.

Next in autobuild.cmd , change the first argument to the directory to watch for changes. The second argument is for the path to autobuild.build. This is the nant build script that autobuild will execute. You can replace this with an existing Nant build script.

Run atuobuild.cmd . That's it. Type some code, save it and watch the window go red because it didn’t compile. Make it compile and watch it go green.

Write a failing test and see it go red, make it pass and see it go green. Fun eh ? :)