Lesser Known Test Processes

New post on the Pathfinder site. I had fun writing this one. It's on some lesser known variants of test-driven development.

http://blogs.pathf.com/agileajax/2008/02/lesser-known-te.html

Two Pathfinder Blog Posts

Two things on the Pathfinder blog.

• Agile Publishing, on publishing experiences and agile methods.

• Live Ruby: Testbed, an attempt to work through a small test and metaprogramming problem live and on the blog.

Enjoy.s

Boring Software Manifesto

Another blog post up on the Pathfinder site. This one is about Agile in general, and features what I think is the World Blog Premiere of the soon to be famous Boring Software Manifesto.

Enjoy.

More On Test-Driven Development

My first post to one of Pathfinder's official blogs is up, it's a companion piece to the blog post here on Test-Driven Development, and you can find it here.

An Agile Musing

Of course, since I muse in an agile way, I reserve the right to change my mind based on future developments...

Software development usually takes place in a complex environment where your goal can change quickly. In general, there are two ways to deal with a complex environment. One is to try to anticipate, in advance, every possible permutation you might need to deal with, and the other is to manage your environment with the flexibility to respond to new challenges with minimum effort. Software is just a specific case where that challenge plays out.

Of course, being software engineers, we've given these approaches names. They are nicely defined by James Bach:

agile methodology: a system of methods designed to minimize the cost of change, especially in a context where important facts emerge late in a project, or where we are obliged to adapt to important uncontrolled factors.

A non-agile methodology, by comparison, is one that seeks to achieve efficiency by anticipating, controlling, or eliminating variables so as to eliminate the need for changes and associated costs of changing.

It seems worth pointing out that, at least in theory, there's no reason why you can't both make a reasonable effort to find potential changes up front, and still make your working environment as flexible as possible.

In practice, of course, this is kind of difficult to do well. There's a logistics problem -- the kinds of things you do when you're trying to control change up front generally involve creating a lot of models and documentation and the like. This is exactly the kind of thing that works against minimizing the cost of change. But it seems to me that a group determined to do some up front analysis of what will largely be an agile process could manage to work around that.

The bigger problem is an issue of mindset -- if you think that you can solve all your design problems up front, then you start to look at any change to the design or implementation with suspicion. At that point, any change is, by definition, a mistake of some kind. Which leads to suspicion of all changes and can mean all kinds of tracking and procedural overhead that pushes up the cost of change. The sort of organic, bottom-up design that you get from test-first and tight iterations doesn't fit in this model at all.

Although I should say that in my experience trying to get groups to adopt XP or agile practices over several different companies is that nearly all programmers and managers agree that automated tests are a good thing. Most programmers are at least willing to consider the idea of test-first, although getting somebody to do it consistently is tough (hey, it's tough even for somebody who's totally bought into it).

The resistance I do get tends to be around the idea that you can start programming with an incomplete design without it leading to disaster.

Waterfall-style up front design can be incredibly seductive. You're brainstorming with other smart people, you're solving all your problems before they even come up. It certainly feels like you are doing something vitally important to the success of your project. Even if you know that many of the decisions will later be revised in implementation, it still feels good to have that crisp UML diagram. You can't have bugs in a UML diagram. The issues that do get solved in design are assumed to justify the time cost of the design, because the assumption is that the time cost of later improvements will be much larger (which is true, in part, because of the amount of design work).

In contrast, starting coding with incomplete information can feel risky, especially the further you get from the actual implementation team. Any issue that gets changed later can be blamed on the relative lack of design, true or not, and the time savings from not doing as much design can be invisible. It's very hard to let go of the idea that all your design problems can be solved up front. But once you are willing to allow that some problems can only be solved in the moment, you're much better equipped to deal with the inevitability of change.

from internet import *

Three posts that caught my eye today.

Ruby School

Gregory Brown over on O'Reilly net has an article about using Ruby in Computer Science courses, at least in later algorithm classes. It's not a bad argument, but I think it'd be more convincing if the Ruby example was a little cleaner and easier to read compared to the pseudo-code.

Let's see... The last time I had to care about this issue was about eight years ago when my grad institution was going through a somewhat controversial revamp of the CS curriculum. The fight, as always, is between the theorists and the pragmatists. The theorists want to teach a lot of "pure" CS up front -- Turing machines, big "O" analysis, computational theory, that kind of thing. The pragmatists want the students to be able to get jobs.

You should know that I spent the better part of three years as part of a group working with an object-oriented class that we taught in Squeak Smalltalk. Lovely language, to be sure, but we had to spend part of every course explaining to some nervous students why we weren't teaching them C++ or Java...

At the time, the initial CS classes were moving to Java, with some relief. This is because a) nobody wanted to inflict C or C++ on unsuspecting new CS majors, and b) the previous most common language, Pascal, was woefully obsolete. Java is reasonably straightforward to teach and is actually used in real programs, both high points.

Personally, I think you can make a pretty nice case for a scripting language like Python or Ruby in the initial CS class. They are both pretty easy to get started with, the syntax is clean enough that algorithms are easy to visualize (which was Brown's original point). In Python you can do it without introducing objects (which most CS1 classes didn't do eight years ago, don't know if that's changed). In Ruby it's easy to teach meta-programming.

Cha-Ching

Paul Julius of ThoughtWorks about how CruiseControl can save you $12,535 per broken test. The money coming from the difference between the cost of fixing a bug immediately versus not catching the bug until integration testing.

I dunno. I love continuous integration, and would shout about it from the rooftops if they'd let me on the roof, and that number still sounds a bit more "look what I can do with numbers" than "look what I can do with Continuous Integration". But then I'm skeptical of nearly every numerical analysis of programming productivity.

Plus, Marshmallows

Over at Roughly Drafted, Daniel Eran goes on about the smooth, harmonious relationship between Apple and Sun. Naturally, I want to talk about one of his sidebars...

The name of Apple's Mac OS X frameworks was even named Cocoa in part to associate it with Sun's Java. The other reason was that Apple already owned the Cocoa trademark, having using it earlier for a children's programing environment.

You know, I've always wondered about that. The original Cocoa was a project that was being worked on in Apple's Advanced Technology Group the summer I interned there, plus it some buzz in Educational Technology circles for a while. Internally, it was called KidSim, but the name was changed to Cocoa when it was being prepared for release. Java was programming for grown-ups, so Cocoa was programming for kids. It seems like Apple isn't really using that connotation of the name anymore.

The project (now called Stagecast Creator) is a graphical rule-based programming language, something like a cellular automata program. The user specifies an initial arraignment of sprites on the screen, then specifies how that arrangement should change in the new time slice. Complex programs could be created with almost no typing (although, like all such programs, you still had to use drawing tools to create your own sprites -- that was still hard). Stagecast still seems to be around, although it's been ages since I tried the software. It was pretty cool, though.

Tips-First for Test-First

Of all the exciting ideas and revelations that came from Kent Beck's original XP book, Test-First Programming has been the one that most significantly affected the way I work on a day-to-day basis.

I love programming test-first. It's a great way to take a large, amorphous task and solve it piece by piece. It's also a nice morale boost -- "Hey, I know that my code does nine things. Let's go for ten..."

Here are a bunch of things I wish somebody had told me about test-first programming.

Unit Testing is not All Testing

So, after I started doing test-first, I walked around for about six months all, "my code is perfect because I wrote tests". My smugness came crashing down when testers found some bugs. My code was better because I wrote tests, but turns out I had made some dicey assumptions about the inputs, and so my tests passed, but were still incorrect. Test-first is not a complete test suite. You still need to do acceptance testing, you still need to do GUI testing where appropriate.

However, you can still automate a very large percentage of acceptance tests. The more you can automate the tests, the more they'll be run, and the happier you'll be.

Test-First is a structure for writing good code at least as much as it's a means for verifying code

Code that has been written in a test-first style tends to have certain qualities. Small methods, loosely coupled. Small objects, loosely coupled. Code that causes side effects (such as output) tends to be separated from code that doesn't. These are all side effects of what's easy to test -- it's easy to write small methods in a tight test-first loop. And dependencies between methods or objects make tests harder to write.

As it happens, those exact qualities -- tight cohesion and loose coupling -- are exactly what characterizes the best software architectures. My test-first experience is that I wind up with much better code architectures from test-first then I do when I try to guess the design before I start. (Which is not to say that a little bit of pre-design can't be helpful, just that it can be overdone).

Test-first is better suited for some things than others

That doesn't mean that you shouldn't try, of course. Test-first is vital in cases where you know the input and output, but not the process. It's also critical in cases where your program can be incorrect in subtle ways. It's somewhat less important for things that will visibly or loudly break. GUI's are a challenge because GUI layouts tend to change in ways that can break unit tests. GUI behaviors are more stable and easier to test. Again, though, you should try and have automated coverage of even those areas that weren't developed test-first.

Trust the process. Look ahead on tests, not implementation.

It works. The tight process is: Write a test. Run the test so that it fails. Make the simplest fix that will pass the test. Run the test so that it passes. Refactor. Keep to that tight loop. Resist the temptation to guess about what you'll need to pass the next test. What I usually do is put a list of the tests I'm going to write in comments in my test class -- that's my lookahead plan, and keeps me from forgetting something. But the design I do in my actual code comes during the refactor step, which is where I see duplication and abstraction.

The earlier you start the better off you are.

It gets increasingly hard to convert a code base to test-first the longer you wait. I've even had 20 line classes that needed significant refactoring to unit test (mostly because output was intertwined with functionality -- the code was better after the refactoring). Test-first is a good place to start, anyway -- pick something to test, and go.

Treat your tests like code and refactor them

Pretty much every test-first guide gives sort of a perfunctory nod to, "oh yeah, keep your test code clean." But I think this could stand a little more attention. For one thing, your unit tests are critically important to your ability to deliver quality code -- and they have no tests of their own. The cleaner your tests are, the better you'll be able to see issues with the tests themselves.

One thing that has worked nicely for me is extracting sets of assertions into custom assert methods. If you are continually making the same five assertions to check validity of your objects, throw them into an assert_instance method of some kind. Another common case is making the same assertion over a range of values -- move the for loop to your custom assertion and pass the range endpoints in.

There are two big advantages to doing this consistently. The first is that it's easier to see what's going on from one line of assert_person(expected_name, expected_addr) then from five lines of assert_equals. The second is that it ensures that you actually do make all the assertions every time. Hey, everybody slacks, and test-first is about making the test setups as quickly as possible. If you can trigger all umpteen tests on your class with one method call, you're more likely to do the whole set every time, rather than just picking one or two at random each time.

Don't reuse instance variables

This is another refactoring issue. It's tempting as you add new unit test cases to do something like this:

Person p = new Person("noel", "david", "rappin");
assertEquals(15, p.nameLength());
p.setLastName("flintstone");
assertEquals(19, p.nameLength());
p.setFirstName("pebbles")
assertEquals(22, p.nameLength());
p.setFirstName("betty");
assertEquals(22, p.nameLength());

The last test fails -- quick, what's it testing? Okay, now we have to trace the life of that instance variable all the way back up. It's hard to read, and prone to dangerous errors. You should never reuse an instance variable like this in a unit test -- every assertion should, where its at all feasible, be completely distinct:

assertNameLength(int expected, String first, String middle, String last) {
  Person p = new Person(first, middle, last);
  assertEquals(expected, p.nameLength())
};

testNameLength() {
  assertNameLength(15, "noel", "david", "rappin");
  assertNameLength(19, "noel", "david", "flintstone);
  assertNameLength(22, "pebbles", "david", "flintstone");
  assertNameLength(22, "betty", "david", "flintstone");
}

Now when the last test fails, you can actually see what's going on.

Avoid tautologies

The scariest issue you can have with tests is a test that passes when it should fail, allowing you to continue blithely along, ignorant of a bug you should have already caught. There will come a day, for instance, when you will forget to put any assertions in a test. There are a couple of things you can do to make tautologies less likely.

1. Follow the process. The process says each test has to fail before you add code. Adding tests that you already know will pass can easily lead to writing a test that will never fail.
   
   
2. If you have constants for text or numerical values in your code, don't reuse those in the tests -- use the literal or create a separate constant in the test.
   
   
3. Be careful with Mock Objects. Try not to test the things that you are explicitly inserting in the Mock when it's created.
   

Mock Objects Rule

Mock Objects are the missing link in helping you test all the things that are traditionally hard to unit test, like databases, GUI, web server... anything where your code is dependent on an external system or person, the Mock can get in the way and pretend to be that third-party and allow you to send and receive data in a testable way. Mock Object packages exist for a variety of languages, and using a package will save you time and effort on your tests.

Hope that helps -- go out and test something.