Who will watch the watchers?

This kind of question is usually uttered by people in regards to authorities such as the police, intelligence agencies, governments and the like. Because, according to them, authority is bad and can’t be trusted. The problem is, it’s not so much a rational question, but a demonstration of a paradox. Tree falling in the woods kind of thing, you know?

This post is about a couple of test driven development practices, let me present them to you with their rationale:

Workflow: Red Green Refactor

The point of this is you write a broken test–run the test, make sure it’s broken. so you know the test is broken, this is very useful when you type something like assert instead of assert_equal. Kent Beck relates a story of how him and a team spent hours trying to get a test to pass, when it was the test itself that was broken. the flip side of this is that you write a test that passes when it should fail. this leads to madness, believe me.

So after you’ve gone from Red to Green, you’re free to turn that ugly implementation into something pretty–and be sure you haven’t broken it during the process, all well and good. We love refactoring, now for the other practice:

You should refactor your tests, too

You should be treating your test code with as much love and due care as you do the actual implementation. Removing duplication and stuff. I must admit it took me a while to start thinking like this, but eventually you really do start to care that your tests are nice and easy to read. I mean, that whole communicating intent kind of thing. Test should do that, so yeah.

The paradox

Here it is. We write tests to allow us to refactor our implementation safely–who or what what is going to test our tests when we refactor those? Who will watch the watchers for sure (or who will test the tests in this case).

I bring this up because I don’t actually have a good answer. We have tools like Heckle which do a good job picking up your blind spots, but does that solve this problem in particular? Is there some other practice I’ve missed? Do we strive to write good tests straight up? Does it even matter?

It’s all so confusing and arbitrary. But that’s how we like it, yeah? Nah?! Aw.

Said on September 03, 2007.

COMMENTS

Ryan,
I understand your perspective.

As the BDD people point out, your tests are really a spec for the behaviour of your code. Whenever you are refactoring the tests you should be making them more expressive and easier to read. The easier they are to read the harder they are to get wrong.

Personally I need to see a test fail to know it is testing what I think it is, so when in doubt I comment out or modify the code and watch the test fail.

Nigel

said Nigel Thorne on September 03, 2007

Thanks for the comment! I suppose it is a bit of a matter of the granularity of your tests.

I find it helpful to develop complex model code with unit tests but if you then move on to write an integration test later down the line it seems a bit redundant, however useful.

I’ve heard of people using Selenium exclusively as their test suite, foregoing all other language / framework level tests… If you did that you could completely swap out the back end…

said Ryan Allen on September 03, 2007

I am a big believer in using UATs. I believe a feature isn’t complete until there are application level tests written in a language the customer can understand.

Selenium gives you half the answer. To go all the way you need to generate the selenium tests from another source. Something that can be kept DRY and still express everything in terms the customer can read.

However… UATs are for the customer. They ensure the whole app is working. Unit tests give me the confidence that my classes are working so I don’t waste my time debugging a complex system as a whole.

Rails sometimes gets in the way when trying to write testable code. Maybe camping is simpler. I’ll have to spend some more time with it.

I don’t tend to write what I would call integration tests (testing how two objects work together). I just rely on the UATs for that. Rails doesn’t let me get away with that as much as I would like.

said Nigel Thorne on September 03, 2007

Hi,
Well, much of the focus these days are placed on getting the “product out”. Less focus is placed on Testing. At least that’s the trend I’ve seen and experienced myself! But I am not staying is right :-) Having quality tests matter.

We use a mix of JUnit, TestNG and Selenium integrated with Maven2 for our testing.

Shamelle
Enhance Life

said Shamelle on December 19, 2007

hey ryan,

found this page through your flashden profile when i was looking for a flash animation. good to see that flashden is going strong.

question: what is this blog running on? it seems very simple and basic which is what i’m after at the moment. the wordpress ones i have tried using are hard for me to integrate seamlessly with certain page designs–at least for me cos i’m not so good at modifying code without breaking
everything.

cousin patrick!

said pat on January 12, 2008

Yeah, I like your blog. I have wondered about this conundrum myself. Maybe our tests should be our first love because our actual implementations if well behaved and not too rebellious look up to them and follow suite ;-) Us humans we naturally test almost everything before we trust it, tests included.
Its a little antagonistic because as rebellious people put the law to the test implementations will put your tests to the test. maybe nah, hey im an indecisive aussie too!

said brodaigh on February 01, 2008

yeahnah.org