Sunday, October 03, 2010

Remembering a dark day ...

It must have been late June 2000, I had been in my first post university job for only a couple of weeks. The 2 main developers at the company were PhD students and desperately want to get back to their research but the current project had been eating more and more of their time until it was all they did.

In the late June 2000 timeframe we are talking about they finally declared the system good enough to launch, so it did. They immediately left the company to complete their research and I remained to maintain the system. Me, a newbie to "professional" development, new to Java and new to the rather large and complex code base they left behind.

For a couple of days things were fine. I tinkered with a few bugs and prepared the next release. That next fatal release when all of a sudden everything went wrong. Old bugs, from before I even started at the company, resurfaced. Nothing made sense and there was no one around that could help.

By about 3am I had managed to patch enough that the system was at least up if not fully functional. Over the next few days I patched bug after bug. Then one of the original developers came back, having put a decent amount of worked into his PhD. He looked over what had happened, the list of bugs that had seemingly returned from the dead and immediately realised what had happen.

You see there was no source control in place. So, to keep life simple, they had split the development into 2. While they both had a complete source to compile against, they would each only change one side of it.

This had worked through most of the development, until the day they left. When they put their source into a single folder for me, for the first time since starting. One developer put his source code in first, then the other put his in. Both his up to date correct half that he had been working on and his old incorrect copy of the other developer's half. Thus all of the bugs fixed in the previous weeks by developer 1 returned.


I recount this story because it has been in my mind of late. I have been putting a lot of effort into planning regression tests for a change I am implementing which first brought this story to mind. Also I was chatting to a developer the other day who was bemoaning bugs returning from the grave. Coincidentally I was also reminiscing with someone about horrendous problems we had on a project that did use source control, the variant of which turned out to be a costly mistake.

From my story and those of others in this area I take away 2 lessons that I try never to forget. Firstly, you must have source control, and an associated plan of how you will use it. Secondly, you cannot assume that simply having and using source control is the answer. While it may prevent the total disaster I presented above, I have seen many projects have source management issues even with source control in place. Be sure to review, on a regular basis, how you are using source control and whether it actually offers you the safety net you might assume it does.