One of my favourite bugs

So a few years ago, well, probably more like 15, I worked as an Senior Engineer at a translation software company. Our whole stack was Microsoft Windows based and we wrote desktop applications, not particularly sexy but they were pretty good (if I do say so myself) - we were making efforts to 'move to the Cloud' (we referred to this internally as The Clown, because really, nobody knew what we were doing back then and it resembled a bit of a circus at times).

We had a team of probably 50-ish people, split into your typical core platform teams, product teams and some people to do things like build scripts and installers, fairly standard set up in those days (and I guess still so today). Life was good, we wrote some cool software that did some clever stuff, and we were a nice sociable team. But something was lurking and it was terrifying us all...

There was a period of time over a few months in 2008 where we strongly believed that there was a computer virus doing the rounds in the office, it would strike seemingly at random, but it would only infect developers, it seemed that our testers, PO's and Scrum Masters had some sort of immunity. Totally bizarre and unexplainable.

So, back to that software stack that we had. Our builds produced a lot of exes and dlls - this was I think before the days of nuget so the way that we handled our dependencies between teams was by versioned zip files (I did say before the days of nuget right :)), which we would deliver to other teams on a schedule, and they would update their dependencies and pass the baton on - I say baton, it wasn't a baton, we had a big black rubber 'baton shaped item' that we physically passed between teams, and depending on whose desk it was on, then they were the ones currently updating their dependency versions before passing it onto the next team. It was effective, and it was funny, it made people move fast because they didn't want this thing on their desks for long, but you probably shouldn't do that in today's world!

We had a giant build script written in PowerShell - PowerShell was pretty new at the time and we were just learning. This script was responsible for firstly cleaning up your old workspace by traversing up to the build output directory and deleting all of the files it found there - this made sure that no old dependencies or files were hanging around and then pulling each of the dependencies in turn, and then finally compiling the product on top, so in just 30 minutes or so you had a fresh build ready to work with. And somewhere in that sentence above is the bug, can you spot it?

Well, it is specifically the part where it says "deleting all the files", and this was our virus. Unbeknown to us, (because who code reviewed build scripts in 2008?), the build script had a feature whereby it would keep stepping up a directory level looking for the debug and release build directories so that it could clean them up. This was fine providing you started the script in the correct directory, but if you started it anywhere else, or above the place where it was meant to start, it would keep stepping up and keep stepping up until it arrived at the root of your drive, in the case of Windows, C:, and once there, what did it do??? It decided it couldn't go any further, it must be in the right place by now, so this would be a good place to start recursively deleting files and folders, THE HORROR!

The first time sign someone had been infected was the virus checker on the machine starting to go mental and flash messages up, and they started to yell - this is one of the reasons we thought it was a virus, not the yelling, the virus checker part. When the script had eventually worked its way into the Program Files directory and System directory, the virus checker knew something was up because something was trying to delete it!

Each time this virus struck the machine was totally trashed and it would then take IT several days to rebuild it - it probably destroyed 10-15 developer boxes over the few months it was in our system, and each time we literally had no idea what was going on until we had two in very quick succession and both developers revealed that they had been trying to do a build. It took all of 5 minutes to locate and fix the problem in the build script, but this was of course after it had caused probably a few hundred of hours of lost development time, kept our IT department gainfully employed and embarrassed our build engineer - although to be fair had anyone else reviewed their work we might have avoided all this!

Moral of the story, do your code reviews no matter what - whilst we don't ship things like build scripts they too are an integral part of your products and should be given the same fine toothed inspections as the rest of your codebase.