Everyone's been to the fair, right? The state fair? The county fair? A travelling carnival? Chuck E. Cheese's? If so, you've more than likely seen (if not played) some variant of the popular game colloquially known to me as "Whack-a-Mole" or by its more obscure moniker "Mole Whack-Em".

For those who haven't enjoyed this passtime, the rules are simple: artificial mole-shaped puppets pop randomly out of roughly mole-sized holes cut into a game board. The player, holding a mallet, tries to strike (or "whack") the artificial moles as quickly as possible since their appearance is timed and, if left unwhacked, they will disappear into the dark belly of the game board whence they came.

A simple game concept, and great fun for all.

Now, consider a version of "Whack-a-Mole" in which the mole-shaped puppets pop up, but do not recede if they are not struck.
Additionally, consider that our hypothetical version of this game has an infinitely large (or at least some very large) domain of potential mole-shaped puppets.
Finally, assume that the game board is free to grow as you play, allowing for an indefinite number of visible mole-shaped puppets at any given time, all waiting for you to clobber them with your mallet.

This game appears to be a bit more complicated, but still kind of fun. An average person would almost certainly recognize the futility of playing once the number of visible (un-whacked) mole-shaped puppets exceeded the number he or she could comfortably squash before the next round appeared but, for a while, it would entertain.

Taking one last step to set up this analogy, let us assume that each mole-shaped puppet is equipped with a device that will deliver a moderate electrical shock to the player; not enough to kill - just enough to wound. The remaining visible mole-shaped puppets on the game board will continue to deliver their shock to the player until they are sent below by the mallet. Oh, and there are multiple mallets - a given mole-shaped puppet might not respond to every mallet...in fact, you may find a puppet that requires you to construct an entirely new mallet just to crush it.

Now turn off the lights.

Deconstructing this analogy, let us assume the following: 

• The game board represents one or more software applications.
• The mole-shaped puppets are either software configuration errors, or bugs in the underlying source code of an application.
• The mallet(s) are software debugging skills (e.g. server profilers, compilers, network trace utilities, integrated development environments etc.)
• The electric shocks are the increasing pressure from management to resolve the problems, regardless of their relative solubility.
• The lights represent the level of understanding available to the developer/troubleshooter about the application(s) experiencing trouble. "Lights off" represent a system into which the minimum amount of visibility is allowed. For instance, application logic written and compiled using a language that has passed into the realm of arcana and cannot be reproduced as source code, hence cannot be read. This is known as a "black box" application.
  

Such has been my lot for the last several weeks (nay, months).

I am a software developer with years and years (and years) of experience. I like to write software. I even like to debug software and test software. I do not like playing tedious games with negative rewards unless they also involve cake.

I work for a company (name omitted to protect the unrepresented) where the analogy above accurately describes the transition from our active development environment (people like me, writing application code) to our quality assurance (QA) environment (where testing is supposed to occur prior to making applications publicly usable).

This company has no records of what applications came before the ones they are currently trying to deploy from the development environment into their QA environment, and so they have no idea what the impact of the newly deployed applications will be, but they insist on installing the new files into the same folder(s) as the old ones.

All information about previously installed applications has either left the company or has passed into the realm of tribal knowledge. There is no source code to be found in our version control system (even if there was code, it would be in a language no-one has used in years, and which I have never used).

All previous application code is compiled. That is to say it is converted from nice, text, human-readable format to decidedly less friendly byte code that looks like '@#$% SAF #@$% $#%' in even the best text editor.

In many ways it's like a cargo cult. The constituents of QA know that they have some set of HTTP addresses they require in order to test whatever it is they test. If any of those addresses is unavailable, or changes, they will assert without reserve that all hell is going to break loose - fire will rain from the sky, swarms of locusts will devour crops, Dick Cheney will develop a conscience - but nobody can say what artifacts make the application work. If an application stops working, the most likely reaction from anyone who regularly uses it will be "Hey! Someone let the magic smoke out of this..."

It is into such an environment that I have been boxed for the last many weeks. Problems arise; I laboriously track down what I think is the problem, make notes, make configuration changes with extreme trepidation, then move on to the next problem if, that is, I do not cause further grief in the act of making my change.

I'm tired of whacking moles. Very, very tired but we haven't reached what my friends in Jersey would refer to as "the ass-kicka"...

This entire exercise will be repeated, from the beginning, with no guaranty of any reusable solutions, when the company chooses to deploy the new application code to their production (customer-facing) servers. The QA environment is known to be wildly divergent from the production environment, and last week I learned that nobody at the company is capable of reinstalling a production server from the ground up, meaning that no catalog of what goes "into" a production server exists.

The long and short, then, is that ultimately I will be back here writing an analogy in which "Whack-a-Mole" is integrated with "Groundhog Day".

Who has thoughts? A better analogy ("hell" has already been suggested and rejected, by the way).