Conventional wisdom that fails for IT

I know of very few aphorisms that tend to be repeated as smugly as this one, particularly by scared people. The implication is that action is generally to be avoided, that the status quo is probably just fine, and that one should wait for a true crisis before intervening. And, of course, that it’s your fault if you’ve ignored this sage advice and intervened anyway. It’s ironic, then, how IT departments themselves end up complaining endlessly about how they’re always in fire-fighting mode.  This prevailing attitude evolves among (and is a telling symptom of) burned-out sysadmins and developers, especially those who are stuck maintaining systems they didn’t themselves write or engineer. It can be equally summed up as a “don’t touch it, don’t breathe on it” kind of superstition. Or, perhaps, it’s akin to the proud but defensive statement that “we’ve always done it that way.”

A short anecdotal example: I visited my company’s colocation facility a few months after I took on the role of their first CTO.  I was immediately horrified, and realized that I should have made the trip much sooner. Cables dangled everywhere around the cluttered cage, and often went nowhere; systems were powered up but unconnected to anything; other systems were unlabeled and unknown as to function or importance.  Systems that I knew for certain were no longer operational in terms of our software architecture were still powered up, blinking away fiercely, mysteriously doing something.  One could easily see how an overwhelmed, undercompetent staff might increasingly adopt the “if it ain’t broke, don’t fix it” mentality, just to save their skins and keep the wolves at bay.

As with so many things, that situation represented a management failure too. It reflected a willingness, whether explicit or implicit, to live on borrowed time, hoping to stave off as long as possible the certain-to-come outage that would then take much longer to resolve.  It showed a willingness to tolerate unnecessary inefficiency and risk. It embodied an ongoing refusal to insist on (and prioritize) the necessary hard work to keep the clutter out of the equation.

On the software development side, the philosophy of “if it ain’t broke, don’t fix it” allows similar cruft to accumulate in software. Elements that should be eliminated or refactored as architectures have expanded over time simply get ignored and retained. Again, cables go everywhere and nowhere, metaphorically; it’s just a lot less visible than it is in a server cage. The risks entered into and the productivity drag caused by these shoddy practices build up a sort of soggy mulch pile, over time, that essentially freezes a dev group in its tracks.

So don’t get too cavalier about changing things (I’m also a firm believer in change control), but also don’t succumb to the complacency reflected in this aphorism.

The people who toss off this old chestnut also often smile triumphantly as if it were both unanswerable and as if they themselves had just invented the clever saying. The aphorism embodies a belief that only a single individual, making all the decisions, can do an effective design.  Note that aside from its humor, the saying doesn’t even make logical sense: a thoroughbred wouldn’t last long in the desert, while a camel is of course a highly optimized creature for its environment.  In addition, people generally apply the aphorism widely, refusing to acknowledge the usefulness of group involvement altogether, in anything. They trot out extreme examples where consensus-gathering has paralyzed action.

For information technology, the usefulness of insisting on the primacy of the individual, as an approach to making key decisions on systems-in-the-large, actually runs counter to my practical experience of what works.  An individual operating in a vacuum, even if extremely brilliant, informed, and motivated, tends to have occasional or frequent biases, tunnel vision, and pride of ownership. He misses errors and issues that the scrutiny of multiple eyeballs, not to mention the careful discussion of pros and cons, can easily catch.

An example of the usefulness of committees is the Project Portfolio Management (PPM) process I’ve described frequently here on this blog.  Having a sole individual, even the CEO, decide on project inclusion simply isn’t viable over the long run in many corporate cultures–it creates classic problems of lack of buy-in and participation, for example. On the other hand, instituting a suitably chartered and well-facilitated steering committee, composed of senior individuals from the major business areas of the company, forces everyone to put on their “big company hat” as they consider priorities, rather than doggedly insisting on their own department’s parochial perspective. When that’s done well, everyone moves forward with a common understanding and solid commitment, one that’s much less likely when there’s an on-high fiat from a single person.

As for software design? Well, Wikipedia’s article on design by committee claims that “Often, when software is designed by a committee, the original motivation, specifications and technical criteria take a backseat and poor choices may be made merely to appease the egos of several individual committee members.” That can admittedly occur, but I’ve also seen poor software designed by talented individuals, who can’t understand that others may not find particular interface elements or features to be as intuitive as they do. Committee involvement in design is no panacea for that problem, but there’s also no reason to eliminate the usefulness of multiple sources of feedback.  As with all things, balance is key. Don’t just adopt a philosophy of “consensus” no matter what. As with a PPM steering committee, you should put effort into designing an effective committee and process, and allow for quick “tie-breaking” authority and action as needed.