Business impact and transparency: expressing system availability

“System availability was 99.83% last month!  That’s up from 99.75% the previous month!”

Sounds kind of good, no? I mean, that’s a high number, right? Right?

Actually, no.  It’s not a very useful number, in and of itself. In fact, I regard the publication of uptime metrics like that as a regrettable symptom of IT focusing on technical aspects, rather than business impacts.  Here’s a discussion of why I see it that way, followed by a presentation of an alternative focus providing much more business value.

So, what’s wrong with a time-honored metric like “the system was 99.83% available”?

• The number is deceptive. Few people can mentally translate “99.83% availability” into a more meaningful real number, such as “system was down for 1.3 hours last month.”  Even fewer can tell you the real difference between 99.3% uptime (also sounds pretty good, right?) and 99.8% uptime.  Both 99.3% and 99.8% look (to the vast majority of business people) at first glance like pretty good numbers for uptime, but the first represents more than three times the number of “down hours” of the second.

• The number is ill-defined. It begs the question of what’s considered down: what if the system is just somewhat degraded in performance? Who decides that it’s down? Who declares that it’s back up? We’ve all seen situations where a technician will insist the system is up, even though no one is able to access it or get sufficient performance to actually use it.

• The number is often not comprehensive in what it includes. Most notably, many shops don’t count scheduled maintenance as downtime when calculating their uptime metric.  When you don’t count scheduled maintenance, you’re ignoring potentially many hours of real business impact per month. Scheduled maintenance can’t be a “free pass” for downtime.

• The number doesn’t compare apples to apples. Just tracking total raw downtime considers all outages to be the same, no matter what the time or situation.  A ten-minute outage at noon may impact your business much more than an hour-long outage at 3 a.m.  An outage that occurs right when you’re running a key campaign to drive people to your site can be especially devastating.

Alternatives

Somewhat better than the “99.83% uptime” style of metric is reporting on the specific number of system DOWN hours or minutes in the period.  It’s when the system is down that there’s business impact, so why not use that as your yardstick, instead of the reverse.  Some call these “impact hours”, in fact.  Stating that the system was unavailable for 20 minutes last week is much clearer (albeit sometimes more distressing!) to most business people, compared to saying it was 99.8% available.  Still, though, expressing the figure as down hours doesn’t address the other problems listed above.

The underlying insight here is that raw outage statistics, whether they’re expressed as an uptime percentage or as downtime hours, are nothing but a proxy for business impact. And not a very good one, for the reasons discussed.

Here’s a radical idea: the goal isn’t simply to report and then reduce your downtime per se.  Instead, you want to assess, promulgate, and then work down, the total business impact of your outages, and to do that effectively, you need to weight your outages by time of day and traffic. And you need to include every single outage in your assessment of business impact, including maintenance.

What’s the clearest expression of business impact?  In a nutshell: DOLLARS. How much did a given outage cost the company, in terms of missed sales and wasted expenses? Yes, there are a lot of factors and assumptions involved in figuring that out, but that’s what models are for.  Make some assumptions and build a model that will calculate the specific dollar cost for an outage, based on information about the outage’s duration, site traffic pattern at the time of the outage, etc.

We did exactly this at an internet site where the revenue stream was on the order of a million and a half dollars a week.  We incorporated considerations such as the following into our model:

• Upon analysis, we realized that any outage resulted in missed subscriptions, site advertising, and ancillary signups for partner services from which the company derived revenue. We needed the model to incorporate an understanding of the patterns and dollar costs for each of these.
• We also realized that an outage also meant that we’d wasted the money spent on external internet advertising during that outage that was driving people to our site. During an outage, external ads of course kept appearing for our site; people would click on those ads and land on our outage page. Ergo, wasted advertising dollars.
• Outages at different times (hour of day, plus even day of week) were vastly different to our site in terms of business impact and cost. Specific time of the outage needed to be a key part of the calculations of business impact.
• Outage impact also depended on whether it was a high-traffic day or not.  A high-traffic day might feature more than ten times the normal volume of traffic and transactions; an outage at that point would therefore arguably cost us ten times as much.
• Not all outages were total: sometimes performance was degraded but access was still possible. We decided to incorporate a “% degradation” parameter in the model for an outage, recognizing that deciding on that percentage would be a judgment call for any outage.
• As noted, most sites don’t count scheduled maintenance in downtime. But that’s still lost revenue. Assessing business impact requires both conservatism and full transparency. We decided to include all outages, scheduled or otherwise, when we published our outage business impact metrics.

Building the model required not just some adept spreadsheet skills, but the up-front accumulation (and regular updating) of various base data to drive the calculations.  Specifically, for each hour and day of week, we had to:

• Collect and average our aggregated historical traffic data
• Collect and average aggregated sales revenue numbers
• Determine how to calculate probable advertising revenue

Once built, our model not only let us state our downtime in terms of true business impact (i.e., total dollars for a given period), but it also provided a ready way of analyzing, up front, the specific cost of a planned outage. Using that as a tool, management could much more effectively assess alternatives and opportunity costs when system interventions to address various problems were being considered.

Here’s a snapshot of the model’s inputs (in yellow) and its outputs, slightly altered for presentation:

From here, it’s a small step to logging and tracking these total estimated costs across a given time period such as a month. (You should anticipate, by the way, that disclosure of the total dollar cost of outages will tend to invoke far more attention and commentary than the old-style percentage figure ever did!  Transparency brings scrutiny. This is a good thing.)

Yes, estimates and aggregations were necessary in coming up with these approximate costs; it’s just a model, after all, with all sorts of incorporated assumptions. And the estimates will never be utterly perfect. But still, they’ll be a far cry better at helping you express the approximate business impact of your downtime than just proudly declaring that the system was “99.83% available” last month.

Lagniappe:

• Publicly visible major SaaS system availability pages:

• • Amazon Web Services
  • Salesforce
  • OpenDNS
  • Intuit Quickbase
  • Clickability
  • Google App Engine

• Evan L. Marcus, “The myth of the nines”, September 1, 2003
• Continuity Central, “Five Nines: Chasing the Dream?”
• Paul Beckford, “Calculating Server Uptime”, February 13, 2010
• Lenny Rachitsky, “Transparent Uptime” blog
• George Ritchie, Serio Ltd, “Introducing ITIL® Availability Management”