Thursday, November 15, 2012

Power, pipe, ping and the bucket brigade

I spent a number of years in the web hosting biz (hosting providers are where organizations place their servers when they don’t want to run their own data centers). My tensure in hosting spanned the grand and glorious years of the Internet boom, through the crash and the burn, and onto the boring this-is-just-another-scratch-it-out-business-being-commoditized era.

Even in the grand and glorious years, colocation services were the low man on the “value” totem poll.

For those who did not spend a number of years in web hosting, in colocation the service provider rents space, electricity, and connection to the Internet. The customers do the rest: lug their own servers in, and take responsibility for the set up and ongoing monitoring and care and feeding of servers, security, application, data...

Simply put, colocation was pretty much power, pipe, and ping, with the customer doing the pinging (sending signals to their servers to make sure they’re alive and kicking – or at least in some sort of communication via the ‘net.)

Colocation was such a commodity play, we spent most of our time heaping scorn on it, focusing instead on the pricier, more “value added” managed hosting, in which the hosting provider took care of a lot more for the customers.

To get customers to pay through the nose for managed services, we went through all sorts of contortions to figure out how to convince them to pony up for our “value add”.  This was plenty tough, given that our services were inexorably being commoditized as operations became more automated, the ranks of techies with the right skill set and experience expanded,  and high bandwidth grew more available.

At Genuity, part of our “value proposition” was “our guys are smarter”, but whether this really translated into major business value for our customers was not all that clear.

In retrospect, we might have been better off focusing on providing the most cost efficient colocation data centers, making a little money from a lot of customers, rather than struggling to keep extracting a lot of money from the few.

Especially when the competition was so intense, prices were being driven down, and the few were wising up.

If I learned nothing else in my long career in technology, it’s that you will always find a few folks to purchase goods and services that are priced at a premium, even if the value isn’t actually there. Inevitably, the market, in its less than infinite wisdom, but wisdom nonetheless, will figure it out.

It’s just not sustainable to have a business where you have to charge more because of your cost structure if that cost structure means pricing that exceeds the worth you’re offering your customers. In the case of Genuity, part of our cost structure was building data centers that were not just functional, but bee-yoo-ti-ful. I remember getting into one heated discussion in which I argued with the data center decorators that I did not want my team burdened with having to figure out how to justify data centers with hand-hammered copper ceilings in the customer briefing rooms. It was hard enough trying to position our brainy techies.

Anyway, as a hosting veteran, I was interested in a Business Week article I read on how companies were recovering from the disaster that was Hurricane Sandy.

Naturally, I was drawn to part about a hosting provider that has a colocation data center in the NYC flood zone:

Peer1 Hosting, a Net outfit based in Lower Manhattan, thought it was in the clear—it had generators on the 17th floor ready to keep the data servers humming. But when the fuel pump in the basement was flooded, Peer1 was unable to get the necessary diesel fuel upstairs. So it improvised, too. “We had a team of 30 people getting buckets of diesel fuel going up 17 floors to keep the generator working and our customers online,” says Rajan Sodhi, Peer1’s vice president of marketing. Employees hoofed it up darkened stairwells for more than 48 hours, replenishing the tanks.

Bucket brigades lugging diesel up 17 flights of stairs! Now that’s dedication.

Peer1 brags about providing redundant power:

Unlike some providers, we don't rely solely on the local power grid to guarantee around-the-clock power. Our onsite diesel-powered generators and uninterruptible power systems (UPS) deliver redundant power if a critical incident occurs, so that all operations are uninterrupted and your dedicated servers remain online.

Yawn.

I don’t know who these “some providers” are, other than that they are the lamest of the lame data centers, the ones that no business in its right mind would host in. Certainly, having personally toured them, I can attest to the presence of backup, diesel-powered backup generators and UPS at any hosting provider I worked with. Whether you’re just providing power, pipe, and ping, or the whole meghilla of services, you need to be prepared for the eventuality that power will go down.

But Hurricane Sandy is once again reminding us that things can gang agley, and that an erratically rising tide can founder them. Thus, while it was a good idea to have those back up generators on higher ground, it doesn’t help if the pump don’t work. Even if the problem – with a nod to Bob Dylan and his Subterranean Homesick Blues - ain’t that the vandals took the handles.

So the folks at Peer 1 in Lower Manhattan replaced pump power with manpower. Which meant lugging fuel up 17 flights of stairs.

Obviously, the fuel consumption depends on the size of the generator, etc., but at about 7 pounds a gallon, the Peer1 brigade was doing plenty of heavy lifting.

Yes, I know they weren’t the only ones. Hospital employees were doing the same. Those still out of power are trudging up and down long flights to make sure they have enough water to gravity flush their toilets and brush their teeth.

Still, this story reminds me of what I always enjoyed about the operations guys: they always seemed to be able to figure out how to make things work, even when all they had to work with was spit and baling wire.

When I looked at the Peer1 website, they hadn’t gotten around to touting this story.

Maybe they don’t want to. Maybe they think it points out that they were flawed in their planning, that they should have had the backup generator on a lower floor. Or had a diesel tank higher up. Or whatever they coulda/shoulda done to anticipate the pump problem.

But I love this story.

I hope there was a nice, fat bonus for the 30 Peer1-ers who “hoofed it up darkened stairwells for more than 48 hours, replenishing the tanks.”

At Genuity, we used to sell on the brains of our techies. But sometimes it’s the brawn that matters.

No comments: