Grabbing For Pennies in Front of an Oncoming Locomotive

I was trying to think of a good title for a post on "Software Defined" stuff, and I settled on the above as being apropos. Apologies to all the believers I may offend in the next few paragraphs. It's not that I think this stuff can never work. (It can.) It's just that people tend to confuse the outcome with the means. The implication to something being Software Defined is that it is not tied or reliant on particular hardware... or ANY hardware. I think that one needs to keep the dialog centered by going back to that definition when things start to get screwy.

The common misconception that I want to discuss is the idea that, in any system, if you break out the software from the hardware, you somehow get huge cost reductions that flow directly into your (i.e. the end user's) pockets. Sorry to tell you this: The world just doesn't work that way. For some reason, though, this idea has become almost zombie-like. No matter how hard one tries to dispel it, it keeps coming back to life.

To understand why you can't just take apart a system product and expect uniform results from assembling it yourself, it is necessary to understand what goes into building a storage array, or a switch, or any technology platform. First, understand that rack mount, industry-standard, Intel-based servers are not commodities in the strict sense of the word. A commodity is a good that cannot be differentiated in the marketplace, i.e. the processes that create them are well known, and the products are identical. Think of a lump of coal.... or gasoline. While it can be argued that servers have been commoditized, that does not really mean the same thing. Technology goods are actually a lot more complicated than a lump of coal. There is a lot of manufacturing finesse that goes into them, and the fact that they are selling for pretty slim margins is an artifact of market dynamics more than anything else. (Digression: the dynamics of the server component supply chain are more frightening than you might think at first)

More to the point, that a server is assembled and sold for so little monetary gain means that the manufacturing process has been streamlined to the point where a few key things are not being done. In my world, we call this "integration", or maybe you know it as something else: "testing".  Hence, when you assemble a system with a chip set from vendor A, memory from vendor B, a network card from C, disks from D, an operating system and application stack from E, and then you run a particular workload; well... you can be assured that not a whole lot of time has been spent making sure this thing holds together. This is what teams of smart people do at EMC, Cisco, NetApp, HP, Dell, and a bunch of other places. In order to accomplish the task, they have to narrow down the components to the point where they are not interchangeable. Then, to make you feel better, they offer to stand behind their work if there is a problem.

There may be a contingent of people out there that believe testing and qualification is something they can do without if they can reap the extra 20% of cost gains. Maybe, if they are lucky. The sad truth is that when some component in the commodity stack proves to be defective, you will need an engineering staff to test and diagnose it - a very time consuming and expensive process. Then, if you do happen to find the root cause of your troubles, you really wont have a lot of leverage with the manufacturer unless you are a buyer of significant scale... meaning millions of units. So, finally, you end up with a lot of expensive wasted time and a cardboard box full of broken parts.

Ultimately, the cost of a small number of bad experiences can wipe out the benefit from years of being lucky. In essence, you have a comparatively small, well defined financial upside: the money you saved. Your downside is all the debugging, diagnosing, and down time; which, can easily wipe out the gains from any upside. As the title of this post suggests, this a suckers' bet. Unless you think you can cover your downside, be very careful.

This doesn't mean that unbundling the software from the hardware in a solution has no benefits. In fact, there are many of them, and I may talk about them when I am feeling less cynical. The key point is that the cost savings is not normally going to be part of the benefits. Hence, focusing on cost really misses the point of building out the ecosystem.