Thoughts on Vertical Integration: The Cloud and (Gasp!) Ma Bell

Although it has been a couple of months since the hysteria of re:Invent, there has been a great deal of consternation by hoards of people in enterprise IT over the emergence of Amazon AWS. The fear is that AWS and it's main competitors will substantially disrupt the market for enterprise infrastructure and create the next big oligopoly or even monopoly in the technology industry: The Cloud. In this weekend's Barrons, we find Tiernan Ray making the near term case for mediocrity, if not eventual doom for the big enterprise technology players (Note: subscription may be required):

No Silver Lining in the Cloud for Blue-Chip Techs

The key concern among investors - both venture and public market -  and users everywhere really centers around the notion of AWS being a "dominant exchange". This is what happens when the game becomes one where the entire solution set is owned by a single player. In this scenario, scale matters most, and customers can no longer see an incentive to go anywhere else for their needs. 

Well, rather than try an imagine a bleak and scary future where all IT is dominated by a large evil corporate entity, it turns out we can put our fantasies aside and just open a few history books. If you are an American of a certain age, or even a current or former resident of select countries, you have probably lived in a world that was eerily similar to what everyone is afraid of. In the twentieth century, the paradigm of technological vertical integration has to be the old, pre-breakup, Bell System - also known as American Telephone & Telegraph. It turns out that looking at AT&T's business model and history is hugely instructive for those who are planning strategies in the Everything-As-A-Service world of the early 21st century.

Voice-As-A-Service circa 1900

If it weren't for some bad behavior and a bunch of concomitant anti-trust lawsuits, the story of AT&T would be an example of a great American capitalist success. Famously, Alexander Graham Bell invented the telephone in the late 19th century and set out to commercialize his discovery with his eponymous company. In order to deliver the service, the Bell System became an end-to-end solution for making voice calls to other Bell customers. This meant that they provided the telephones to their customers, they strung the wires on the streets, built and maintained central offices where operators would help patch point-to-point connections for customer calls, manufactured all of the equipment that the operators and customers used, and serviced all parts of the system in case of trouble.

To make the story interesting, the company was an aspiring monopolist. They refused to interconnect with other telephone providers unless they sold out to Bell, and they set the standard for how interconnection would work when they owned such a subsidiary. This got them into trouble when they bought out their biggest competitor, Western Union, and attracted the attention of the anti-trust authorities of the pre-World War I federal government. Eventually, AT&T was granted a legal monopoly and became a regulated entity. What is fascinating to consider, however, is how they operated within those constraints for 60 years. While the world has changed a lot in the last 100 years, the recent discourse around the cloud has had me making a comparison: Is The Cloud - or AWS, in particular - destined to be the new Bell System?

The tradeoffs of this extreme level of vertical integration are particularly interesting. As a result of owning everything, The Bell System all actually worked. The phone system, the switch gear, the telephones, and everything along the way was overbuilt and extremely reliable. One struggles to remember a time in the 1970s when their phone service was unreliable or when they needed to troubleshoot a telephone problem. On the other hand, the pace of innovation was stiflingly slow. If it weren't for telecom deregulation in the 80s and 90s, we would not have meaningfully fast data network access available today. We would not have VoIP. Long distance calling would be inexplicably expensive. Most importantly for most of us, the economy would be bereft of trillions of dollars in market value that has been derived from IP networking.

When one considers the direction of the products and technology in the 80's, it is obvious that AT&T did not perish as a result of anti-trust action, but rather from obsolescence, just like every other tech company. They were so busy attending to that enormous customer base, that they could not address the needs of their more innovative customers. In fact, any enterprises that had advanced telecom needs had already found ways to work around AT&T long before it was broken up. Media companies were sending transmissions via satellite links, and large enterprises were buying private switching equipment from a slew of competing companies that catered more to their needs than the one-size-fits-all approach offered by "the phone company".  If it weren't for their status as a legal monopoly, they would have been disrupted much sooner.

Is IT-As-A-Service Destined to Remake Enterprise IT?

Looking at the Cloud, the key innovation that the established enterprise players are finding disruptive is really a business model. Turning a product into a service in a way that makes economic sense is a ground-up endeavor. Setting that aside, this generation of technologies built for The Cloud are just as easily disrupted as previous generations. Whether or not the establishment players will be able to hang on to their customers hinges on the same factors that have always been important: Will the incumbents be able to adjust to the evolving needs of IT, or will someone new take the business away? Most importantly, do AWS and the other service vendors provide any meaningfully new technologies that need to be consumed as a service rather than a product? It is still far to early to tell. Certainly, it is not yet time to sell the incumbents short. The next few years will be interesting, though.

Sunday Night Scotch: Disruption Is So... Well... Disruptive

The reason that God was able to create the world in seven days is He didn't have to worry about the installed base.  - Enzo Torresi

Maybe it's a sad testament to the human condition, but the technology industry does not seem to lend itself to attracting individuals with humility or self awareness. Since my very early days as a junior software engineer, I have noted that it is only a matter of time before many archetypical tech nerds, having achieved a modicum of monetary success, start to behave erratically. Before long, you see them standing on top of the roof at corporate headquarters, laughing maniacally. In a thunderstorm. Waiving a five-iron in the air. I'll grant that this happens in other professions as well, but there's something about the rags-to-riches nature of tech that makes it all the more poignant.

If you are an executive in a leadership position at any firm, it therefore follows that you need to go through the history of other people's mistakes obsessively so you learn from them. (I recommend doing so very clinically - like the NTSB examining the wreckage of an airplane crash.) What you inevitably find, tragically, is that history really does rhyme even if it doesn't repeat. With that thought in mind, I present to you the scariest part of my reading list from last week: The Globe and Mail's investigative piece on the failure of Blackberry:

Inside the fall of BlackBerry: How the smartphone inventor failed to adapt

In it, you'll find a freakish retelling of pretty much every cliché of every tech downfall you can imagine. If you are a technology leader, and you'd like to engage in some negative reinforcement therapy, I recommend it highly. Likewise, if you need an excuse to have a strong drink, you'll find all you need there. Here are some of my thoughts:

Disruption Is Never Evident Until After The Fact

In her book entitled Being Wrong: Adventures In The Margin Of Error, Katherine Schulz states that being wrong feels exactly like being right. What we remember most painfully is the moment when we realize we are wrong. What will make you cringe, therefore, is the fact that much of the investigative narrative does not occur in the months after the iPhone's market success in 2007 and 2008. It is a story set in events of 2012 and maybe late 2011. It wasn't until then that the mobile phone market had been disrupted to the point where Blackberry sales were beginning a secular, irreversible decline. That is a long time. Friends, that is what disruption looks like.

The point here is that you can't advertise a sea change. Smart people do not screw up this badly unless they are completely unprepared. By definition, a technology market can't be disrupted if all the players were anticipating the change. So, all you Storage Geeks take note: Things like flash technology are really more evolutionary. Everyone knows about them and they are all making bets. It is unlikely that anyone will go out of business. In fact, recent failed IPO's are testament to that. True disruption sneaks up on you.

Listening To Your Customers Can Kill You Too

Conventional business wisdom would have executives in a bear-hug with customers, listening and catering intently to their every whim. In a disruptive situation, the customers you are listening to aren't the ones that are going to rock your world. It is the early adopters and the rebels that are driving the change. A large, established technology company cannot pay attention to these people because they do not represent a viably large market. When it would have mattered to do otherwise, Blackberry was clearly making decisions based on the needs of their huge revenue base: the one that was paying them for physical keyboards on their phones. They were doing precisely the correct thing. They were wrong to do so.

Just hiring a CTO to go watch these people wont work either. I know exactly what it feels like to point out that a fringe minority has a better idea about how to do things. In a publicly held company, the business worships at the "church of what's happening now". There is no accounting gimmick or spreadsheet that you can conjure, which makes an ROI case for investing in a disruptive technology. You will lose your case every time. When the return is 4 years into the future, no one will care. This is why there are venture capitalists.

Finally, we can see what can go wrong when you send off  a bunch of rebels to "do the right thing" as Blackberry did with the touch screen products. They can easily lose touch with your customer base, and build something that your existing customers wont transition to. At the same time, they may not get any new customers either, unless they are truly brilliant in their execution.

No One Can Explain the Ten Year Run Apple Has Had

When the history books are written for business over the last decade, I am not sure how they will be able to distill the moves Apple made into a real repeatable formula. It is not lost on me the hypocrisy of criticizing the seemingly stupid, calcified, large incumbent technology organizations, when the player bringing the disruption is actually a big company, too. The ability to attack huge, seemingly unrelated markets from a position of financial strength helped. Not having to drag an existing user base along also seemed to go a long way. So did the organizational structure that protected the development of these new products from the vicissitudes brought forth by the markets. ...But getting it right every time on these giant bets? That is pure genius or pure luck.

Cheers!

Dispatch From SNIA SDC: Hardware Defined Software

Much to my surprise, the most contentious portion of my June SPDECon discussion on software-defined storage was the seemingly innocuous slide I added at the end: In it, I simply decreed that hardware widgets couldn’t be a part of a software-as-storage solution.  This seemed to be a self-evident assertion and, accordingly, I put it at the end of my slides thinking that no one would notice it.  We never got to the Q&A part of my talk because this discussion usurped all of the extra session time.

After a good amount of thinking on the subject, I think I have changed my mind somewhat. Software defined storage is tragically a hardware-constrained technology. We run out of compute cycles, and the speed of light may be too slow. These sorts of vexing architecture problems are more directly and easily solved with specialized hardware.  Clearly, the people in the room last June who made a living from solving problems with hardware had a legitimate concern. After that smack-down, I have done fair amount of evolution on my thinking on hardware in a software-defined world.

It is looking like I will be doing the same talk again tomorrow at SDC. This time, I am using a different deck. It might turn out to be a bit more depressing if you are expecting a cookbook on how to build SDS. While I am not going to spoil my own discussion topic, I will say that it is instructive to look at places where platform properties have successfully been abstracted, so that the software doesn’t cease to work when specific hardware is missing. One example of this is the advanced SCSI functionality that has gone through standards over the last few years. VAAI, ODX, and the like are all hardware specific functions that make life substantially better for people without de-virtualizing their virtualization platform.

All that said, I’m sure I’m going to learn something new again… 

Thoughts on the MS/Nokia Deal: Software Now Comes in a New Box

Now that the dust is starting to settle a little from this weekend's hysteria over Microsoft's acquisition of Nokia's mobile assets, it probably is a good time to step back a little, take a deep breath, and think about what this means to the IT ecosystem. A good deal of ink and lots of electrons have been tortured on the Internets in breathless criticism of this deal. I think that its safe to say the usual suspects will always pop up to complain about what a horrible waste of money this is, and ostensibly, how it will mess up their carefully crafted, but meaningless financial models. Setting aside the finances, this move portends a lot of big things in the world of IT and application development. Let's walk through the list in order of obviousness.

The "New New Client" is Mobile and not MS

This can't possibly be the first time you are reading this, so I'll spare the details. Every decade or so, the predominant client architecture and it's user interface paradigm goes through a profound change. To what should be no one's surprise, it's happening again in a couple of distinct ways. Firstly, the world is going to touch interfaces. Apple's iPad can be credited for making this change stick. Secondly, the world's end users have now become inured with consuming application content on their phones. Again, Apple largely owns credit for getting this over the mass market chasm. Both of these things have a lot of repercussions if you are writing apps going forward.

What is different this time, is that the big client paradigm change was NOT driven by Microsoft. It was not influenced by Microsoft. In fact, it's hard to tell whether they were even in the room while all this was happening. So here we are, with Apple calling the shots, and the Redmondians are occupying the cheap seats. Back when I was much younger, Microsoft was the one doing the disrupting by forcing everyone to rewrite their apps for the Windows UI. This was one of the big reasons that MS succeeded in owning the PC by the early 90's. That this same strategy is now being perpetrated on their Windows platform is not lost on them. Clearly, this deal is a major part of getting their mojo back in their view.

There is a Change of Thinking Afoot in Redmond

Microsoft has always been as pure play a software company as can be imagined. Their recent trend has them veering away from that somewhat. I think this is very significant. For the last 15 years, the notion of "tin-wrapped" software has been steadily gaining credence. At first, the cognoscenti hated it because with the tin - and it's enclosed hardware - came costs, and hence lower profit margins. It has been proven, however, that this approach has a lot of advantages, especially when it comes to controlling and managing the customer experience. Intuitively, one can see that the more of the stack that a single vendor integrates, the better the customer outcome. The customer gets a more seamless, plug and play experience. The vendor also gets better lock in loyalty. Nowhere is this approach more evidently successful, than at Apple. (Oops. There's that name again.)

So, it seems that if you are going to make a technology product for mass markets, the right way to do it is to own as much of it as possible. Hence we see Microsoft making tablets (e.g. Slate), and now, they are going to be making phones. I am not going to opine on whether they can successfully pivot in this manner, but it is significant that they are slowly aligning themselves with what they see as working well in the market. They have done this before, and regardless of the outcome, they will make things very interesting for everyone involved.

If You Are Writing Apps, You Have Some Tough Choices Ahead

In the Web Era, the client was largely defined by a browser, HTML, and things like JavaScript, Java, OCX, Flash, etc. This is how you delivered your application experience to your user. That paradigm may have already Jumped The Shark as I write this. When all of the truly interesting, interactive user experiences are from natively written, touch enabled, mobile applications; the market will have to follow suit. Users will have spoken. This is already happening with services like Yelp, Uber, and Facebook, where usage has increased due to the quality of the mobile experience. Slapping an HTML 5.0 page in a browser window will be an obvious step down for a mobile user. 

Now that Microsoft has gotten religion on mobile and touch, you can be certain that it will tip whatever remaining part of the market that was on the fence. This means that app developers now need to think about native UIs for iOS, Android, and maybe even Windows. That's an awful lot of work, and some tough choices to make. For guys like me, it also means lots of cool new things to build. I love paradigm shifts.

Relax, man. We Are Only Looking For Your Metadata.

I've always been known to be a cynic. There is, in fact,  something inside my brain that trips the minute I start to partake in any kind of exuberance over matters related to technology. But even in my most hype-averse moments, I never imagined to think as skeptically as James Glanz, who penned a most sober article in this weekend's New York Times:

Is Big Data An Economic Dud?

After pointing at the reams of data being collected by everyone from Amazon to Google, and all of the hype surrounding its purported value, he gets right to work:

There is just one tiny problem: the economy is, at best, in the doldrums and has stayed there during the latest surge in Web traffic. The rate of productivity growth, whose steady rise from the 1970s well into the 2000s has been credited to earlier phases in the computer and Internet revolutions, has actually fallen. The overall economic trends are complex, but an argument could be made that the slowdown began around 2005 — just when Big Data began to make its appearance.

So, let's calm down a minute and really draw a few distinctions about what has happened recently versus what has been happening for quite some time. It is very easy to conflate the old with the new in this space, and then easily draw the wrong set of conclusions.

There's a field called Operational Research that has been around for quite some time that looks, feels, and smells just like Big Data, but it really isn't as bleeding edge as one might think. Mostly because it seems boring, not a whole lot of people have noticed it. Rest assured, though, it has been making your life better and creating tons of economic value. That the fact airports schedule all those flights, quants at hedge funds squeeze out all those pennies, and UPS manages to deliver more packages every year without clogging every city with brown trucks is evidence towards that. (Stupid fact of the day: If you follow a UPS truck through the city, you will notice that they do not make left turns. Faster that way.) The old economy has been using OR to increase efficiency for decades, and it is probably one of many reasons that productivity continued to climb through the 90's.

There is a "new new" thing that, depending on who you ask, is still largely untapped. This is all of the metadata that gets generated when the world creates transactional data. The sheer size and scope of this makes it "Big" - it challenges the limits of the compute infrastructure that companies have to apply to it. To make matters worse, much of it is unstructured - it does not live neatly in the rows and columns of a database. You can't just throw up some SQL queries on it. Unless you understand it deeply, you don't even know what the relations would look like.

What is this stuff? Think of all of the log data from all of your servers and virtual machines. Consider all of the time stamps and file sizes, or maybe how many of the videos posted to YouTube are of kittens. There's a huge amount of intelligence there that can be leveraged to improve efficiency of your business in the same way OR has helped in the past, but the tools are not fully cooked.

I would argue that Google and Amazon, and the like have done a truly outstanding job of leveraging this metadata to streamline their businesses. This, and the hype, is why their stock prices are as stratospherically high as they are: They are far more efficient at targeting ads, moving goods, etc. than their old world peers. Sadly, they are just a small part of the economy. Leveraging all that metadata in every business requires insight, intelligence, and tools. These aren't there yet. Hence, its a bit premature to sit there tapping your toe waiting to feel the effects of the Big Data revolution.

Grabbing For Pennies in Front of an Oncoming Locomotive

I was trying to think of a good title for a post on "Software Defined" stuff, and I settled on the above as being apropos. Apologies to all the believers I may offend in the next few paragraphs. It's not that I think this stuff can never work. (It can.) It's just that people tend to confuse the outcome with the means. The implication to something being Software Defined is that it is not tied or reliant on particular hardware... or ANY hardware. I think that one needs to keep the dialog centered by going back to that definition when things start to get screwy.

The common misconception that I want to discuss is the idea that, in any system, if you break out the software from the hardware, you somehow get huge cost reductions that flow directly into your (i.e. the end user's) pockets. Sorry to tell you this: The world just doesn't work that way. For some reason, though, this idea has become almost zombie-like. No matter how hard one tries to dispel it, it keeps coming back to life.

To understand why you can't just take apart a system product and expect uniform results from assembling it yourself, it is necessary to understand what goes into building a storage array, or a switch, or any technology platform. First, understand that rack mount, industry-standard, Intel-based servers are not commodities in the strict sense of the word. A commodity is a good that cannot be differentiated in the marketplace, i.e. the processes that create them are well known, and the products are identical. Think of a lump of coal.... or gasoline. While it can be argued that servers have been commoditized, that does not really mean the same thing. Technology goods are actually a lot more complicated than a lump of coal. There is a lot of manufacturing finesse that goes into them, and the fact that they are selling for pretty slim margins is an artifact of market dynamics more than anything else. (Digression: the dynamics of the server component supply chain are more frightening than you might think at first)

More to the point, that a server is assembled and sold for so little monetary gain means that the manufacturing process has been streamlined to the point where a few key things are not being done. In my world, we call this "integration", or maybe you know it as something else: "testing".  Hence, when you assemble a system with a chip set from vendor A, memory from vendor B, a network card from C, disks from D, an operating system and application stack from E, and then you run a particular workload; well... you can be assured that not a whole lot of time has been spent making sure this thing holds together. This is what teams of smart people do at EMC, Cisco, NetApp, HP, Dell, and a bunch of other places. In order to accomplish the task, they have to narrow down the components to the point where they are not interchangeable. Then, to make you feel better, they offer to stand behind their work if there is a problem.

There may be a contingent of people out there that believe testing and qualification is something they can do without if they can reap the extra 20% of cost gains. Maybe, if they are lucky. The sad truth is that when some component in the commodity stack proves to be defective, you will need an engineering staff to test and diagnose it - a very time consuming and expensive process. Then, if you do happen to find the root cause of your troubles, you really wont have a lot of leverage with the manufacturer unless you are a buyer of significant scale... meaning millions of units. So, finally, you end up with a lot of expensive wasted time and a cardboard box full of broken parts.

Ultimately, the cost of a small number of bad experiences can wipe out the benefit from years of being lucky. In essence, you have a comparatively small, well defined financial upside: the money you saved. Your downside is all the debugging, diagnosing, and down time; which, can easily wipe out the gains from any upside. As the title of this post suggests, this a suckers' bet. Unless you think you can cover your downside, be very careful.

This doesn't mean that unbundling the software from the hardware in a solution has no benefits. In fact, there are many of them, and I may talk about them when I am feeling less cynical. The key point is that the cost savings is not normally going to be part of the benefits. Hence, focusing on cost really misses the point of building out the ecosystem.

Storage Seems Exciting and Networking Seems Dull

I have to apologize for not posting, as I've had a pretty busy week traveling to the Bay Area for meetings with investors and other assorted smart people. As some of you know, trying to define a product - and a business around it -  is generally a difficult thing. Lots things to think about, so let's dive in.

One of the things that I keep bringing up in conversation around here is the notion that I rudely expressed in the title of this post. Taking it down to the next level: If you were to come up with a new datacenter innovation, would you want to make it part of the networking infrastructure, or something else? What the hell has happened to innovation in the networking space, anyway? Sadly, the title of the post is pretty close to my conclusion... for now. After talking to a lot of folks, from a lot of different parts of the value chain, I got a lot of different perspectives, but really all pointing to the same conclusion. It's enough to get you depressed. Here's what people see:

Adoption curve - It used to be that network ports were like chocolate covered coffee beans in my office. You couldn't get enough of them. Moreover, there were manifest bottlenecks everywhere in the topology. Nothing was more urgent to a growing business than getting more, faster ports as soon as possible. Alas, this has changed a lot. The transition from 1G Ethernet to 10G is still not complete years after the technology was first introduced. Further, workloads that can fill those pipes are not commonplace. Except for some unique, large scale situations, IT does not see the need to go beyond current technology for a while.

Sales motion - The above adoption rate has led to a much longer sale cycle, and the expected life span of networking gear has grown commensurately longer as a result. Servers and storage get replaced every 2 or three years. Both get add-on investments as capacity is needed. Networking gear replacement has become a 5 to 8 year event. Put most succinctly by a reseller friend, "I can sell you storage today, and return to sell you more in a few months. I sell you switches, and then I have no reason to call you for 5 years." Add to that the obvious truth that one vendor dominates the space in a way that causes resellers great discomfort, and the general reluctance to compete is understandable. It just isn't fertile ground to grow new businesses.

Burden of Innovation - While nothing is more thorough than the interoperability testing that goes on in networking, it has a cost. Network infrastructure is completely closed. What does that mean? In order to propose a new innovation in the networking space, it is nearly impossible to do it on existing gear for technical reasons. Standards have reduced network management to a completely decoupled state, where all control plane services are inaccessible to software running outside the switches. There's nothing to be done with the standard interfaces. This means that the way to innovate is to build switches. Of course, if you build those, you quickly have to face the adoption curve issue, and the sales motion problems.

For a lot of reasons, many people might be reminded of the old stove-piped systems of the 70's and 80's when they look at the space. The typical approach back then was to build a new hardware platform, build an operating system (or buy some UNIX code from AT&T) and bring up the new machine. Then, you could build the pieces that make you different.

Isn't this what SDN is about? Well, yes. Software Defined Networks have the potential to change this pathology, but the evolution of the technology is taking some bizarre twists that make it a scary place to start a business. This is probably the subject of another post, but suffice it to say, if a reasonable person can't see a path to implementing a new forwarding algorithm on a network without building the entire stack from scratch, it will all fail.

So there you have it. The only counterpoint I can offer is that I remember a time when networking was exciting and storage was dull. I'd love to hear more thoughts from all of you.. especially if you disagree. What say ye?