Thoughts on Vertical Integration: The Cloud and (Gasp!) Ma Bell

Although it has been a couple of months since the hysteria of re:Invent, there has been a great deal of consternation by hoards of people in enterprise IT over the emergence of Amazon AWS. The fear is that AWS and it's main competitors will substantially disrupt the market for enterprise infrastructure and create the next big oligopoly or even monopoly in the technology industry: The Cloud. In this weekend's Barrons, we find Tiernan Ray making the near term case for mediocrity, if not eventual doom for the big enterprise technology players (Note: subscription may be required):

No Silver Lining in the Cloud for Blue-Chip Techs

The key concern among investors - both venture and public market -  and users everywhere really centers around the notion of AWS being a "dominant exchange". This is what happens when the game becomes one where the entire solution set is owned by a single player. In this scenario, scale matters most, and customers can no longer see an incentive to go anywhere else for their needs. 

Well, rather than try an imagine a bleak and scary future where all IT is dominated by a large evil corporate entity, it turns out we can put our fantasies aside and just open a few history books. If you are an American of a certain age, or even a current or former resident of select countries, you have probably lived in a world that was eerily similar to what everyone is afraid of. In the twentieth century, the paradigm of technological vertical integration has to be the old, pre-breakup, Bell System - also known as American Telephone & Telegraph. It turns out that looking at AT&T's business model and history is hugely instructive for those who are planning strategies in the Everything-As-A-Service world of the early 21st century.

Voice-As-A-Service circa 1900

If it weren't for some bad behavior and a bunch of concomitant anti-trust lawsuits, the story of AT&T would be an example of a great American capitalist success. Famously, Alexander Graham Bell invented the telephone in the late 19th century and set out to commercialize his discovery with his eponymous company. In order to deliver the service, the Bell System became an end-to-end solution for making voice calls to other Bell customers. This meant that they provided the telephones to their customers, they strung the wires on the streets, built and maintained central offices where operators would help patch point-to-point connections for customer calls, manufactured all of the equipment that the operators and customers used, and serviced all parts of the system in case of trouble.

To make the story interesting, the company was an aspiring monopolist. They refused to interconnect with other telephone providers unless they sold out to Bell, and they set the standard for how interconnection would work when they owned such a subsidiary. This got them into trouble when they bought out their biggest competitor, Western Union, and attracted the attention of the anti-trust authorities of the pre-World War I federal government. Eventually, AT&T was granted a legal monopoly and became a regulated entity. What is fascinating to consider, however, is how they operated within those constraints for 60 years. While the world has changed a lot in the last 100 years, the recent discourse around the cloud has had me making a comparison: Is The Cloud - or AWS, in particular - destined to be the new Bell System?

The tradeoffs of this extreme level of vertical integration are particularly interesting. As a result of owning everything, The Bell System all actually worked. The phone system, the switch gear, the telephones, and everything along the way was overbuilt and extremely reliable. One struggles to remember a time in the 1970s when their phone service was unreliable or when they needed to troubleshoot a telephone problem. On the other hand, the pace of innovation was stiflingly slow. If it weren't for telecom deregulation in the 80s and 90s, we would not have meaningfully fast data network access available today. We would not have VoIP. Long distance calling would be inexplicably expensive. Most importantly for most of us, the economy would be bereft of trillions of dollars in market value that has been derived from IP networking.

When one considers the direction of the products and technology in the 80's, it is obvious that AT&T did not perish as a result of anti-trust action, but rather from obsolescence, just like every other tech company. They were so busy attending to that enormous customer base, that they could not address the needs of their more innovative customers. In fact, any enterprises that had advanced telecom needs had already found ways to work around AT&T long before it was broken up. Media companies were sending transmissions via satellite links, and large enterprises were buying private switching equipment from a slew of competing companies that catered more to their needs than the one-size-fits-all approach offered by "the phone company".  If it weren't for their status as a legal monopoly, they would have been disrupted much sooner.

Is IT-As-A-Service Destined to Remake Enterprise IT?

Looking at the Cloud, the key innovation that the established enterprise players are finding disruptive is really a business model. Turning a product into a service in a way that makes economic sense is a ground-up endeavor. Setting that aside, this generation of technologies built for The Cloud are just as easily disrupted as previous generations. Whether or not the establishment players will be able to hang on to their customers hinges on the same factors that have always been important: Will the incumbents be able to adjust to the evolving needs of IT, or will someone new take the business away? Most importantly, do AWS and the other service vendors provide any meaningfully new technologies that need to be consumed as a service rather than a product? It is still far to early to tell. Certainly, it is not yet time to sell the incumbents short. The next few years will be interesting, though.

Can A Slingshot Take Down The AWS Juggernaut?

If your purpose in life is to entertain the gods, you might as well put on a good show.

 --Unknown

Amazon's Re:Invent show last week turned out to be quite a coming out party for the predominant infrastructure service provider. There were a number of interesting announcements that moved the stocks of perceived competitors, and of course, there was a disproportionately large showing of attendees at the Venetian in Las Vegas. For me, the highlight was provided by good friend, and super smart VC, Jerry Chen, who went on  The Cube to lay down the gauntlet starting at about 7:00 in the video below:


Watch live video from SiliconANGLE.com on Justin.tv

The salient point Jerry made was the comparison between an ascendant Amazon AWS and the now-decidedly incumbent Microsoft circa the 1990's. His argument is that there are really two investable bets to make (well, he said three, but the third is less interesting to me). The first is that there are companies to be formed that can make AWS more enterprise-ready, and the second is that you can invest in someone that will take down AWS. If nothing else, it is heartening to see Jerry has taken up the mantle of the venture capitalist, and has pointed to the next big hill for the army to conquer. As entrepreneurial cannon fodder in this battle, I greatly appreciate the affirmation.

Well, since I can't sleep well on airplanes, and I had the misfortune of taking an overnight flight home from Las Vegas, I had plenty of time to think about this matter. After reaching back into my memories as a newly minted engineer in 1991-1993, it is obvious that history does indeed rhyme if it does not repeat. Hence, I think I have come to a conclusion on which will be the better bet if I were investing venture money right now.

A Tale of Two Microsofts

Having the pleasure of attending some of the original Win32 Developer's conferences, the Microsoft PDC's, during the early 90's, as well as the WinHEC conferences up until 2008, I had a semi-privileged, front row view of the revolution that Microsoft led over the course of a decade. Given the day-to-day reality in which we operate, it is easy to forget that the world at the time of the 1993 PDC was radically different than what we have today. In fact, most readers may find my observations of being a Windows "dev" back then quite amusing.

As I recall, developing code for Windows before Windows NT was released was a trying process. The development tools before Visual C++ were not that friendly. The operating systems were either very buggy and unreleased (as in the 32 bit NT), or super-duper buggy and shipping (Windows 3.0, 3.1, 3.11). In the case of the latter releases, you had no meaningful memory isolation between tasks, and so a simple coding typo could take down the whole box while you were working. Having learned to code in the Berkeley and AT&T UNIX operating systems, it felt like I was playing with a toy rather than a tool of enterprise transformation.

Back then, the "real" systems that businesses ran on were either UNIX-based mid range servers, or mainframes. The PC client was really just an over-powered terminal that also had some desktop apps, as well as file and printer sharing. Strategy discussions in PC software companies centered around how overpowered the clients were in comparison to the big iron, and what that fact foretold about the future. Indeed, the developers' conferences were largely full of optimistic young developers trying to change the role of the PC architecture. The rhetoric coming from Gates, Ballmer, and Allchin included lots of chest-pounding bravado about how good the next generation would be, and how it would take on a huge role in the enterprise - if only we developers would agree to write awesome new apps for Win32. We all would go home with palpable excitement and a religious zeal.

A little over a decade later, the world was radically changed. Perhaps not for the better. Substantially all of the mid-range systems vendors had vanished. Windows was dominating both the client and well as the back office of enterprises everywhere. The ecosystem that had developed all those awesome Windows apps had largely been cannibalized by Microsoft, and all those developers had moved on to writing web apps or other cool, Linux-based  things that were out of the way of the perceived MS predatory machinery. It was here that compute and storage virtualization emerged as dominant technologies. It is easy to argue that AWS and VMware tipped the datacenter market from the new incumbents right here.

Amazon Looks More Like the MS of the 90's

Last week's conference was full of developers. The rhetoric brought back memories of the 90's. The show floor was full of small, venture-funded companies. The representatives of the big incumbents were trying to make themselves invisible. Everyone, including the VC's, were talking about how AWS was not ready for the enterprise - yet. Although rumors suggest that AWS is a $5 billion revenue stream, it does not look like the big players are using it yet.

It's hard to envision that AWS can be tipped over when it's user base is not the demanding enterprises that make up the bulk of IT spend in the market. The fact that it got to this point based on the grass-roots support of a big developer community makes it very scary. People are right to be afraid that this company could be the next big IT monopolist. However, if you are an investor, would you consider it an easier bet to take them down, or to help them achieve the dominance they seek? The key, in my opinion, rests with Amazon. If they take a page from their neighbors in Redmond, and eat their ecosystem, the community will move on very quickly, and the bet is an easy one.

So, Where Do Clouds Go To Die?

The last week's events have been very interesting, although perhaps more unnoticed than I think they should have been. We began the week with the rumor, ultimately proven to be true, that Nirvanix was about to take a dirt nap. This was a very interesting company for a number of reasons: First, they were primarily a cloud storage provider. Second, they were not Amazon or Google. They were venture funded, and clearly unprofitable. Most importantly, they had a good number of customers.

Admittedly, reading headlines like this has somewhat of a NASCAR effect on me: Watching the cars crash can be most of the entertainment. There were certainly customers that would rightly lose their composure if something like this were to happen. I didn't hear of a whole lot of angst, however. In fact, except for the obvious admonitions that this is a bad development from a number of folks in the investment community, I was surprised how quietly this all went down. Nobody wrote any scathing copy urging IT pros to call their storage vendors and buy good old fashioned physical arrays, although I'm sure folks are adjusting their PowerPoint decks for their sales calls this week. (Why this seems so casual is the subject of another post...)

I was pretty much all set to shrug this event off as if it were a typical market consolidation shift until I got an email from Ziptr. Ziptr, for those who are unaware, was a secure document sharing service. As I write this a few days later, there's no point in linking to their web site. They, too, are long gone.  Their final email to me read as follows:

Dear Lazarus,

  

Please export your data by 12:00 noon Eastern tomorrow, September 27, 2013.  Ziptr is closing, and all Ziptr products will be discontinued as of tomorrow.

If you do not need future access to your data, you do not need to take any action.  Once we shutdown the Ziptr service this Friday, we will be deleting ALL user data that remains in our systems. 

To guide you through the process of exporting your important information, please visit www.ziptr.com/FAQ or contact support@ziptr.com.  

Sincerely, 

Ziptr Support

Ziptr, Inc.

Oddly, several people had used the service to send me confidential documents. These did not belong to me, thankfully, but it suddenly hit me: The company is gone, and it has left behind some assets that someone is going to liquidate. In the old days, no one would consider anything but the physical assets to have any value, i.e. servers, printers, desks, chairs, etc. In today's analytics-driven world however, data and it's metadata have become a sort of currency. The value of whole companies can be largely attributed to the data they house, and the leverage that they are able to create from it. This now-defunct company was a data management company.

Thus, a series of seemingly unanswerable questions keep coming up in my head. Some of them are quite interesting. For example, in a liquidation of a company, who owns the data? The creditors? (Check the terms of use...) What obligation do they have to abide by the same agreements as the defunct entity? (None, apparently...) If they were to receive  a subpoena for your data, would they fight on your behalf? (I'm thinking not.) If they were to make your data public, would you have any legal recourse?

Here's what might freak everyone out the most: It's clear that the Ziptr guys were intent on at least trying to do the right thing to preserve security of the data they were holding - i.e. they say they will delete it. Whether they succeed at this is another matter. Whether they had other physical media containing data as backup is yet another further consideration. What happens if you are dropping your data into a service provider cloud that left the encryption up to you? What if you didn't bother to encrypt your data? Most importantly, what if they didn't bother to destroy the data after a bankruptcy?

In the Brave New World in which we live today, it's clear that the legal framework we are using to operate may be sorely inadequate to protect us from the kind of predatory entities that may exist out there. I am not aware of any of the scenarios I am contemplating having really been tested in American courts. The outcomes may not be in favor of the entity that created the data. It really is unclear who would win in a lawsuit. So what do you do?

First, it is doubtful these happenings will materially dent the growth of the cloud business model. We have gone too far to reverse, and the value proposition is established. On the other hand, a bit more due diligence may be required before you sign on to a service provider if they are going to be holding your data. If you are going to put it in a cloud, you really need to think about a different sort of "disaster recovery", because our current set of laws are probably not going to adapt in an expedient fashion to the march of technology. At a minimum, this means that you need to be sure you retain the rights to your data, that it is secure, that you can move it when needed, and that you can be certain it can be vaporized if you deem it necessary. Is this the beginning of a "Data Bill of Rights"? Well, we should be talking about this.

Software Defined Storage - Zombie Box Huggers are Winning

In all honesty, I first submitted the abstract for a talk on Software Defined Storage to SNIA very early this year. It seemed like a different world then, and I really had no idea what I was getting into. For the purposes of full disclosure, I was the sponsor for an SDS project before I left Dell, and so I had spent a lot of time sifting through all the data and the nonsense that surrounds it. It felt apropos to put together a talk on what we had learned about the topic: the use cases, the technologies, the market, and the business. In the intervening months, something happened out there to cause the topic to become an epic technological hot potato. I've now given two talks at two separate conferences, and served on two panels, all in the vain attempt to place a clinical definition on SDS. I don't think that such a definition will be possible. Additionally, I want to make it clear:  I am a skeptic when it comes to the products on the market today. Let's talk about my somewhat slanted viewpoint:


SDS is Like Tobacco...

While this may sound like a cruel metaphor, it is apt. You can substitute alcohol, or any other recreational drug here, but the relationship would be the same. There is a short term buzz you get from the products, but there are long term problems and dangers that need to be managed. The sellers are fundamentally hoping that you are adult enough to not go on a bender and crash your car, or destroy yourself in some way, so you can keep coming back. There are use cases where the products work, but one should be very wary of assuming that, in a super-competitive industry, storage arrays are so massively mis-priced that it is economically advantageous to construct one from parts. Integration and testing is a huge task that effectively "de-commoditizes" all of the storage products available today. A bug that has a .1% chance of occurring may seem like an acceptable risk with 4 disks. With 100 disks, you have a different calculus to consider.

One of the comments that I heard at SDC is that we have effectively unlearned all the things that we learned 20-30 years ago about creating resilient systems from cheap disks. This isn't entirely true. What we have learned is that there may be better ways to get storage resiliency at very large scale. It's a sexy concept, and those who think they have the physical scale necessary owe it to themselves to try it. Those without cloud-sized data centers need to consider the possibility that the best bet is to rent disk space from companies that have the mass, i.e. Amazon, Google, etc. Trying to convince someone that they can be Google or Amazon if they only just bought your software, is much like selling steroids to people who don't work out. Nice story, but missing key facts.


The Box Huggers Are Not Who You Think

It has been a long-speculated axiom of storage that the people want boxes. If you want to sell storage, you need to put it in a metal box and sell it. Upon much contemplation and discussion on the topic, I have come to the conclusion that his has more to do with economics than anything else: customers are used to buying capacity and sellers are used to selling in the same manner. In fact, there are very few people who have gotten past the notion of paying for something other than capacity. Generally, this is how storage works: you have some data of a particular size, and you want to put it in storage commensurate with that size.  The more hazy your needs are, the more likely you will overbuy and hence overpay for your need.

It should not be a surprise then, that the people who are most religious about clinging to boxes tend to be those who are selling them. In full disclosure, I have to confess that I may be one of this cohort. It really represents the easiest way to comprehend what you are selling and what the customer is buying. You would like to be able to offer X Terabytes of highly available storage, with quantified performance, up to N LUNs, up to M snapshots, etc. This is really an indication of tested limits more than anything else, and it should be construed as a support statement from the vendor to the customer. Without that metal box to test, guaranteeing any level of performance or functionality can quickly spiral into an unbounded problem. Hence, we love boxes more than anyone. Perhaps this is why many perceive that the market is troubled by the encroachment from the cloud scale services such as Amazon's and Google's. Nonetheless, the box hugger's view of the world is that any product without clearly defined and testable performance objectives is fundamentally a toy.


Most Customers Are Actually Data Huggers

In my wanderings over the last decade, I have met many, many customers who buy and manage infrastructure. The one most important commonality among all of them is how tightly the security of their employers' data is tied to their success. The word "security" in this case is not to be understood as only protection from intrusion, but also its availability, performance, and the general control of its destiny. That last item - control - is perhaps the most important of all. Putting their data in the cloud gives them the same detachment that they get from sending their kids off to college. Risking data loss is so unacceptable, that making copies and scattering them in as many places as possible seems to be de rigueur. That is pretty paranoid.

This should be a hint to everyone. There is a mentality evident among many software vendors that their role in providing the aforementioned security can be conveniently redlined above the disks or the hardware platform or the network. I don't think that there is a bias against software only solutions. Rather, it's evident that the data-hugging masses aren't getting the feelings of security that they need to widely adopt the approach. As I said in the panel discussion last week, this is less of a technical problem and more of a business model problem. If a model exists that allows the technology to be delivered to customers with the warm, fuzzy feeling of control, I'm sure this market will find it. I don't see one right now.

So what will happen? Without some major redirection, it's clear that there is going to be a shakeup of sorts in this space. The minority of shops that have the expertise, the mandate, and the spare time, will make their choices. The bulk of those choices will be for free software because that is the easiest way to rationalize the internal support costs of the do-it-yourself approach. In the end, the paying market for many of these solutions will not be large enough to support all the players. The winners will be either open source (i.e. profit-free), or solutions incorporated into existing platforms at no extra cost (watch VMware and Microsoft here). Finally, without addressing the needs of the data hugging mid-market, its hard to see any of these products seeing more than limited acceptance.

I'm pretty flexible with my opinion. I change my mind when the facts before me change. Right now, this is what I believe.

Dispatch From VMworld: Surprise! It's Still All About The Box.

Its only the second day of what is the biggest IT conference out there these days, but I think I can already see the direction that things will be going for the next couple of years. We all like to believe that we are heading onto a post-virtual "cloud" era for computing infrastructure. Certainly, if you listened to the panel participants, we are all going to be devoid of any physical computing infrastructure someday soon. It will all magically migrate to vast data centers in the desert, leaving us free to interact with it through processors in our phones, cars, watches, underwear, etc.

So then I went to lunch with one of my former colleagues from Dell. We talked through the usual pleasantries and then we got into the happenings at the show. "You know," he says, "it's still all about the box." "Yes," I replied. "Everyone is fighting about who owns the box. Nothing has changed."

The box, in this case, is your server, or your storage, or your network. By virtualizing them, we really haven't changed the discussion much. We only changed the venue. The new NSX (nee Nicira) product is an obvious play to redefine the edge of the network, to the exclusion of Cisco. Nowhere was the tension more evident than on the stage, where Cisco distinguished itself simply by it's absence from the list of partners. I've made my opinions on the networking space known elsewhere on this blog, but kudos to VMware if they can actually succeed in shaking up the space a little.

Then there's storage. If you haven't noticed, it's nothing but boxes there. Physical, ever-expanding, power sapping boxes. There, you see a whole host of players trying to collapse your storage into your server, or replace your old storage arrays with new ones that contain flash, or maybe throwing some flash into your server and then collapsing. Of course there are the products that push you to just use servers as storage arrays - VMware's distributed storage software, for example. No part of the box is more hotly contested than this one. Here, it's not clear that there is going to be a single winner, for the simple reason that there is just too much data to manage in too many different ways. (Hence, a reason to gravitate to this market for new ideas)

When you juxtapose the battle for the box against the lofty cloud talk that Andreesen and company were spouting up on the stage, its hard to draw a linear transition from the real world to the vision. Perhaps vision is just that: an aspirational goal that is not intended to be achieved. I'm not sure that I even want cloud-connected boxers. Perhaps for consumer IT, the vision is easily achievable. In enterprise IT, the legal, security, and psychological hurdles just seem to generate far too much inertia to get us there. If you are a strategist, it seems foolhardy to bet that IT will be physically gone in 5 years. Hence, we have to place our bets on some fuzzy, semi-cloudy, hybrid middle ground.

Lots to think about.

Shocking news! Renting More Expensive Than Buying!

I came across a blog post on Wired Enterprise this evening that made me smile. More precisely, I got that warm feeling that only vindication brings after a long argument. That post, written by Cade Metz, was entitled "Why Some Startups Say the Cloud Is a Waste of Money". In it, he frames an argument that I was having just a few weeks ago with certain folks that should know better. I was being called a blasphemer for suggesting it, and clearly, Metz is aware that he is off the beaten path. I need to buy him a beer. Without further ado, here's the money quote:

...in May, about two years after MemSQL was founded, [Eric] Frenkiel and company came down from the Amazon cloud, moving most of their operation onto a fleet of good old fashioned computers they could actually put their hands on. They had reached the point where physical machines were cheaper — much, much cheaper — than the virtual machines available from Amazon. “I’m not a big believer in the public cloud,” Frenkiel says. “It’s just not effective in the long run.”

The post goes on to describe a number of other companies that have reached the same conclusion: At scale, it simply is not economically viable to layer your SaaS or PaaS on top of someone else's rented infrastructure. Oddly, the disproportionate cost of cloud infrastructure is well known to all the startups that I know who are using it. Everyone is moving off of Amazon as far as I can tell. If you run the numbers, it is also pretty clear that AWS is likely reaping quite a fat profit from their operations.

In my case, I was forced to run the numbers because no one believed me. Not a problem. I love math. Here's what they looked like based on pricing available mid June of this year: I could get managed physical server hardware, colocated in several large data centers that would run 20 AWS Linux instances of my specification, at a third the cost. That included the VMware license. Even better, if you wanted to buy the hardware yourself and just rent the rack space, power, and connectivity, it got even cheaper. Based on my math, AWS only makes sense for a small number of compute instances, or when running for a short period of time. One more thing: The two approaches are headcount-neutral. There is no meaningful difference in the number of people and internal operating expenses that go with one way versus the other. 

When you step back from this, you realize the big obvious truth: Nowhere in nature does renting cost less than buying: Not with leasing cars. Not with apartments and real estate. Certainly not with compute resources. Renting has always been a way to avoid major capital expenditures - either because the capital does not exist, or the need is short term. The longer and larger your commitment, the more advantageous it is to use capex if you have it available to you. In the case of the SaaS providers, the name of the game is to reduce service costs. Why let your service providers expropriate your profits if you don't need to? 

What to rent? Buying a giant data center in a large metro is insanely expensive. It makes sense to rent space in someone else's. Renting someone else's PowerEdge R720 maybe does not.

Anyway, I feel better now that I don't feel alone...