Why flash in the cloud will shake up enterprise storage

flash in the cloud
The big news in storage lately is that EMC is acquiring XtremIO. XtremIO is crafting a next-generation storage array, purpose built for 100% solid-state flash memory. EMC is the market leader in the $30 billion global market for disk-based storage. In acquiring a nascent, unabashedly competitive product to its existing mainstay disk-based storage product line, EMC is foreshadowing the impending flash revolution in data center storage.

Flash memory has already transformed consumer devices like smartphones, tablets, and ultra-notebooks (think MacBook Air). These devices use a solid state drive, also know as SSD or flash drive, rather than traditional mechanical disk. Presumably, if you have a flash-based device, you don’t miss those quaint whirring and chirping noises that a PC’s hard drive makes, and flash also delivers radically better performance and battery life. What you may not know is that a similar shift is already underway in the data center: Google’s instant search and Facebook’s performance intensive applications are powered by flash rather than hard drives.

What’s going on is that storage based on mechanical disk is becoming antiquated according to Moore’s Law: While CPU (and hence servers and networks) get denser, cheaper, and faster, disk only gets denser, cheaper, and slower. (By packing ever more data on a disk whose mechanical performance is flat, it’s as if we are making a bus 1,000 times longer but not adding any doors.) From the perspective of a CPU, then, hard drives today look slower than tape did two decades ago. As Turing Award winner Jim Gray put it several years ago, disk is the new tape, and flash is the new disk.

Gartner recently forecast that this all-flash storage market (i.e., storage with no spinning disk at all) is going to reach $4 billion by 2015. It’s not at all surprising, then, that EMC wants a piece of the action. What is surprising is that EMC has already shipped more enterprise flash within its disk-centric arrays than any other vendor. The fact that it is acquiring a competitive architecture so that it can sell all-flash technologies in the future is indicative of an impending sea change. We can draw two conclusions from EMC buying XtremIO that will have a broad impact on the storage market:

1.     Storage architectures designed for mechanical disk are not a fit for all-flash. I’ve  long believed that taking proper advantage of flash requires a comprehensive redesign of the array hardware and software. For years, EMC has been selling flash cache and tiers as performance accelerators for its disk products. With XtremIO, EMC has endorsed the view that all-flash storage demands entirely new products that shed the legacy of mechanical disk.

2.     Deduplication technology is key for all-flash. XtremIO was pursuing a solution similar to what my own company, Pure Storage, does that uses deduplication to both lower cost and increase performance. With flash memory, reads are very fast and writes are relatively slower, so it does not make sense to write the same data over and over again to flash the way it is done to optimize the performance of disk.

In the decade ahead, mechanical disk will get pushed out of performance-critical workloads, much in the way disk displaced tape for backup over the last 10 years. It is ironic that when tape started failing to keep up, the industry proposed Virtual Tape Libraries (VTL) in which a performance-enhancing “disk cache” was placed in front of the large tape silos – which is directly analogous to the hybrids of flash and disk the incumbent vendors are selling today. Instead, the all-disk approach prevailed with Data Domain’s launch of a solution that was purpose-built for disk and that relied upon deduplication (and compression) to bring the cost of disk inline with the cost of tape. These are precisely the trends EMC just reinforced for flash storage with their XtremIO acquisition, and they run counter to EMC’s focus on selling hybrid arrays.

We believe history is poised to repeat itself in the all-flash market. If you want to accelerate the transition from a slow, cheap storage media (tape, disk) to a faster, more expensive one (disk, flash), find a way to take cost out of the equation. You can go 100% flash at or below the cost of what most customers are paying for fast disk. With all-flash storage, 10X faster, more space and power efficient, and simpler than performance disk arrays, why buy disk?

The next few years will be the most exciting in storage for decades. The biggest winners will be customers, as they push 20th century mechanical storage and all of the associated complexity out of the critical performance path of their business and find they can do so within the constraints of their existing budgets.

Scott Dietzen is CEO of Pure Storage.