Hu Yoshida

Have we reached the tipping point for Flash?

Blog Post created by Hu Yoshida Employee on Jan 24, 2016


During my university days I worked on the night shift operating the latest IBM computers at the time, loading in the programs on punched cards and mounting and dialing in the tapes of input data. Then I went into the military, and spent 5 years in a rifle company far removed from any computer technology. When I was discharged, I went to find a job with IBM based on my experiences of 5 years ago. The first thing I was asked in the interview was what I knew about DASD.  I was dumb founded. The punched cards and spinning tapes that I knew 5 years ago were being replaced by this thing called a Direct Access Storage Device! In the 40 plus years since then I have often wondered if anything could replace the disk storage technology in such a short span of time.


Now I believe this is happening with flash.


The game changing difference between tapes and disk was the ability to access data randomly. The game changing difference between disk and flash is software that enables the ability to access data virtually. Flash is a software device compared to disk, which is a hardware device. Data is no longer tied to sectors on tracks, subject to rotational and seek latencies, and expensive rebuilds when a sector or track fails. With flash, data is mapped to cells, and the user does not know the physical location of the cell. If a cell goes bad the data can be remapped to another cell. While the durability of flash is much lower than disks, it can use internal spare memory to increase the overall durability and lower maintenance costs. Software can add new functions to the flash device or offload  work from the storage controller. Hitachi has added software in our second generation Flash Module Drive (FMD DC2) to compress data in the device without any impact to the storage controller or the application.


The physical mechanics of disk, rotating media and a single access arm, required the physical mapping of data to blocks and sectors.  With flash, the mapping is flexible and can be changed without the user being disrupted.  It is not confined to a given geometry. In order to drive down storage costs, any storage technology must be able to increase increase bit densities. Disk had a fabulous 50 year run where they were able to double bit densities every 18 months. But that run seems to be over. Flash has the advantage of being programable and can scale in multiple directions beyond the one dimension of mechanical disks. Flash cells can contain multiple signal levels, from one bit per cell (SLC), to two bits (MLC), to three bits  (TLC), which can provide 8 discrete signal levels in one cell. Flash can also be stacked vertically to increase data density and lower bit costs.  At the last Flash Summit, in Santa Clara, Toshiba claimed that it would be possible to have  a 3D 128 TB flash devices by 2018. The best we can expect from disks in this time frame is 10TB, which will require additional platters and R/W heads, as well as shingled magnetic recording that will impact performance. You can easily imagine the environmental costs of 12 disk versus one 128 TB flash drive and each year the gap will widen.


Our portfolio of VSP storage systems has already closed the price gap with 15K and 10K performance disks with our 6.4TB Flash modules that use the current MLC technology. Notice how we closed the price per bit gap with lower maintenance (it costs less to maintain fewer flash drives) and the ability to add compression into the FMD module so that it is always on with no impact to performance.

SSD vs disk.jpg

We realize that not every one is able to use banks of 6.4 TB flash drives, so we recently announced the Hitachi Flash Storage (HFS) a 2U flash appliance that starts with 10 x 1.6TB SSDs and can scale to 60 SSDs in the same 2U appliance. With compression and deduplication this can scale from 64 TB to 384 TB of effective capacity. This brings the price point down to about $1.5/GB of effective capacity. (For more details about our HFS and how we position it with our VSP line of flash storage see Bob Madaio’s post.)


The roadmap for Flash, and the continuous improvement in data reduction technologies, are further indications of how quickly disk will be displaced. Manufacturing costs are driven by volumes, and the volumes for flash are increasing at the expense of disk especially in the commodity markets for PCs and compute devices. Enterprise disk manufacturers can no longer depend on commodity disk volumes to offset their manufacturing costs. While enterprise flash volumes are still a small percent of the overall flash market, enterprise flash vendors can leverage their manufacturing costs across the commodity market.


As flash becomes ubiquitous the current class of  storage known as "All Flash Arrays" will disappear. The vendors who only sell these types of arrays with limited functionality will need to increase their portfolios or cease to exist.,  There is a huge inventory of disk already out there, which will be around for some time. Hybrid storage arrays that can virtualise or front end the disks so that most of the advantages of flash performance can be realized with legacy systems will be required during the transition. As new flash devices and future non-volatile memory technologies become available, hybrid storage arrays with virtualisation and dynamic tiering will be required to ensure a smooth transition. We are seeing some customers who have already committed to go all in with flash. One of our customers was running out of power and floor space in their data center. Rather than spend $200m to build another data center, they decided to convert their enterprise storage systems to flash. The conversion cost less than $200m and it left them with plenty of room for growth. Another customer converted to all flash two years ago, when the costs for flash were still much higher than disk, because it was easier to manage and performance was greatly improved. The operations manager said that he was feeling more rested since he no longer had to respond to those 3 am calls complaining about slow response times.


Although flash storage has an order of magnitude higher performance than mechanical disk, we have not realised its full potential. Since flash is virtual, access no longer needs to be sequential, burdened with long queues of requests. The flash device sees all the cells and knows what is in them so it can go directly to the data that is requested. This makes it possible to serve multiple requests for data. With current SCSI commands this is not possible but new protocols like NVMe (Non-Volatile Memory express) and SCSI express will be able to exploit this in the near future, further adding to the demise of disk. While the discussion today is around flash non-volatile memory (NVM), there are future NVM technologies which are even faster and more durable than flash. These technologies will blur the boundaries between memory and storage and will require these new protocols. We may be having these same types of discussions 5 years from now or even sooner.


I believe we have reached the tipping point for flash. What do you think?