Providers of all-flash arrays believe they can make HDD a thing of the past. Their argument: that denser and more efficient storage solutions will eventually lower the TCO of flash below that of HDD.
It’s been a turbulent time for the storage industry. HDD manufacturers are feeling the squeeze due to decreased demand. Meanwhile, a surge of interest in AI is driving demand storage solutions which allow low-latency operations.
All-flash storage proponents like Pure Storage and VAST Data are seizing the moment to push their case. In bringing the fight to HDD, the prophets of an all-flash future argue that flash storage, as it figures in their own products, has a lower total-cost-of-ownership (TCO) than disk-based solutions.
Few would doubt that all-flash arrays are good for suitable use-cases. The question is whether price considerations will eventually let flash supplant disk drives across the board. To find out how much a threat all-flash data centers pose to HDD manufacturers, we need to cut through the noise and separate temporary market trends from enduring price differences.
Price Parity: Price-per-TB vs TCO
When it comes to comparisons of cost, the term “price parity” is frequently bandied about. This is reached when flash and HDD are sold at roughly the same price-per-TB. Lower cost is the most important factor in HDDs remaining the backbone of large datacenters. This means that flash, and hence all-flash arrays, will be the more attractive option once price parity is reached.
Of course, both SSDs and HDDs come in a variety of capacities, form factors, and intended uses. This means that in practice, it makes sense to be more fine-grained when discussing price parity. For example, Tom’s Hardware reported at the end of last year that 512GB SSDs hit price parity with 500GB HDDs for performance laptops.
SSDs are slowly supplanting disk storage in laptops, with TechRadar reporting that three of the largest laptop manufacturers (Dell, Lenovo, and HP) will no longer sell HDD-equipped laptops in the U.S. On the software side, Microsoft is pressuring OEMs to use SSDs in Windows 11 PC pre-builds.
Lowering SSD prices mean HDD-equipped laptops are increasingly rare. But this doesn’t imply that HDDs for data centers are going anywhere. Last year, TechRadar estimated that the cost of the cheapest 1TB and 2TB SSDs are within striking distance of the cost of 1TB and 2TB HDDs. However, they also concluded that the prices of higher capacity HDDs are unchallenged—for now.
While price parity might be sufficient for flash to supplant HDD, it is only one way for flash to be cheaper. Companies like Pure Storage emphasize comparisons of TCO (total cost of ownership). Even if flash has a higher price-per-TB, it can save money overall. This is either by saving on power or cooling costs, or by enabling efficient compute which saves money in the long run.
Spinning Media Spirals Downwards
It’s not a new question: industry watchers have long wondered whether or not HDD will continue to dominate in certain use cases, or whether flash will eventually supplant HDD everywhere. However, HDD doomsayers have found that recent developments have provided more grist for the mill.
It’s been a rough year for HDD. In 2Q2022, HDD unit shipments had plummeted by 33% YoY. The biggest drop, at 40%, was in the 2.5” form factor. Meanwhile, enterprise HDD shipments remained relatively flat. However, this changed: even nearline drives were down 42% y-o-y over Q4. While some of the damage was due to decreased PC demand, the dip in nearline drive shipments indicated a larger trend.
Drops in NAND Demand
However, this decline is hardly sign of HDD making it’s way towards the exit. NAND flash prices have been in freefall, and have led chip manufacturers to slash production. Fabs and designers alike have faced huge losses in the face of decreased demand.
Indeed, prices may have fallen even faster in 2022 if it hadn’t been for a flash contamination incident. A fab co-run by WD & Kioxia saw the loss of as much as 16EB of flash.
What gives? For some, such as Micron, it’s due to a glut of supply after frenetic production during the economy’s post-pandemic bounce-back. Even more broadly, high interest rates have ramped up pressure on the whole tech industry, chipmakers included. In other words, lower flash prices aren’t just due to lower production costs.
However, not everyone is hurting equally, and all-flash companies have seen growth even as other companies struggle. VAST Data was named by Deloitte as the fifth fastest growing tech company. Pure Storage has also seen sustained revenue growth, although it continues to post modest losses.
While a far-cry from the economic turmoil at the pandemic’s height, it’s still proven difficult to predict tech trends in our new normal. It may be that at the end of the day, the fate of HDD won’t be due to transparent trends in drive demand, but to innovation in how flash is used, and whether proponents of all-flash arrays proponents can make good on their promise to lower the TCO of their solutions.
Pure Predicts TCO Crossover
With high capacity nearline drives becoming an ever larger proportion of shipped HDD capacity, the future of HDDs depends on their cost efficacy. Just as tape retains a niche position in cold storage, some predict that high-cap disk drives will remain crucial in storing data cheaply at scale.
RELATED READING
Though an old technology, tape is still the go-to option for archival storage.
Pure Storage disagrees. It insists that the disk/flash TCO crossover is closer than you may think.
“For the last 15 years, it has always been the next [hard drive] tier to fall, but now there is only one tier left, and only one reason left to use disk, and that’s cost.” says Peter Kirkpatrick, VP of Hardware Engineering at Pure Storage. “With this crossover, it will be the last tier of drives to fall.”
His reasoning: while high-cap HDD is still much cheaper than flash of similar capacity, the price-per-bit of NAND is decreasing faster than that of HDD. Price-per-bit parity is still a ways off, but the TCO crossover is imminent. Flash is denser, allowing for savings in power, cooling, and space.
Pure Storage also emphasizes that performance can have a significant impact on effective cost. Greater reliability reduces the amount of redundancy one must build into one’s systems. Flash also allows for much quicker rebuild times in the event of failure.
Performance enables more efficient use of capacity. “Yes, your spinning drives may hold 20TB, but if the performance of the system taps out and they’re all only 16TB full, you aren’t getting as effective a cost-per-bit.” says the firm. “Flash doesn’t have nearly as much trouble in this regard, especially because read performance stays predictable, even as drives fill up, unlike HDD”.
All-Flash Storage as a Self-Fulfilling Prophecy
Rather than sit back and watch a flash-dominant future unfold, Pure Storage is doing its best to self-fulfill their prophecy with its own all-flash solutions. Its Pure FlashArray and FlashBlade operating systems are built upon Pure’s self-manufactured Direct Flash Modules (DMFs), which are effectively a collection of NAND chips. The firm’s lofty goal is to produce 300TB SSDs by 2026.
“The plan for us over the next couple of years is to take our hard drive competitive posture into a whole new space. Today we’re shipping 24 and 48TB drives. You can expect … larger and larger drive sizes with our stated ambition here to have 300TB drive capabilities, by or before 2026.”
一Pure CTO Alex McCullen.
VAST Ambitions for All-Flash Storage
Pure Storage isn’t alone in chasing an all-flash future. VAST Data is also finding ways to reduce the TCO of flash-based data center solutions. Since VAST’s Hardware is made by third parties, much of its innovation has come via creative techniques to improve data reduction.
VAST’s architecture uses similarity hashes to increase effective capacity by an average ratio of 3:1. This means that a VAST customer with 12PB of raw flash capacity has 36PB of effective capacity. Again, allowing companies to do more with less cuts down on power, cooling, and space requirements.
The firm is now seeking ways to make the reduction ratio even higher. Internal testing with techniques such as “adaptive chunking” have shown that an additional 25% data reduction is feasible. CMO Jeff Denworth believes that the average data reduction ratio will rise to 4:1 or 5:1 sometime this year.
VAST has a helpful framework for reducing the TCO, which it believes has allowed up to 90% savings in reduced energy costs compared to legacy systems.
For his own part, Denworth is confident that this strategy will continue to lead towards increased savings.
“Next generation all-flash doesn’t just provide new opportunities to realize significant storage power efficiency gains, but the conversion to scalable silicon storage also makes compute resources much more efficient” he explains. “The result: even greater power savings across the data center.”
AI And The Need for Speed
The past decade has seen an explosion of interest in artificial intelligence. With stunning advances in large language models like ChatGPT dominating headlines, this interest will only increase. Since machine learning involves training programs on vast amounts of collected data, being able to access and process this data at lightning speed can help give AI firms a competitive edge.
A study by the Enterprise Strategy Group has indicated rapid adoption of all-flash object storage, with the acceleration of AI/ML environments as one of the biggest drivers. All-flash proponents are well aware of this trend, with both VAST Data and Pure Storage emphasizing how their models can be used for AI/ML applications.
Trends in AI and big data are clearly fueling interest in all-flash arrays. However, for all its popularity, not all firms are equally focused on AI. Even for those which are, HDD may still remain preferable for data storage where low-latency solutions aren’t at a premium.
More Food For Thought
All-flash storage is advancing fast. But at least for now, arguments from all-flash gurus are suggestive rather than decisive. A Gartner report for 2Q222 showed that all-flash arrays were almost five times as expensive as hybrid and HDD arrays, though it predicted a crossover in total revenue sometime this year.
Prices have fallen since that Gartner report. A great deal depends on whether flash prices stay low, how fast HDD can innovate, and how difficult it is to transition a storage base from HDD to SSD. These trends remain hard to predict.
One point in favor of HDD is that the industry is more vertically integrated than the NAND industry. When it comes to producing NAND flash, there are a lot of players involved, with design, wafer production, and testing generally carried out separately. This makes it harder for NAND companies to deal with inventory in the face of fluctuating demand.
Finally, a great deal rides on the ability of disk drive manufacturers to increase areal density via HAMR and MAMR technologies. Seagate has shipped samples of HAMR drives, but these have yet to see widespread release. Meanwhile, while Toshiba’s used FC (flux controlled) MAMR in a drive released last year, it was the addition of a 10th platter which allowed the firm to surpass 20TB for the first time.
In short, while HAMR & MAMR may some day be available cheaply and at scale, the ball is very much in the court of HDD manufacturers.
RELATED READING
Western Digital, Seagate, and Toshiba are all innovating in a bid to make sure HDD remains the backbone of hyperscale data centers for a long time to come.
A Complex Present and an Open Future
Short term, the picture is fairly clear. It’s been a rough year for HDD manufacturers, while all-flash storage companies like VAST Data have seen strong growth. But short term wins and woes are much easier to find than entrenched long-term trends. In particular, it’s too soon to tell whether all-flash solutions will become more cost effective than high-capacity HDD within the next few years.
Breathless predictions of HDD’s imminent demise are difficult to evaluate. Easier is knowing what the continued success of spinning media depends upon. Simply put, the fight to mass-produce HAMR or MAMR drives cheaply is an existential one for the HDD industry. Another factor is how far and how fast AI/ML applications become essential to everyday data centers, as this makes all-flash arrays more attractive.
In the end, the fate and future role of HDD may depend not on inexorable forces of technological development, but on the canny business decisions and market positioning of players involved.
Get in touch with Horizon to see how you can repurpose or resell end-of-use data center hardware to reduce the total cost of ownership.
