The Future of Data Center Storage

Speculating on the future of data center storage is hardly an exact science. This is even more true in a year when economic pressures and low demand have spelled trouble for HDD and SSD alike. However, look closely at the trend lines and some patterns begin to emerge. Most notably, economic conditions haven’t slowed the adoption of cloud storage, cementing the status of the hyperscale data center as critical infrastructure. Here are four indications where data center storage is heading as manufacturers strive to meet fast-changing industry needs.

1 The Shifting Storage Mix

One thing is for certain: demand for data storage will keep growing in the coming years. Analyst firm IDC calculates that global data generation will grow at a CAGR of 21.2% through 2026. This leads to a challenge. Ocient, a hyperscale data analytics solutions company, found in a survey that 62% of respondents believed that the volume of data is growing faster than they can store and analyze it.

What does this mean for the relative prevalence of different media? For the time being, HDD will likely remain the dominant storage technology for storing data cheaply at scale. But this is conjecture, rather than a guarantee: both the HDD and Flash industries are suffering from a lack of demand, and future success depends on how they respond.

Despite a decline in nearline shipments, nearline HDD remains the data storage capacity solution of choice, although flash storage continues to chip away at disk’s share of the expanding pie. Tape is also growing in share from 14% (2018) to 18% (2024) over the same period.

For some analysts, the growth of flash—with its numerous performance upsides—spells the eventual demise of HDD in smaller data centers. “The hard disk will disappear from the small data center,” says Enrico Signoretti. “No more hard drives in the houses, no more hard drives in small business organizations, no more hard drives in medium enterprises as well. These kinds of organizations will rely on flash and the cloud, or only the cloud probably.”

While it’s unclear whether and when SSD will completely replace HDD in smaller data centers, Signoretti is correct on the direction of travel. Flash will become increasingly ubiquitous in smaller data environments. However, in the cloud, hard drives and tape may continue to play an outsize role.

2.   Converging Form Factors and Storage Protocols

Rather like tape, HDD is here for the long haul. However, the interface protocols supporting it—SATA and SAS—are proving increasingly limiting for flash-based technologies, particularly for workloads in hyperscale data centers where flexibility is key. Pressure is building on HDD makers to think in “flash-friendly” ways when designing higher-cap hard drives.

As a case study, consider the increased adoption of shingled magnetic recording (SMR). SMR is designed to increase areal density in HDDs by allowing read/write tracks to overlap each other on the platter in a similar manner to shingles on a roof.

This arrangement increases the density of tracks on the medium, which is good for storage capacity. But the shingling complicates the writing process, since writing to one track can result in the overwriting of valid data on the adjacent (overlapping) track. 

This technical limitation prompted the emergence of a banded, or zoned, approach to SMR. Since SMR prevents overwriting individual tracks, one groups chunks of tracks together into “zones”, which are written, erased, and overwritten as a unit, rather than allowing partial overwriting such as in drives used in traditional RAID arrays.

Sound familiar? This zoned storage is akin to how NAND chips function in solid state drives, with SMR zones analogous to flash blocks. Converging form factors in this way makes subsequent convergence of protocol possible.

Accordingly, Western Digital’s Zoned Storage Initiative encourages the deployment of Storage Zone Technology regardless of whether the device is SAS-SATA (HDD or SSD) or NVMe, the interface protocol specifically designed for SSD. This allows for system-wide performance management and at-scale infrastructure efficiency, factors critical to cloud infrastructure.

With SMR’s zoned approach, new data is appended to existing zones or added to new ones and not partially overwritten in the manner of conventional hard drives, such as those drives used in traditional RAID arrays. Each zone can only be overwritten in full, similar to how data writing works for a solid state drive. 

While for a long time storage manufacturers required SSDs to conform to the SATA-SAS interfaces built for HDD, flash-first thinking—driven by the promise of NVMe—is increasingly front of mind. Over time, expect flash protocols to become the new default, with HDD (and even tape) conforming to this rebalancing of the underlying systems architecture.

In the era of composability, tolerance for landlocked drive capacity will markedly decrease. The storage capacity on HDD must be reasonably available for deployment in a flash-driven world.

Just ask the open source movement, which strongly advocates for unified form factors for flash. Take the growth of EDCSSFF, with SSD form factors free from the physical constraints of HDD. Why should flash storage limit itself to the shape requirements of spinning disks? 

3. NVMe for HDD

What else would flash-first thinking entail? Extending the NVMe protocol, designed with flash in mind, to HDD might seem counterintuitive. How would the slower data transfer rates associated with HDD benefit from the massive parallelism of NVMe-enabled flash?

Nonetheless, there is growing industry interest in exploring what NVMe for HDD might look like. Could such a development deliver a unified protocol for the data center that supports the upsides of flash—low latency and high availability—while seamlessly accommodating a heavy concentration of spinning disks? 

This isn’t just a hypothetical. In 2021, Seagate created an experimental NVMe native HDD, which it connected to a host using a PCIe interface. A year later, Seagate revealed samples of NVMe enabled Exos drives, bringing flash-ified HDD one step closer to the market.

Of course, certain challenges must be overcome before NVMe HDDs come into their own. But as standards work continues, NVMe disk drives will streamline hardware deployment, integration, and operation.

“Seagate’s leadership in implementing native NVMe HDD storage devices will help enable delivery of optimized access to data in support of streamlined storage composability.

–David Allen, NVMe board member and employee of Seagate’s CTO office.

There’s much to gain, as there are many potential advantages of NVMe HDDs, including: 

• enabling a more effective composable data center architecture, with NVMe-HDDs simplifying NVMe-oF hardware and firmware stacks 
• eliminating the protocol translation between SSD and HDD
• putting HDDs behind smart NICs (network interface controllers) 
• performing computational storage with ASIC/FPGA accelerators shared between multiple HDDs  
• better optimizing HDD dual actuator technology, particularly with SATA in mind
• allowing for streamlined protocols which advance security

4. AI and Software-Defined Storage

This ongoing convergence of form factors and interface protocols ultimately leads to standards that are device agnostic. This, in turn, enables the greater application of software to the data center stack. And it’s here where the real benefits of systems optimization kick in.

Of course, talk of the software-defined data center is hardly new. Software already defines the contours of data center computing to a massive degree. Making storage resources programmable and more easily scalable are key elements of a software-driven approach to storage, as NetApp puts it.  

But the ability to unlock crucial capacity in mechanical storage devices and place it on the same playing field as their faster solid state counterparts will materially expand what is possible by bringing the technical reality into greater alignment with the software-driven vision.

That doesn’t mean a total move away from traditional storage architectures, such as NAS and SAN. For one, a strong case for local arrangements will remain in smaller data centers and for meeting niche applications. What’s more, the software around composability remains relatively immature, with a long development road ahead of it. 

Nonetheless, what we will continue to see is a rebalancing away from local storage architectures. Expect to see more open and disaggregated models driven by the large-scale needs of the major cloud service providers.

Data Center Storage: Key Trends

The future of data center storage breaks out into several clear areas:  

• The continued growth of data storage capacities. Despite decreased demand in the short term, HDD will likely remain the dominant technology for hyperscalers while flash grows in prominence and tape deployment similarly expands.
• The increasing adoption of flash-friendly form factors that recognize the significant performance benefits that flash offers over HDD. 
• An ongoing attempt to retrofit the NVMe protocol to SAS and SATA HDD. This will constitute a 180-degree turnaround from the previous decade. Here the emphasis was on rigging solid state drives for HDD interfaces.  
• The ability of data center software to comprehensively access all the possible storage under its watch. This unified view will increasingly extend across the range, from hyperscale data centers to the edge.

Underscoring each of these trends is the inexorable growth of massive cloud data centers. Here we will continue to see the hyperscalers driving the agenda around convergence.

The future of data center storage is a critical component to the commercial success of your organization. From data center decommissioning to secure data sanitization, Horizon Technology can help get more out of your storage hardware while returning funds to your IT budget.