In the enterprise data center, the next big thing is always around the corner. But in the case of NVMe Over Fabrics, the hype may prove justified. From almost every angle, NVMe-oF technology, which unlocks the NVMe (Non-Volatile Memory Express) specification within the server and extends the super low latency of flash storage across the data center, appears set to be a game changer.
The Five-Year Outlook
In the first half of the 2020s, developments in enterprise computing will come thick and fast.
The major thing to anticipate is a further sharp uptick in data volume. According to projections from Seagate and IDC, the total number of zettabytes generated annually by 2025 will reach a staggering 175 zettabytes—more than a fivefold increase from 2018.
During that time, flash technology will keep dropping in price and increasing in adoption, accelerated by the emergence of the NVMe protocol.
The global market for NVMe will expand in lockstep with demand. According to research from G2M, the total NVMe market, including NVMe solid state drives and NVMe storage arrays, is set to grow to more than $80 billion by 2022. If the figure proves credible, that constitutes a significant slice of hardware spending for enterprise computing as a whole.
What will also emerge is an approach to enterprise computing disaggregated from individual servers that stretches across the data center and beyond. The fabric will itself become intelligent, tapping storage and compute resources from where they are most readily available with real-time flexibility.
NVMe —A Quick Overview
- The NVMe specification is overseen by non-profit industry group NVM Express.
- The specification is open source. Industry members are invited to join the group and contribute to its future development.
- The most recent version of the NVMe specification (1.4) was published in June 2019.
- The initial specification for NVMe-oF came out in 2016. The NVMe-oF specification “enables the connection of remote subsystems with a flash appliance to achieve faster application response times and better scalability across virtual data centers,” according to NVM Express.
- Although the vast majority of SSD workloads remain based on SAS/SATA protocols, most analysts agree NVMe adoption is set to grow in the next few years as prices fall and server architectures get reworked to accommodate higher proportions of flash.
Eventual Consolidation in an Emerging Marketplace
The NVMe marketplace remains highly fragmented and awash with emerging startups.
There are currently more than 100 companies in the marketplace, shipping approximately 380 NVMe (or NVMe-oF) products, according to G2M Research. This includes 127 models of NVMe SSDs and 121 servers with NVMe drive bays, with every top storage system vendor shipping at least one all-flash array with NVMe SSDs or NVMe-oF target interfaces.
Expect a two-pronged movement in the next five years: continued expansion of startups looking to gain a foothold in the sector set against a slow but persistent drumbeat of acquisitions. Purchases, such as Western Digital’s snapping up NVMe-oF hardware developer Kazan, will form the backdrop to a sector of computing that has the potential to prove dominant in the coming decade.
The development of the NVMe sector will likely follow a similar pathway to that of the hard drive industry, now in a state of advanced maturity consolidated under just three providers: Seagate, Western Digital, and Toshiba. This is a far cry from how the HDD market looked in previous decades with its motley crew of contenders jostling for market position amid a steady stream of mergers and acquisitions. Expect a similar trajectory for the NVMe sector as it enters the beginnings of its own Cambrian explosion.
Amid a diverse range of industry players, key companies seeking to blaze a trail on the back of NVMe include:
Excelero
Founded in 2014, Excelero offers “local performance at data center scale” through NVMesh, its core product offering software-defined storage for NVMe. According to Excelero, NVMesh connects multiple server nodes to enable distributed storage while retaining the ultra low latency of NVMe DAS. Among other sectors, Excelero targets post-production houses in the media and entertainment industry where ready access to large data files is a critical contributor to workflow productivity.
VAST Data
With its gutsy marketing and claims that its “universal storage” solution is the future of enterprise computing (and will ultimately spell the end of the hard disk drive), you might be forgiven for thinking that VAST has been on the block for longer than a year. Nonetheless, it’s hard not to give credit to a firm that backs up its swashbuckling claims with an attractive hardware solution that puts the JBOF into NVMe and promises aggressive scale-out for its VAST NVMe enclosures. VAST is going heavily after AI and deep learning deployments, and with good reason: flash remains the natural choice for AI, and what better way to run AI workloads than within a custom-flash system optimized for NVMe and disaggregation.
Lightbits Labs
New hardware is great, but what about getting more out of the machines you already have deployed? Lightbits Labs’ LightOS promises to unlock the potential of NVMe flash through software-defined storage. “Lightbits is galvanizing the NVMe revolution, providing private clouds and data centers the performance of local SSDs over standard networks with the hyperscalability of public clouds,” the company claims.
Pavilion Data
BYOM (Bring Your Own Media), says Pavilion of its NVMe-oF storage platform. No vendor lock-in and no waiting for a three-year refresh cycle to introduce higher storage capacities into your IT mix. Pavilion says its subscription hardware model gives organizations the flexibility they need to incorporate all-flash storage into the data center without the frustration of the IO bottlenecks and stranded capacity associated with systems not properly optimized for NVMe.
The Drive for Greener Storage: Balancing Competing Forces
Lowering the ecological footprint of enterprise computing is a central goal of any modern, responsible IT program. If you’re in any doubt on this point, examine the hyperscalers—Google, for one, has long been serious about reducing its carbon emissions and prolonging the useful life of its hardware.
When evaluating the eco-impact of NVMe-oF on the data center, it’s important to balance competing forces. With no end in sight to the data explosion, the aggregate environmental footprint of computing is not likely to reduce any time soon, with or without NVMe. Instead, what a technology such as NVMe-oF absolutely enables is greater efficiency within the context of rapid growth.
Take the potential for NVMe-oF-enabled architectures to boost storage optimization. The advent of composability, whereby machines can draw on whatever appropriate compute and storage resource is available to them across an IT ecosystem, will drive up relative cost savings while increasing rack density. It will also help build bridges between remote locations and central data centers, mitigating the potential for stranded capacity at the edge.
Don’t Forget Reuse
With the rise of open source hardware in IT procurement and a decrease in vendor lock-in, the BYOM mentality championed by Pavilion Data will likely gain traction.
At the same time, and as it becomes easier to swap in and swap out technology from NVMe-enabled infrastructure, companies mustn’t overlook the reuse potential of their end-of-deployment hardware.
Whether it’s HDDs that are no longer needed or SSDs earmarked for upgrading, these storage assets retain value through remarketing. End-of-use assets should equally be considered for internal reuse. Not every company function needs the latest, supercharged equipment.
Assessing the Future State: Parallelism Unleashed
As NVME SSD, supported by NVMe-oF, helps unlock the huge parallel transfer capacities of flash, challenges around the bottlenecking of data under SAS-SATA SSD will erode over time.
The rebalancing won’t happen overnight, though. For many use cases, the continued deployment of HDD, at its lower price point and at steadily higher capacities, will work just fine. For its part, Seagate believes HDD will hold on to its price competitive position with SSD for at least another 15 years.
That takes us into the middle of the 2030s, by which time the world of enterprise computing will be significantly more sophisticated, intuitive, and free flowing than today.
Mid-Range Projections
In the coming years, expect to see steady adoption of NVMe protocols in private clouds across a wide range of industries, including those heavily dependent on edge computing, such as oil and gas.
As NVMe-oF works to radically transform storage and compute, creating myriad possible connections among points, we will also see an ongoing simplification around the core of the NVMe specification: what are the basic building blocks of NVMe and what mission is it seeking to serve?
We may even see non-NVMe technologies themselves transformed and assimilated into the emerging ecosystem of NVMe-based computing in different ways.
For more information about how Horizon can help get more out of your data center storage, contact us today.
