Benefits of Disaggregating NVMe Storage with NVMe-oF Technology

By Mark Miquelon, Western Digital

The motivations and benefits of accessing SSD storage using NVM Express^® (NVMe^®) architecture are well known and often discussed. High bandwidth, extreme IOPS rates on mixed read/write workloads and consistently low latency make NVMe SSDs the best media on which to run your transactional applications. Even when we narrow our focus of discussion down to enterprise and data center class NVMe devices, we find a broad product offering from a large number of vendors, an array of endurance classes, and even support for PCI Express^® (PCIe^®) 3.0, PCIe 4.0, and soon to be PCIe 5.0 architectures.  All of this is good news for consumers and IT users!

However, in some cases, those who deploy NVMe SSDs become limited by the PCIe infrastructure. PCIe lanes are always highly valued within today’s servers. Multiple add-in-card slots consume up to 16 lanes each, PCIe devices soldered onto the motherboard require lanes, and the growing number of SSD slots in U.2 and now EDSFF form factors require multiple PCIe lanes to access storage. Architects and designers of today’s server platforms take great care to utilize the available PCIe lanes, and in some cases add switches to fan out lanes to an even greater number of devices. With all these options, data center administrators can often right-size a server to the intended application’s demand. What do you do when the data set or the application demand more capacity or bandwidth than a single server can provide? What if your application demands change over time?

Enter NVMe over Fabrics (NVMe-oF™) technology. Disaggregation of storage from servers is nothing new. SCSI, SAS and Fibre Channel interfaces have enabled this for decades. Disaggregation allows any size storage array to be availed to any number of compute resources running applications via the fabric. Storage resources and compute resources can be scaled independently, greatly easing the difficulty of adding resources to a data center. The NVMe-oF specification defines extensions to the NVMe interface that enable operation over a fabric other than PCIe technology, enabling disaggregation of NVMe SSDs. Specifications describing operation on both Ethernet and Fibre Channel fabrics are available. All the IOPs and throughput of the NVMe SSDs remains available on the fabric. Latency is slightly, but not significantly higher than with direct attached NVMe technology.

NVMe-oF technology based flash arrays are available that provide hundreds of TB of flash storage with just 10s of micro-seconds of latency to your applications. Namespaces can be created, deleted and even grown in size without any disruption to your compute environment. NVMe-oF architecture establishes a foundation for a new composable architecture, which brings previously unseen levels flexibility to data center configuration and control, but that is best left to another blog!

About Mark Miquelon

Mark Miquelon is Director of Partner Alliances for Western Digital’s Platforms business.  In this role he seeks out strategic partnerships that provide additional value to customers consuming JBOD/JBOF and servers developed by Western Digital.