For much of the last year, vendors have been discussing end-to-end NVMe, but have not had anything tangible outside of statements of direction and “technology demonstrations”. With the announcement of end-to-end FC-NVMe solutions from both Dell-EMC and NetApp – and planned support from the other major storage vendors including Hitachi, HPE, IBM – clearly, NVMe over fabrics is getting real.
NVMe, or Non-Volatile Memory express is a way to transport data with lower overhead and latency than traditional storage access protocols are able to deliver. The SCSI protocol was developed in the 1980s when 10 milliseconds of latency for storage was considered good, thus 500 microseconds of SCSI overhead was irrelevant.
Over the last 10 years, storage media latency has dropped dramatically, with NAND Flash media providing access times under 100 microseconds. New storage class memory such as Intel’s Optane are capable of 10 microsecond latency, which is 1,000 faster than disk drives typically deliver. However, access protocol latencies must be reduced in order for these new media types to deliver an advantage.
The transition to NVMe devices is important. However, changing from SCSI to NVMe was the easier problem to solve, as it did not require cooperation with other vendors, or finicky things like standards and interoperability. NVMe over fabrics, on the other hand, requires cooperation between multiple parties including NIC, HBA and switch vendors, plus storage devices and operating environments including Windows, Linux and VMware.
Storage networking device vendors such as Broadcom (including both Brocade and Emulex divisions) and Cavium (Qlogic) had previously announced support for NVMe over Generation 6 Fibre Channel switches and HBAs. With two out of three pieces of the puzzle in place (HBAs and switches), all that was needed was for a vendor to announce a storage system that supported NVMe over a fabric. While several startups have made announcements, it was not until now that major storage vendors have stepped in with official support.
As I have written in previous blogs, NVMe will ultimately operate on multiple fabrics, including Ethernet based (RoCE), InfiniBand and Fibre Channel. Having options for the transport fabric is good, as it enables choice for both vendor and (more importantly) for IT users.
Unfortunately, there has been a lot of debate and hype surrounding which network or fabric is the best choice for NVMe. That choice needs to provide low latency and high bandwidth while being more efficient through use of zero buffer copies and direct memory access (DMA) techniques.
Fibre Channel has used zero buffer copy and DMA to the host bus adapter almost since the inception of fibre channel. This was not the case with Ethernet, as historically it copes data before transferring from the network interface. Moving data directly from application space to a fabric is critical for low latency and scalable performance.
Recently, a new Ethernet fabric known as RoCE (RDMA over Converged Ethernet) was designed to overcome earlier networking and TCP/IP inefficiencies, by eliminating buffer copies and enabling DMA transfers. Although end-to-end NVMe solutions using RoCEv2 and InfiniBand have been demonstrated, the first production NVMe over fabric deployments with established storage vendors appear to be using Fibre Channel as the fabric of choice.
For many environments, running NVMe over Fibre Channel will be the path of least resistance, particularly for enterprises, as most have existing Fibre Channel equipment deployments. By upgrading to the latest Gen 6 equipment, Fibre Channel remains 100% backward compatible with existing FC attached storage, while also supporting NVMe access for the newest storage gear. In contrast, RoCEv2 requires new equipment and InfiniBand remains a niche protocol outside of high-performance environments.
Thus, the path of least resistance to production NVM over fabric to NVMe target devices may be FC, which can leverage existing FC tools and management processes. Rarely do technologies that require massive investments and changes win out over less intrusive alternatives. The path of least resistance is a successful path both for IT vendors and users.
For IT architects and application owners that have been pushing the limits of existing storage technology, NVMe and NVMe over Fabric will enable new workloads, faster processing and ultimately more efficient operations. NVMe has been a technology for those on the cutting edge for several years, now it’s time for real production deployments with enterprise vendors and equipment.
For mainstream IT users, many of whom may have heard of NVMe and know that it has some advantages, the availability of end-to-end NVMe over fabric will not have an immediate impact. The keyword is “immediate” because over time it is likely that NVMe will replace SCSI as the dominant storage protocol, due to its significant advantages. System vendors will incorporate NVMe into their designs, just as they have incorporated NAND flash into nearly all enterprise storage products. The adoption rate of NVMe and NVMe over fabrics may occur as rapidly as the shift from rotating drives to flash for enterprise storage systems.
As new solid-state storage media technologies evolve, products using phase change memory and other “storage class memory” such as Intel’s Optane will continue to drive the need for lower latency methods for accessing shared storage over a network fabric. The NVMe protocol is likely to become the new enterprise standard for storage access as SCSI fades away. Nothing happens overnight and SCSI will continue to exist and be supported for perhaps a decade or longer. But, make no mistake, NVMe is the heir apparent, it is here and now it is shipping.
