Last month, experts from Dell EMC, Intel, Mellanox and Microsoft convened to take a look ahead at what’s in store for Ethernet Networked Storage this year. It was a fascinating discussion of anticipated updates. If you missed the webcast, “2017 Ethernet Roadmap for Networked Storage,” it’s now available on-demand. We had a lot of great questions during the live event and we ran out of time to address them all, so here are answers from our speakers.
Q. What’s the future of twisted pair cable? What is the new speed being developed with twisted pair cable?
A. By twisted pair I assume you mean USTP CAT5,6,7 etc. The problem going forward with high speed signaling is the USTP stands for Un-Shielded and the signal radiates off the wire very quickly. At 25G and 50G this is a real problem and forces the line card end to have a big, power consuming and costly chip to dig the signal out of the noise. Anything can be done, but at what cost. 25G BASE-T is being developed but the reach is somewhere around 30 meters. Cost, size, power consumption are all going up and reach going down – all opposite to the trends in modern high speed data centers. BASE-T will always have a place for those applications that don’t need the faster rates.
Q. What do you think of RCx standards and cables?
A. So far, Amphenol, JAE and Volex are the suppliers who are members of the MSA. Very few companies have announced or discussed RCx. In addition to a smaller connector, not having an EEPROM eliminates steps in the cable assembly manufacture, hence helping with lowering the cost when compared to traditional DAC cabling. The biggest advantage of RCx is that it can help eliminate bulky breakout cables within a rack since a single RCx4 receptacle can accept a number of combinations of single lane, 2 lane or 4 lane cable with the same connector on the host. RCx ports can be connected to existing QSFP/SFP infrastructure with appropriate cabling. It remains to be seen, however, if it becomes a standard and popular product or remain as a custom solution.
Q. How long does AOC normally reach, 3m or 30m?
A. AOCs pick it up after DAC drops off about 3m. Most popular reaches are 3,5,and 10m and volume drops rapidly after 15,20,30,50, and100. We are seeing Ethernet connected HDD’s at 2.5GbE x 2 ports, and Ceph touting this solution. This seems to play well into the 25/50/100GbE standards with the massive parallelism possible.
Q. How do we scale PCIe lanes to support NVMe drives to scale, and to replace the capacity we see with storage arrays populated completely with HDDs?
A. With the advent of PCIe Gen 4, the per-lane rate of PCIe is going from 8 GT/s to 16GT/s. Scaling of PCIe is already happening.
Q. How many NVMe drives does it take to saturate 100GbE?
A. 3 or 4 depending on individual drives.
Q. How about the reliability of Ethernet? A lot of people think Fibre Channel has better reliability than Ethernet.
A. It’s true that Fibre Channel is a lossless protocol. Ethernet frames are sometimes dropped by the switch, however, network storage using TCP has built in error-correction facility. TCP was designed at a time when networks were less robust than today. Ethernet networks these days are far more reliable.
Q. Do the 2.5GbE and 5GbE refer to the client side Ethernet port or the server Ethernet port?
A. It can exist on both the client side and the server side Ethernet port.
Q. Are there any 25GbE or 50GbE NICs available on the market?
A. Yes, there are many that are on the market from a number of vendors, including Dell, Mellanox, Intel, and a number of others.
Q. Commonly used Ethernet speeds are either 10GbE or 40GbE. Do the new 25GbE and 50GbE require new switches?
A. Yes, you need new switches to support 25GbE and 50GbE. This is, in part, because the SerDes rate per lane at 25 and 50GbE is 25Gb/s, which is not supported by the 10 and 40GbE switches with a maximum SerDes rate of 10Gb/s.
Q. With a certain number of SerDes coming off the switch ASIC, which would you prefer to use 100G or 40G if assuming both are at the same cost?
A. Certainly 100G. You get 2.5X the bandwidth for the same cost under the assumptions made in the question.
Q. Are there any 100G/200G/400G switches and modulation available now?
A. There are many 100G Ethernet switches available on the market today include Dell’s Z9100 and S6100, Mellanox’s SN2700, and a number of others. The 200G and 400G IEEE standards are not complete as of yet. I’m sure all switch vendors will come out with switches supporting those rates in the future.
Q. What does lambda mean?
A. Lambda is the symbol for wavelength.
Q. Is the 50GbE standard ratified now?
A. IEEE 802.3 just recently started development of a 50GbE standard based upon a single-lane 50 Gb/s physical layer interface. That standard is probably about 2 years away from ratification. The 25G Ethernet Consortium has a ratified specification for 50GbE based upon a dual-lane 25 Gb/s physical layer interface.
Q. Are there any parallel options for using 2 or 4 lanes like in 128GFCp?
A. Many Ethernet specifications are based upon parallel options. 10GBASE-T is based upon 4 twisted-pairs of copper cabling. 100GBASE-SR4 is based upon 4 lanes (8 fibers) of multimode fiber. Even the industry MSA for 100G over CWDM4 is based upon four wavelengths on a duplex single-mode fiber. In some instances, the parallel option is based upon the additional medium (extra wires or fibers) but with fiber optics, parallel can be created by using different wavelengths that don’t interfere with each other.
Unlike traditional local or scale-up storage, scale-out storage imposes different and more intense workloads on the network. That’s why the SNIA Networking Storage Forum (NSF) hosted a live webcast “Networking Requirements for Ethernet Scale-Out Storage.” Our audience had some insightful questions. As promised, our experts are answering them in this blog.
Q. How does scale-out flash storage impact Ethernet networking requirements?
A. Scale-out flash storage demands higher bandwidth and lower latency than scale-out storage using hard drives. As noted in the webcast, it’s more likely to run into problems with TCP Incast and congestion, especially with older or slower switches. For this reason it’s more likely than scale-out HDD storage to benefit from higher bandwidth networks and modern datacenter Ethernet solutions–such as RDMA, congestion management, and QoS features. Read More