How is 10GBASE-T Being Adopted and Deployed?

David Fair

For nearly a decade, the primary deployment of 10 Gigabit Ethernet (10GbE) has been using network interface cards (NICs) supporting enhanced Small Form-Factor Pluggable (SFP+) transceivers. The predominant transceivers for 10GbE are Direct Attach (DA) copper, short range optical (10GBASE-SR), and long-range optical (10GBASE-LR). The Direct Attach copper option is the least expensive of the three. However, its adoption has been hampered by two key limitations:

- DA’s range is limited to 7m, and

- because of the SFP+ connector, it is not backward-compatible with existing 1GbE infrastructure using RJ-45 connectors and twisted-pair cabling.

10GBASE-T addresses both of these limitations.

10GBASE-T delivers 10GbE over Category 6, 6A, or 7 cabling terminated with RJ-45 jacks. It is backward-compatible with 1GbE and even 100 Megabit Ethernet. Cat 6A and 7 cables will support up to 100m. The advantages for deployment in an existing data center are obvious. Most existing data centers have already installed twisted pair cabling at Cat 6 rating or better. 10GBASE-T can be added incrementally to these data centers, either in new servers or via NIC upgrades “without forklifts.” New 10GBASE-T ports will operate with all the existing Ethernet infrastructure in place. As switches get upgraded to 10GBASE-T at whatever pace, the only impact will be dramatically improved network bandwidth.

Market adoption of 10GBASE-T accelerated sharply with the first single-chip 10GBASE-T controllers to hit production. This integration become possible because of Moore’s Law advances in semiconductor technology, which also enabled the rise of dense commercial switches supporting 10GBASE-T. Integrating PHY and MAC on a single piece of silicon significantly reduced power consumption. This lower power consumption made fan-less 10GBASE-T NICs possible for the first time. Also, switches supporting 10GBASE-T are now available from Cisco, Dell, Arista, Extreme Networks, and others with more to come. You can see the early market impact single-chip 10GBASE-T had by mid-year 2012 in this analysis of shipments in numbers of server ports from Crehan Research:

 

Server-class Adapter & LOM 10GBASE-T Shipments

Note, Crehan believes that by 2015, over 40% of all 10GbE adapters and controllers sold that year will be 10GBASE-T.

Early concerns about the reliability and robustness of 10GBASE-T technology have all been addressed in the most recent silicon designs. 10GBASE-T meets all the bit-error rate (BER) requirements of all the Ethernet and storage over Ethernet specifications. As I addressed in an earlier SNIA-ESF blog, the storage networking market is a particularly conservative one. But there appear to be no technical reasons why 10GBASE-T cannot support NFS, iSCSI, and even FCoE. Today, Cisco is in production with a switch, the Nexus 5596T, and a fabric extender, the 2232TM-E that support “FCoE-ready” 10GBASE-T. It’s coming – with all the cost of deployment benefits of 10GBASE-T.

Note: There is a poll embedded within this post, please visit the site to participate in this post's poll. Note: There is a poll embedded within this post, please visit the site to participate in this post's poll. Note: There is a poll embedded within this post, please visit the site to participate in this post's poll.

SSSI Highlighting PCIe SSDs at the Storage Visions Conference

Marty Foltyn

Join the SSSI at the Storage Visions Conference, January 6-7,2013 at the Riviera Hotel in Las Vegas, NV.  With a theme of Petabytes are the new Terabytes, the 2013 conference will explore the convergent needs of digital storage to support cloud content distribution and sharing, user- generated content capture and use, and professional media and entertainment applications.

The SSSI booth is #6 on the Exhibit floor, and will showcase a PCIe SSD display of drives from SSSI members BitMicro, Fusion-io, IDT, Marvell, Micron, STEC, and Virident, and a live demonstration by Fusion-io.  The latest information gathered by the WIOCP Project will be presented.  Featured SSSI member speakers at Storage Visions include Jim Handy of Objective Analysis, who will examine how new storage developments are driving new storage systems with panelists Jim Pappas of Intel, Paul Wassenberg of Marvell, Mike Fitzpatrick of Toshiba, Paul Luse of Intel, and Sumit Puri of LSI; and Jim Pappas of Intel, who will moderate a panel on new frontiers in storage software with SSSI member panelists Walt Hubis of Fusion-io, Doug Voigt of HP, and Bob Beauchamp of EMC.

Follow our activities on Twitter at

twitter.com/#!/sniasolidstate

Ethernet Storage Forum – 2012 Year in Review and What to Expect in 2013

Jason Blosil

As we come to a close of the year 2012, I want to share some of our successes and briefly highlight some new changes for 2013. Calendar year 2012 has been eventful and the SNIA-ESF has been busy. Here are some of our accomplishments:

  • 10GbE – With virtualization and network convergence, as well as the general availability of LOM and 10GBASE-T cabling, we saw this is a “breakout year” for 10GbE. In July, we published a comprehensive white paper titled “10GbE Comes of Age.” We then followed up with a Webcast “10GbE – Key Trends, Predictions and Drivers.” We ran this live once in the U.S. and once in the U.K. and combined, the Webcast has been viewed by over 400 people!
  • NFS – has also been a hot topic. In June we published a white paper “An Overview of NFSv4” highlighting the many improved features NFSv4 has over NFSv3. A Webcast to help users upgrade, “NFSv4 – Plan for a Smooth Migration,” has also been well received with over 150 viewers to date.  A 4-part Webcast series on NFS is now planned. We kicked the series off last month with “Reasons to Start Working with NFSv4 Now” and will continue on this topic during the early part of 2013. Our next NFS Webcast will be “Advances in NFS – NFSv4.1 and pNFS.” You can register for that here.
  • Flash – The availability of solid state devices based on NAND flash is changing the performance efficiencies of storage. Our September Webcast “Flash – Plan for the Disruption” discusses how Flash is driving the need for 10GbE and has already been viewed by more than 150 people.

We have also added to expand membership and welcome new membership from Tonian and LSI to the ESF. We expect with this new charter to see an increase in membership participation as we drive incremental value and establish ourselves as a leadership voice for Ethernet Storage.

As we move into 2013, we expect two hot trends to continue – the broader use of file protocols in datacenter applications, and the continued push toward datacenter consolidation with the use of Ethernet as a storage network. In order to better address these two trends, we have modified our charter for 2013. Our NFS SIG will be renamed the File Protocol SIG and will focus on promoting not only NFS, but also SMB / CIFS solutions and protocols. The iSCSI SIG will be renamed to the Storage over Ethernet SIG and will focus on promoting data center convergence topics with Ethernet networks, including the use of block and file protocols, such as NFS, SMB, FCoE, and iSCSI, over the same wire. This modified charter will allow us to have a richer conversation around storage trends relevant to your IT environment.

So, here is to a successful 2012, and excitement for the coming year.

Why NFSv4.1 and pNFS are Better than NFSv3 Could Ever Be

AlexMcDonald

NFSv4 has been a standard file sharing protocol since 2003, but has not been widely adopted; party because NFSv3 was “just good enough”. Yet, NFSv4 improves on NFSv3 in many important ways; and NFSv4.1 is a further improvement on that. In this post, I explain the how NFSv4.1 is better suited to a wide range of datacenter and HPC use than its predecessor NFSv3 and NFSv4, as well as providing resources for migrating from NFSv3 to NFSv4.1. And, most importantly, I make the argument that users should, at the very least, be evaluating and deploying NFSv4.1 for use in new projects; and ideally, should be using it wholesale in their existing environments.

The background to NFSv4.1
NFSv2 (specified in RFC-1813, but never an Internet standard) and its popular successor NFSv3 was first released in 1995 by Sun. NFSv3 has proved a popular and robust protocol over the 15 years it has been in use, and with wide adoption it soon eclipsed some of the early competitive UNIX-based filesystem protocols such as DFS and AFS. NFSv3 was extensively adopted by storage vendors and OS implementers beyond Sun’s Solaris; it was available on an extensive list of systems, including IBM’s AIX, HP’s HP-UX, Linux and FreeBSD. Even non-UNIX systems adopted NFSv3; Mac OS, OpenVMS, Microsoft Windows, Novell NetWare, and IBM’s AS/400 systems. In recognition of the advantages of interoperability and standardization, Sun relinquished control of future NFS standards work, and work leading to NFSv4 was by agreement between Sun and the Internet Society (ISOC), and is undertaken under the auspices of the Internet Engineering Task Force (IETF).

In April 2003, the Network File System (NFS) version 4 Protocol was ratified as an Internet standard, described in RFC-3530, which superseded NFSv3. This was the first open filesystem and networking protocol from the IETF. NFSv4 introduces the concept of state to ameliorate some of the less desirable features of NFSv3, and other enhancements to improved usability, management and performance.

But shortly following its release, an Internet draft written by Garth Gibson and Peter Corbett outlined several problems with NFSv4; specifically, that of limited bandwidth and scalability, since NFSv4 like NFSv3 requires that access is to a single server. NFSv4.1 (as described in RFC-5661, ratified in January 2010) was developed to overcome these limitations, and new features such as parallel NFS (pNFS) were standardized to address these issues.

Now NFSv4.2 is now moving towards ratification. In a change to the original IETF NFSv4 development work, where each revision took a significant amount of time to develop and ratify, the workgroup charter was modified to ensure that there would be no large standards documents that took years to develop, such as RFC-5661, and that additions to the standard would be an on-going yearly process. With these changes in the processes leading to standardization, features that will be ratified in NFSv4.2 (expected in early 2013) are available from many vendors and suppliers now.

Adoption of NFSv4.1
Every so often, I and others in the industry run Birds-of-a-Feather (BoFs) on the availability of NFSv4.1 clients and servers, and on the adoption of NFSv4.1 and pNFS. At our latest BoF at LISA ’12 in San Diego in December 2012, many of the attendees agreed; it’s time to move to NFSv4.1.

While there have been many advances and improvements to NFS, many users have elected to continue with NFSv3. NFSv4.1 is a mature and stable protocol with many advantages in its own right over its predecessors NFSv3 and NFSv2, yet adoption remains slow. Adequate for some purposes, NFSv3 is a familiar and well understood protocol; but with the demands being placed on storage by exponentially increasing data and compute growth, NFSv3 has become increasingly difficult to deploy and manage.

In essence, NFSv3 suffers from problems associated with statelessness. While some protocols such as HTTP and other RESTful APIs see benefit from not associating state with transactions – it considerably simplifies application development if no transaction from client to server depends on another transaction – in the NFS case, statelessness has led, amongst other downsides, to performance and lock management issues.

NFSv4.1 and parallel NFS (pNFS) address well-known NFSv3 “workarounds” that are used to obtain high bandwidth access; users that employ (usually very complicated) NFSv3 automounter maps and modify them to manage load balancing should find pNFS provides comparable performance that is significantly easier to manage.

So what’s the problem with NFSv3?
Extending the use of NFS across the WAN is difficult with NFSv3. Firewalls typically filter traffic based on well-known port numbers, but if the NFSv3 client is inside a firewalled network, and the server is outside the network, the firewall needs to know what ports the portmapper, mountd and nfsd servers are listening on. As a result of this promiscuous use of ports, the multiplicity of “moving parts” and a justifiable wariness on the part of network administrators to punch random holes through firewalls, NFSv3 is not practical to use in a WAN environment. By contrast, NFSv4 integrates many of these functions, and mandates that all traffic (now exclusively TCP) uses the single well-known port 2049.


Plus, NFSv3 is very chatty for WAN usage; and there may be many messages sent between the client and the server to undertake simple activities, such as finding, opening, reading and closing a file. NFSv4 can compound these operations into a single RPC (Remote Procedure Call) and reduce considerably the back-and-forth traffic across the network. The end result is reduced latency.

One of the most annoying NFSv3 “features” has been its handling of locks. Although NFSv3 is stateless, the essential addition of lock management (NLM) to prevent file corruption by competing clients means NFSv3 application recovery is slowed considerably. Very often stale locks have to be manually released, and the lock management is handled external to the protocol. NFSv4’s built-in lock leasing, lock timeouts, and client-server negotiation on recovery simplifies management considerably.

In a change from NFSv3, these locking and delegation features make NFSv4 stateful, but the simplicity of the original design is retained through well-defined recovery semantics in the face of client and server failures and network partitions. These are just some of the benefits that make NFSv4.1 desirable as a modern datacenter protocol, and for use in HPC, database and highly virtualized applications.
NFSv3 is extremely difficult to parallelise, and often takes some vendor-specific “pixie dust” to accomplish. In contrast, pNFS with NFSv4.1brings parallelization directly into the protocol; it allows many streams of data to multiple servers simultaneously, and it supports files as per usual, along with block and object support through an extensible layout mechanism. The management is definitely easier, as NFSv3 automounter maps and hand-created load-balancing schemes are eliminated and, by providing a standardized interface, pNFS ensures fewer issues in supporting multi-vendor NFS server environments.

Next post; the Advantages of NFSv4.1

FOOTNOTE: Parts of this blog were originally published in Usenix ;login: February 2012 under the title The Background to NFSv4.1. Used with permission.

pNFS Advances

dougoflaherty

Building an industry standard is a series of incremental steps – from the original concept through ratification, followed by education and promotion, and ultimately to the development of an ecosystem of solutions. For a number of years the SNIA Ethernet Storage Forum (ESF) has been successfully advocating and promoting the NFSv4.1 standard and pNFS extensions.

Today, we welcome the open-pnfs.org community in its goal of extending the work of the SNIA ESF in promoting pNFS and NFSv4.1. Open-pNFS adds to the progression from standard to solution, by focusing and highlighting the commercial products coming to market and the maturation of the ecosystem.

Carry the Momentum into 2013! Renew Your SNIA Membership – or Join for the First Time!

khauser Leave a comment

As a new SNIA fiscal year begins,  SNIA Business Development and Membership Services would like to express our sincere thanks to all SNIA members for making 2012 a great success.  Through your efforts, the SNIA achieved tremendous milestones, including:

Members of the SNIA share a common goal: advancing IT technologies, standards, and education programs for all IT professionals. To this end, the SNIA is uniquely committed to connecting the IT industry with end-to-end storage and information management solutions and to delivering standards, education and services that will propel open storage networking solutions into the broader market.

Today, the SNIA has grown to over 400 member companies located all over the world. As we continue to expand into 2013, we invite you to continue your participation in our programs, and reap the benefits that SNIA brings to you and your organization.

Membership Categories & ROI

Membership in the SNIA offers a tremendous return on investment for individuals, companies, and organizations involved in the storage industry.

As a Vendor, your membership in the SNIA signals to customers that you are an industry leader driving specifications, architecture and standards.  Regardless of your company’s size, the SNIA has created an environment and pricing structure that encourages cross-company collaboration to develop real solutions that serve the ever expanding and ever changing needs of organizations.

As a Service Provider, the SNIA’s vendor-neutral environment provides the ideal backdrop for gaining insight into the storage, networking, and technology markets, essential for developing on-target sales and marketing strategies.  Access to equipment at the SNIA Technology Center in Colorado Springs allows you and fellow SNIA members in all membership categories to conduct in-depth evaluations of potential offerings, as well as analyses of future technologies.

Perhaps you are an Channel company.  The members-only area informs you quickly about market strategies and products in a vendor neutral atmosphere.  This is crucial when making planning and purchasing decisions for a diverse and technical customer base..

If you are a SNIA individual member, you stay current, connected and marketable.  Individuals in the SNIA have a clear, competitive edge.  Working along side of industry leaders, individual members have the opportunity to sharpen their professional skills, gaining both technical and/or marketing knowledge that otherwise could take years of on-the-job training.

The 2013 year holds bright promise, so don’t forget to renew your membership today.  Contact Lisa Mercurio, Membership Services Manager at (781) 293-9860 or email lisa.mercurio@snia.org.  And if you are a not yet a SNIA member, what are you waiting for?  Contact Marty Foltyn, SNIA Business Development Manager, at (858) 720-9780 or marty.foltyn@snia.org today!

Solid State on Stage

Marty Foltyn

The media buzz continues on solid state storage, now a prominent feature of storage, virtualization, and developer discussions.  Reaching the ears of business is SSSI’s Marketing Committee Chair Tom Coughlin, who has a blog on Forbes.  Latest posts discuss  a new age of digital storage interfaces, many based on the PCIe computer interface, that can take advantage of the higher data rates that NAND flash-based digital storage can provide.

Solid State Storage was featured at recent storage conferences including SNIA’s Storage Developer Conference and SNW (Storage Networking World) US and Europe.  At SNW US, Lucas Mearian of Computerworld and Jim Pappas, SNIA Board and SSSI member, discussed a sea change in the non-volatile memory (NVM) market over the next five years, with more dense and reliable technologies challenging dominant NAND flash memory now used in solid-state drives (SSD) and embedded in mobile products.  Mearian’s article is here.

At Powering the Cloud – SNW Europe, Randy Kerns of the SNIA member Evaluator Group led a spotlight session on Solid State Storage and Its Impact on the Environment, with presentations from SSSI members Kim Gardner of STEC and David Dale of NetApp.

SNIA and the Solid State Storage Initiative urge you to get involved in fast-developing SSSI activities in PCIe and NVM.  Visit our website at http://www.snia.org/forums/sssi or email asksssi@snia.org

Flash Webcast Q&A

David Fair

Our recent Webcast: Flash – Plan for the Disruption was very well received and well attended. We thank everyone who was able to make the live event. For those of you who couldn’t make it, it’s now available on demand. Check it out here.

There wasn’t enough time to respond to all of the questions during the Webcast, so we have consolidated answers to all of them in this blog post from the presentation team. Feel free to comment and provide your input.

Q. Are you going to cache both read and writes in NetApp FlashCache?
A. Flash Cache is a level 2 Read cache and it is used to accelerate random read operations. NetApp offers an additional capability called Flash Pool which caches both random reads and random overwrites. Both technologies are part of the NetApp Virtual Storage Tier family within the Data ONTAP operating environment.

Q. Is eMLC flash available today?
A. Yes, a number of Flash vendors are shipping eMLC today.

Q. Also can you review the write cycle performance of SLC vs. MLC?
A. Write cycles for SLC are typically around 100,000. With eMLC, write cycles of 30,000 per bit can be achieved.

Q. Has specific analysis been conducted on what applications and relative data can be cached at the server versus at the storage controller (tolerance for latency, user patience, etc.)?
A. This varies but server caching will typically be used for applications with the most hot spots such as databases. If there is a particular requirement for ultra low latency such as in OLTP environments, server caching may be appropriate. Server caching can also yield significant benefit to increase VM density. Generally, server caching will be deployed to accelerate a specific application while storage controller caching will be used to accelerate storage which is shared across multiple applications.
Q. Does the data running over the network storage PDUs or Ethernet Layer2/IP traffic?
A. Ethernet Layer 2 in this demo, thought it could have been scaled to for L3 IP routed traffic.
Q. What is the difference between flash tier and flash cache?
A. A flash tier is persistent storage whereby datasets are pinned to flash technology for some period of time (or permanently). In Automated Storage Tiering, data may be migrated to and from the flash tier based on the temperature of the data. A flash cache, on the other hand is a caching technology in which the most frequently accessed data is copied to flash for data access but then evicted as the data cools down. Data is copied to the flash cache either on the basis of calculated data temperature or on a first-in first-out basis.
Q. Given the large advantages of flash on power (direct), cooling, and DC footprint, why do enterprise data centers not just completely switch out their HDDs? It seems like there is a good ROI even without considering performance. Is it the operational complexities that make this challenging?
A. For many applications, this is not cost justified given the significant price difference of the SSD and HDD devices. Since hot data typically amounts to less than 20% of total data, a small amount of flash can be deployed successfully. In the caching case, this can be around 1%.

NFSv4.1 Webcast Q&A

AlexMcDonald

Our recent Webcast: NFSv4.1 – Plan for a Smooth Migration was very well received and well attended. We thank everyone who was able to make the live event. For those of you who couldn’t make it, it’s now available on demand. Check it out here.

There wasn’t enough time to respond to all of the questions during the Webcast, so we have consolidated answers to all of them in this blog post from the presentation team. Feel free to comment and provide your input.

Q. Will NFS 4.2 be any easier to migrate to than 4.1? Would it be worth waiting for?

A. NFSv4.2 is a set of additional functionality that will be easy to take advantage of – if you’re on NFSv4.1. The first move is to NFSv4.1, as it offers a wealth of features over and above NFSv3. Waiting for NFSv4.2 features wouldn’t be advisable; it’s unlikely to be ratified until the end of 2012, and enterprise server solutions and the required downstream client distributions will be a lot further out than that.

Q. Since NFS 4.1 is out, what is the uptake in the industry?

A. There aren’t any global figures, since not all suppliers collect detailed information about protocol usage, and of those that do, many can’t differentiate between NFS versions. Anecdotally, it’s slow. That’s because NFSv4.1 servers (particularly for file-layout) have only been available for less than a year, and the needed Linux client support has only recently made it through to the enterprise distributions.. NFSv4 (as opposed to 4.1) is more widely used; but the only figures I have are anecdotal, and would be misleading.
Q. Are there any network architecture design considerations that need to be taken before implementing NFSv4.1?

A. No. In fact, (if you’re not using pNFS) NFSv4.1 should get you more “bang for your buck” as there’s a reduction in network traffic compared with NFSv3. pNFS requires a different architecture; your storage vendor should be able to assist in the planning.

Q. Clustered servers – you mentioned that vendors had to provide a special server for this… are these enhancements going to be ported into the general linux nfs server stream?

A. I’m not sure to what this refers; perhaps the MDS (metadata server)? Although this server is often shown as a separate box in diagrams for simplicity, that’s not how it is normally implemented. The MDS is normally part of the cluster running on one or more fo the data servers.

Q. If you recommend AD for kerberos, do all of the NFS clients need to be joined to the same AD domain as well? Or only the servers?

A. Any time a client in one domain (or realm) attempts to access a server, the server must be in the same realm as the client, or if it’s in another realm, there must be cross realm trust so that the principal (the client) can be correctly authenticated.

Q. Can you talk about any difficulties in using Active Directory with NFS? Are there changes needed on AD?

A. No changes are needed to AD. It’s relatively straightforward security administration, and storage vendors should be able to provide you with implementation checklists.

Q. What is the impact on clustering and failover by introducing statefulness?

A. Significant! And much better. Recovery is much improved, as the server and client after a failure can attempt to agree on what locks were held, what files were open, what data had been written and so on. It’s a big improvement on NFSv3.

Q. Will it be possible to mount root file systems from NFSV4? Like boot from the SAN that we already have in FC or iSCSI?

A. Yes, that doesn’t change.

Q. Can you explain the reasons why home dir and hpc would benefit with v4.1?

A. Home directories are an easy win; no application (well, at least that you care about) and easily migrated. The same is often true of HPC. For example where the data is transient – served from a store to local disk, computed and crunched, and then sent back to the store – the store could be migrated to NFSv4 and the app later; or the app first and the store later.

SNIA Highlights NVM Programming TWG at Intel Developers Forum

Marty Foltyn

SNIA is proud to participate at the Intel Developers Forum this week at Moscone West In San Francisco.  Our booth in the NVM Express Community area focuses on SNIA and SSSI work in non volatile memory.

Attendees are very interested in the newly launched SNIA NVM Programming Technical Work Group (NVMp TWG).

The purpose of the NVMp TWG is to create specifications which will provide guidance to operating system, device driver, and application developers to be able to accelerate the availability of hardware-enabling Non-Volatile Memory (NVM) software.  A concrete example is database software which can use NVM to assure the database is available quickly after a system reboot.

The TWG will develop two specifications. The first is for NVM Extensions between OS Components. The work on a first version is starting now with an expected completion of about Q4’12.  The second specification is for NVM Application Extensions; with the first version expected Q2’13.

The Solid State Storage Initiative (SSSI) will  provide marketing and outreach support to the NVMp TWG, extending its educational mission.

The NVMp TWG is supported by EMC, Fusion-io, HP, Intel (also chairs the TWG), LSI, NetApp, Oracle, Samsung, SanDisk, Seagate, Symantec, Toshiba, Virident, and VMware.  More companies are in the process of joining SNIA and the SSSI.

If you are interested in the NVMp TWG, and/or want more information on SSSI activities  visit www.snia.org/forums/sssi