Category: LTFS

SNIA LTFS Format – New Version with Improved Capacity Efficiency

Diane Marsili Leave a comment

The SNIA Linear Tape File System (LTFS) Technical Work Group (TWG) is excited to announce that the new version of LTFS Format Specification has just been approved.  LTFS provides an industry standard format for recording data on modern magnetic tape. LTFS is a file system that allows those stored files to be accessed in a similar fashion to those on disk or removable flash drives.

The SNIA standard, also known as an ISO standard ISO/IEC 20919:2016, defines the LTFS Format requirements for interchanged media that claims LTFS compliance. Those requirements are specified as the size and sequence of data blocks and file marks on the media, the content and form of special data constructs (the LTFS Label and LTFS Index), and the content of the partition labels and use of MAM parameters.

The data content (not the physical media) of the LTFS format shall be interchangeable among all data storage systems claiming conformance to this format. Physical media interchange is dependent on compatibility of physical media and the media access devices in use.

SNIA on Storage sat down with Takeshi Ishimoto, Co-Chair of the SNIA LTFS Technical Work Group, to learn what it all means.

Q. What is this standard all about? Read More

SNIA Storage Developer Conference-The Knowledge Continues

khauser Leave a comment

SNIA’s 18th Storage Developer Conference is officially a success, with 124 general and breakout sessions;  Cloud Interoperability, Kinetiplugfest 5c Storage, and SMB3 plugfests; ten Birds-of-a-Feather Sessions, and amazing networking among 450+ attendees.  Sessions on NVMe over Fabrics won the title of most attended, but Persistent Memory, Object Storage, and Performance were right behind.  Many thanks to SDC 2016 Sponsors, who engaged attendees in exciting technology discussions.

For those not familiar with SDC, this technical industry event is designed for a variety of storage technologists at various levels from developers to architects to product managers and more.  And, true to SNIA’s commitment to educating the industry on current and future disruptive technologies, SDC content is now available to all – whether you attended or not – for download and viewing.

20160919_120059You’ll want to stream keynotes from Citigroup, Toshiba, DSSD, Los Alamos National Labs, Broadcom, Microsemi, and Intel – they’re available now on demand on SNIA’s YouTube channel, SNIAVideo.

All SDC presentations are now available for download; and over the next few months, you can continue to download SDC podcasts which combine audio and slides. The first podcast from SDC 2016 – on hyperscaler (as well as all 2015 SDC Podcasts) are available here, and more will be available in the coming weeks.

SNIA thanks all its members and colleagues who contributed to make SDC a success! A special thanks goes out to the SNIA Technical Council, a select group of acknowledged industry experts who work to guide SNIA technical efforts. In addition to driving the agenda and content for SDC, the Technical Council oversees and manages SNIA Technical Work Groups, reviews architectures submitted by Work Groups, and is the SNIA’s technical liaison to standards organizations. Learn more about these visionary leaders at http://www.snia.org/about/organization/tech_council.

And finally, don’t forget to mark your calendars now for SDC 2017 – September 11-14, 2017, again at the Hyatt Regency Santa Clara. Watch for the Call for Presentations to open in February 2017.

Linear Tape File System Now an International Standard

khauser Leave a comment

By David Pease, Co-Chair SNIA Linear Tape File System Technical Working Group

In 2011 the Linear Tape File System (LTFS) earned IBM an Engineering Emmy Award after being recognized by FOX Networks for “improving the ability of media companies to capture, manage and exploit content in digital form, fundamentally changing the way that audio and video content is managed and stored.”  Now, the International Standardization Organization (ISO) has named LTFS an International Standard (ISO/IEC 20919:2016).

LTFS’s road to standardization was a long one.  It started with IBM and the LTO (Linear Tape Open) Consortium jointly publishing the LTFS Format Specification as an open format in April, 2010, the day that LTFS was announced at the NAB (National Association of Broadcasters) show in Las Vegas.  In 2012, at the invitation of the Storage Networking Industry Association (SNIA), we formed the SNIA LTFS Technical Work Group, with a specific goal of moving towards international standardization.  The LTFS TWG and SNIA proceeded to publish several revisions of the LTFS Format Specification, inviting all interested parties to join the work group and contribute, or to comment on the specification before formal publication.  In 2014 SNIA helped the LTFS TWG format the then-current version of the specification (V2.2) to ISO standards and worked with the ISO organization to publish the specification as a draft standard and solicit comments.  After review and comments, the LTFS Format Specification was approved by ISO as an international standard in April of 2016 (just 6 years after it was first announced).

We are thrilled by the recognition of LTFS as an ISO standard; it is one more step towards guaranteeing that the LTFS format is a truly open standard that will continue to be available and usable for the foreseeable future.  In my opinion, two of the major inhibitors to the widespread use of tape technology for data storage have been the lack of a standard format for data storage and interchange on tape, and its perceived difficulty of use. LTFS addresses both of these problems by providing a general-purpose, open format that can easily be used like any other storage medium.

As the world’s data continues to grow at an increasing pace, and the need for affordable, large-scale storage becomes more important, the standardization of LTFS will make the use of tape for long-term, affordable storage easier and more attractive.

Use Case: Making Digital Media Storage Open and Future-Proof

Just as in personal photography, the last couple of decades have seen a major shift from analog and film technologies to digital ones in the Media and Entertainment industry, where modern cameras record directly to digital media. This has led to the need for new technologies to replace traditional film as a long-term storage medium for television and movies.

Film has some specific advantages for the Media & Entertainment industry that a new technology needs to replicate, including long shelf life, inexpensive, and zero-power storage, and a format that is “future-proof.” Tape storage is a perfect match to several of these criteria, including long (30+ years) shelf life, and zero-power, inexpensive storage. However, a stumbling block for the wide-spread acceptance of tape for digital storage in the media and entertainment business had been the lack of an open, easy-to-use, future-proof standard for the format of the data on tape. You can imagine an entertainment company using proprietary storage software, for example, only to run into problems like the provider going out of business or increasing its software costs to an unacceptable level.

We created LTFS to be an open and future-proof format from the beginning: open, because when we published the format, we made it publicly available at no charge, and future-proof because the format is self-documenting and can be easily accessed without the need for proprietary software.

Being an international standard should make anyone who is considering the use of LTFS even more comfortable with the fact that it is an open standard that is not owned or controlled by any single company, and is a format that will continue to be supported in the future.  As such, becoming an international standard has the potential to increase the use, and therefore the value, of LTFS across industries.

For more information about the work of the SNIA LTFS TWG, please visit www.snia.org/ltfs.

A Deep Dive into the SNIA Storage Developer Conference – The File Systems Track

khauser Leave a comment

SNIA Storage Developer Conference (SDC) 2015 is two weeks away, and the SNIA Technical Council is finalizing a strong, comprehensive agenda of speakers and sessions. Wherever your interests lie, you’re sure to find experts and topics that will expand your knowledge and fuel your professional development!SDC15_WebHeader3_999x188

For the next two weeks, SNIA on Storage will highlight exciting interest areas in the 2015 agenda. If you have not registered, you need to! Visit www.storagedeveloper.org to see the four day overview and sign up.

Year after year, the File Systems Track at SDC provides in depth information on the latest technologies, and 2015 is no exception. The track kicks off with Vinod Eswaraprasad, Software Architect, Wipro, on Creating Plugin Modules for OpenStack Manila Services. He’ll discuss his work on integrating a multi-protocol NAS storage device to the OpenStack Manila service, looking at the architecture principle behind the scalability and modularity of Manila services, and the analysis of interface extensions required to integrate a typical NAS head.

Ankit Agrawal and Sachin Goswami of TCS will discuss How to Enable a Reliable and Economic Cloud Storage Solution by Integrating SSD with LTFS. They will share views on how to integrate SSD as a cache with a LTFS tape system to transparently deliver the best benefits for Object Base storage, and talk about the potential challenges in their approach and best practices that can be adopted to overcome these challenges.

Jakob Homan, Distributed Systems Engineer, Microsoft, will present Apache HDFS: Latest Developments and Trends. He’ll discuss the new features of HDFS, which has rapidly developed to meet the needs of enterprise and cloud customers, and take a look at HDFS at implementations and how they address previous shortcomings of HDFS.

James Cain, Principal Software Architect, Quantel Limited, will discuss a Pausable File System, using his own implementation of an SMB3 server (running in user mode on Windows) to demonstrate the effects of marking messages as asynchronously handled and then delaying responses in order to build up a complete understanding of the semantics offered by a pausable file system.

Ulrich Fuchs, Service Manager, CERN, will talk about Storage Solutions for Tomorrow’s Physics Projects, suggesting possible architectures for tomorrow’s storage implementations in this field, and showing results of first performance tests done on various solutions (Lustre, NFS, Block Object storage, GPFS ..) for typical application access patterns.

Neal Christiansen, Principal Development Lead, Microsoft, will present Support for New Storage Technologies by the Windows Operating System, describing the changes being made to the Windows OS, its file systems, and storage stack in response to new evolving storage technologies.

Richard Morris and Peter Cudhea of Oracle will discuss ZFS Async Replication Enhancements, exploring design decisions around enhancing the ZFS send and ZFS receive commands to transfer already compressed data more efficiently and to recover from failures without re-sending data that has already been received.

J.R. Tipton, Development Lead, Microsoft, will discuss ReFS v2: Cloning, Projecting, and Moving Data File Systems, presenting new abstractions that open up greater control for applications and virtualization, covering block projection and cloning as well as in-line data tiering.

Poornima Gurusiddaiah and Soumya Koduri of Red Hat will present Achieving Coherent and Aggressive Client Caching in Gluster, a Distributed System, discussing how to implement file system notifications and leases in a distributed system and how these can be leveraged to implement a client side coherent and aggressive caching.

Sriram Rao, Partner Scientist Manager, Microsoft, will present Petabyte-scale Distributed File Systems in Open Source Land: KFS Evolution, providing an overview of OSS systems (such as HDFS and KFS) in this space, and describing how these systems have evolved to take advantage of increasing network bandwidth in data center settings to improve application performance as well as storage efficiency.

Richard Levy, CEO and President, Peer Fusion, will discuss a High Resiliency Parallel NAS Cluster, including resiliency design considerations for large clusters, the efficient use of multicast for scalability, why large clusters must administer themselves, fault injection when failures are the nominal conditions, and the next step of 64K peers.

Join your peers as well – register now at www.storagedeveloper.org. And stay tuned for tomorrow’s blog on Cloud topics at SDC!

LTFS Bulk Transfer Standard Q&A

David Slik

Our recent live SNIA Cloud Webcast “LTFS Bulk Transfer Standard” is now available on-demand. Thanks to all the folks who attended the live event. We did not have time to address all of the questions, so here are answers to them. If you think of additional questions, please feel free to comment on this blog.

Q. The LTFS standard seems to support shared extents between files, and by extension, deduplicated files. Is this a correct assessment, and how does it play in the bulk transfer standard?

A. The LTFS Bulk Transfer Standard supports shared extents as supported by the LTFS standard, which can transparently reduce space used by having multiple references to common data stored on tape (deduplication). This typically happens below the bulk transfer layer, by the software used to read and write the LTFS volumes. At this point, few software packages support this feature due to the wear and latency consequences of read seeks resulting from using this feature.

Q. What is the state of the standard in its lifecycle? (e.g., working group draft, public review, published, etc.)

A. The LTFS standard has been around for some time; more information can be found here at http://www.snia.org/tech_activities/standards/curr_standards/ltfs. The LTFS Bulk Transfer Standard is here at http://www.snia.org/tech_activities/publicreview#ltfsbulk, and is in public review.

Q. The standard seems to be based on the idea of moving physical tapes to the cloud. Is there a definition of a virtual LTFS image that can be moved between systems over the network?

A. Not yet, but that is a great idea we’ll be taking forward in the next versions of the proposal.

Q. One of the barriers to greater use of LTFS in the Cloud is the relative lack of enterprise grade management software that ensures that the tape media is refreshed / upgraded as it ages, that its integrity is periodically checked, that reclamation and compaction is done. It needs open standards for support in standard volume management systems as well. Until these things are in place, LTFS will be interesting largely to specialized industries like film/entertainment, seismic, and bulk transfer & bulk storage — but not about the steady-state use of tape as a true additional layer of the cloud storage hierarchy. Tape with LTFS plus proper management could fill this role — but not until the full lifecycle tape management is available and integrated.

A. The management that is always required for a physical product with a well-defined and finite lifetime is not a unique requirement of LTFS. Tape has a long history of use as a backup and archive medium, and there are a number of tape management products that are commercially available from LTO tape suppliers and independent software companies, as well as open source products. A Google search for “tape management software” will provide you with a number of alternative solutions.

Q. Do you have a list or people that sell LTFS based solutions?

A. No we don’t, but it’s a very good idea, and we’ll investigate it further.

 

New SNIA-CSI Webcast: LTFS Bulk Transfer Standard

Alex McDonald

Mark your calendar for February 10th as we conclude our Cloud Developer’s series by hosting a live Webcast on the LTFS Bulk Transfer Standard. LTFS (Linear Tape File System) technology provides compelling economics for bulk transportation of data between enterprise cloud storage.

This Webcast will provide an update on the joint work of the LTFS and Cloud Technical Working Groups on a bulk transfer standard that uses LTFS to allow for the reliable movement of bulk data in and out of the cloud, and mechanisms for verification, error handling and the management of namespaces. Register now to hear David Slik, Co-Chair of the SNIA Cloud Storage Technical Work Group, discuss:

  • LTFS standard mandate and history
  • LTFS adoption and use cases
  • LTFS bulk transfer to, from, and between clouds
  • Error handling and recovery
  • Security considerations

I’ll be hosting the event, taking your questions, and hopefully shedding some light on the importance of this standard. I hope you’ll join us.