Replication & Tiering

The Right Data at the Right Place at the Right Time


Digital Media Storage Management – Some Common Challenges

The challenges faced by content creators have shifted from simple sharing to workflow management, and coping with the explosion of data experienced as productions shift to high-resolution pipelines for 2K, UHD, 4K, and even 8K.

Producers find themselves asking these questions:

  • How do I free up disk space for new projects?
  • How can I minimise my need for expensive, high-performance storage?
  • How do I meet my production and legal requirements for backups and archives?
  • Do I want to archive a project to tape when I know there are changes expected?
  • Where can I keep project files while waiting for client approvals?
  • How can I easily access footage from a backup?
  • How can I ensure users and applications can always find their files?
  • How do I reduce error prone and time-consuming manual human intervention in media management activities?
  • What role can lower performance, lower cost storage play in my digital media workflow?

Digital production using central storage greatly increases efficiency, collaboration, and productivity and is a fundamental requirement for most professional production work. But to achieve a complete end to end workflow system for digital media, content creators need more than just file sharing for all users – they also need user and project management, as well as effective storage management that lets them control how their data is stored, used, and backed up. The most cost effective solution would use a combination of high-speed online storage for real-time production, nearline NAS for staging and parking, and offline tape or object storage for long-term backup. Using these different storage tiers together in an integrated Performance Defined Workflow system is referred to as Hierarchical Storage Management.

A Performance Defined Workflow implemented with Tiger Replication & Tiering addresses these questions and lets you extract the most value from multiple storage tiers of different price/performance levels. Replication & Tiering is specialized virtualisation technology that allows users and applications to have direct access to media, regardless of where it is physically stored, giving you an elegant, user- friendly, efficient, and cost-effective tiered storage management solution.

As an integrated Tiger module, Replication & Tiering offers a much simpler workflow than third party media management products, and benefits users by being natively developed and supported by the same manufacturer as their shared storage system, which increases functionality and eases technical support and troubleshooting.

Dynamics of a Performance Defined workflow

An efficient and cost-effective storage pipeline must consider these factors:

  • Production demands
    • Rapid production turnaround
    • Media loading and unloading
    • Access to backups
  • Backup and archive requirements
    • Backup completed projects
    • Simple operation
    • Easy restore
    • Ecosystem integration and compatibility with 3rd party solutions
  • Storage unit Performance
    • Certain data must be assigned to storage with appropriate performance characteristics, i.e. real-time playback of higher resolution media can only be achieved with high-performance storage
  • Project management
    • Certain projects must be moved from one storage system to another, i.e. workflow requirements demand that data reside in specific locations for application and user access

Replication & Tiering addresses all these factors automatically in one simple policy-driven system.

What It Does

Tiger Replication & Tiering lets digital media producers easily control which storage unit will store which data, thereby creating a cost-effective storage solution with backup and archive capability.

Simply, Replication & Tiering does two things:

  1. Creates a second copy of your data (protection).
  2. Makes space available on source storage unit by substituting 2 kb stub files for the original material.

Users can easily set up 2nd and 3rd tier storage management schemes, and Replicate files from and to Tiger Series appliances from and to slower, value storage such as NAS,  object, or cloud. Once files are replicated they can be purged from the online storage and a stub file is left behind in the original file location. This stub acts as a pointer to the replicated file, and any user or application that opens the stub file will trigger an automatic recall of the replicated file to the Online Tier. This recall takes place in the background without having to search for the file in an index or mount any tape when the backup device is available.

Use Replication & Tiering for:

  • Staging and Parking Projects
    • Ingest and offload media to NAS for easy transfer to and from online storage.
    • Transfer media to and from Tiger online storage “just in time”.
    • Load material from nearline media pools.
  • High-Speed for High-Resolution
    • Uncompressed high resolution color grading and mastering require costly extreme speed storage units.
    • Use high-speed Tiger Box1 SSD for real-time playback.
    • Replicate to/from lower cost Tiger expansion boxes or NAS.
    • Work on one job while offloading the last one and pre-loading the next one.
  • Automatic Backup
    • Easily satisfy your studio’s backup requirements.
    • Set it and forget it operation.
    • Fast operation with 10 Gb Ethernet connections.
  • Easily Access Replicated Material
    • Automatic recall of files from other storage tiers with stub-file workflow.
    • Users can see and access the original file locations where they will find the stub files pointing to the Replicated files.
    • Users and applications that call for the file will trigger a restore of the Replicated file from the storage tier.
    • Achieve fast restore speeds with 10 Gb Ethernet connections.
  • Advanced Archive Capability with Integrated Partner Solutions
    • Further extend your MAM/PAM and Enterprise archive capabilities.
      • DDN WOS
      • Other Object Storage
    • Exploit Recall of a Recall with 2 Tier Solutions.
      • XenData
      • BlackPearl
      • StrongBox
  • Space Reclaiming
    • Once material is replicated, users may choose to automatically or manually free space on their Tiger managed storage in preparation for the next job.
    • Users can see and access the original file locations where they will find the stub files pointing to the Replicated files.
  • Ingest
    • Files may also be Replicated to the Tiger online storage from the 2nd tier so you can use your nearline storage as an ingest station. Replication & Tiering will scan the nearline storage and then generate Tiger stub files on the production storage which can then trigger the transfer of material to the online storage.
  • Restore
    • Just open a file and the recall process begins immediately with no other intervention required when the backup device is available.

How It Works

Tiger Technology Replication & Tiering is an automatic, policy-driven system (with manual controls) that manages where data is stored among multiple storage units with different performance characteristics. Once the policies are set, operation is automatic and no user-intervention is required.

Simple Policies

The flow of data from Online storage to the other tiers respects these variables:

  • File Age
    Regardless of storage capacity, material that has not been accessed in a user-defined time period is Replicated from the Online Tier to the Nearline Tier.
  • Capacity Threshold
    If a volume reaches a certain capacity threshold, previously replicated material is purged from the online storage to free space for new files.

The policies include the source and destination storage units.

Compatible Storage Units

Choose the backup device that suits your needs:

  • Directly Integrated Devices
    • DDN WOS
    • IBM LTFS
    • Spectra Logic BlackPearl
  • Backup and Archive Products
    • XenData
    • Cache-A
    • Storage DNA
    • StrongBox
    • etc.
  • DAS (Tiger Expansion Box with Tiger Box, Box1, and Box1 SSD Provides Excellent Restore Speeds)
  • NAS (or File Server)
  • Object Storage
    • DDN WOS – direct integration
    • Spectra Logic BlackPearl – direct integration
    • NEC HydraStor

Advanced Workflows

Tiger Replication & Tiering offers a number of advanced workflow options.

  • High-Speed for High-Resolution
    Use SSD Tiger storage in combination with nearline storage, such as a Tiger Expansion chassis, to create an ultra-high-performance workflow with SSD drives to allow real-time playback of 2K, 4K, and 8K material for online color grading, finishing, and mastering. This solution is ideal for high-res stereoscopic 3D work as well.
  • Archive
    Third party archive products can be pointed to the Nearline Tier, then write the archive to tape, object storage, or the cloud. Offloading the archive process to nearline reduces the usage of your online storage and preserves resources for production work. Easily integrate with XenData, Archiware, Storage DNA, and other backup and archive solutions.
  • Integrated 3 Tier Solutions
    Tiger has partnered with 3rd-party providers of advanced Hierarchical Storage Management (HSM) systems to provide end-to-end 3 tier workflows. Use an elegant online to nearline to tape backup and archive system with Spectra Logic BlackPearl or StrongBox. These solutions provide an easy way to backup and recall material from online to nearline to tape with a simple, double-click to recall mechanism.
    Use the power of 3rd party asset and production management systems to index and catalog Online and Nearline storage tiers and easily recall files necessary for production.

Storage Tiers

Storage is generally divided into three categories, or tiers, and each type can play a role in a performance and cost defined workflow.

  1. Online Storage
    This is the highest performance storage in your setup. It needs to sustain playback of multiple streams of high-resolution material to multiple clients, all in real-time. Usually, it is RAID protected, and for true multi-user collaborative workflow without bottlenecks, is configured as a SAN (storage area network), and client workstations usually connect to it directly. The material stored on it is readily accessible, and so it is said to be “online”. It is the most expensive type of storage used in digital media production.
  2. Nearline Storage
    This type of storage does not provide enough playback performance for real-time playback of high-resolution files. It too is usually RAID protected. In most cases, this type of storage is configured as a NAS (network attached storage), and is commonly referred to as a file server. It usually would be accessible through a 10 Gb or 1 Gb Ethernet connection. All data written to and read from this volume must pass through the dedicated machine, which creates a bottleneck. It is more affordable than online storage but its limitations mean it can’t really be used for production.
  3. Offline Storage
    This is usually LTO tape in either a stand-alone single cartridge drive or in a multi-cartridge library capable of storing petabytes of material. Usually, an archiving product is used to control these devices, and files stored offline are not accessible without first being restored, which requires accessing an index and locating and mounting the cartridge. This is time consuming and you will have to wait before being able to use the files in production. It offers the lowest cost container for data, and is suitable for long-term storage.

Object storage blurs the lines between nearline and offline storage. It is very dense, disk-based storage, so offers high capacity, and the potential for faster restore times. It is generally costlier than tape. Object storage also overcomes limitations of file system based storage, given they are self-healing, scalable, and distributed. Files held in object storage can’t be written or read by traditional applications on common operating systems without the use of special translator called an “abstraction layer”, so it is not a replacement for tape. However, its density, speed, and other advantages make it a viable alternative to nearline and tape storage in certain situations.

Scale-out NAS can be considered for use as production storage, however the underlying architecture of NAS solutions and the latency of the IP protocol running over Ethernet restrict NAS to low-res material or proxy editing and low user-counts. These limitations can only be diminished with complex and expensive configurations that nullify any cost savings from NAS.