6 min read

Media Storage and Organization for Creators

Media Storage and Organization for Creators
Photo by Jakob Owens / Unsplash

Organizing your footage, photos, and other digital media after a day out is always a chore. We made this video to help explain how we manage some of our content and some of the programs and methods we use to offload this footage from the capture cards (SD Cards, or CF express) to our NAS or external SSDs.

We have also have a YouTube video for this topic that might help!

Video

Storage Overview


Speeds

MAX Capacity

Durability

Flash (SD Cards)

≈300MB/s

<1TB

High

SSD

≈550MB/s - 4000+ MB/S

1TB- 8TB

High

HDD

≈300MB/s

22+TB

Low

So just a quick recap on storage there is basically three types that most content creators use regularly: Flash storage, Solid State Drives (SSD), and Hard Disk Drives (HDD). (There is technically more but if you are interested in those you should probably watch our previous video)

Photo by Markus Winkler / Unsplash

Flash

This is your SD Cards, and CF Express cards that are most commonly used “capture cards”. These are generally pretty reliable, and durable, but because they are compact there is some tradeoffs that have been made.

In the past I have had a few SD cards fail and they generally fail in one of two ways, corrupted files, or just stop functioning. This is mostly likely because of the failure of either the controller or the flash storage (NAND). Like almost all tech they have a life span; the modern cards in the last few years are much better than the cards from the early 2000s, but if it’s an important shoot use simultaneous recording where you write the same data to multiple cards.

An Angelbird AtomX 500GB SSD which we use with our Atomos Shogun Inferno and Sumo 19" Monitor/Recorders. The only odd thing about these is that even though they're listed as compatible with the Atomos products, they don't fit into the caddy's and therefore get buried in the Sumo card slot. But nothing a little gaff tape pull-tab can't fix, am I right?
Photo by Voice + Video / Unsplash

SSD

Solid State Drives are a more advanced form of flash, and have added features like wear leveling (some SD cards have this now), multiple NAND chips for parallel reads/writes, and use DRAM to cache operations. This makes SSDs significantly faster than SD cards and they also will last longer under sustained loads and heavy usage.

Endangered technic.
Photo by Patrick Lindenberg / Unsplash

HDD

Hard Disk Drives are spinning rust…. Well kinda… the are the traditional method of bulk storage where there are magnetic cells that spin around on metal disks coated in a magnetic alloy. Each of these cells can hold data, and because HDDs have been around for quite a while the technology is a proven method of storing and archiving files. HDDs are also available in much larger sizes and are much cheaper option for bulk storage.

There is a downside, HDDs are not durable and much bigger than their flash and SSD counter parts.  HDDs are quite slow due to the mechanical nature, they also are sensitive to mishandling and prone to failure if dropped. The HDD failure modes are also interesting they have a reputation for being finicky as sometimes just letting the drive cool off you are able to read the data.

Hashing

Now how do you detect if the data coming from your storage device is the same as when you put it there?

Well there are a few ways to do this but the industry best practice is known as hashing. Hashing is using a mathematical algorithm to create what is basically a “summery” of the file in a fixed length string. No matter how small or large the file is the summery will always be a fixed length. For example SHA-256 will always produce a 256bit string or “file summery”.

There are also different types of hashing algorithm but most of them fall in to “cryptographically secure” and “Non-cryptographically secure”. Cryptographically secure algorithms are what you should use when you need to be 100% certain the file has not been tampered with (also you should know this changes over time as flaws are found in older algorithms) for example banks, hospital, security and other critical applications. But this does not mean non-cryptographically secure algorithms do not have value. One such hash is the XXhash64 is significantly faster (as its more optimized and less computationally intensive) and provides “good enough” resistance to collisions for most file verification tasks. It’s also supported by many studios like Netflix for quick file validation and media delivery.

The Files
Photo by Maksym Kaharlytskyi / Unsplash

Media/Content Management Tools

There are a few tools that can help you offload your content and transfer it to your storage devices…. As a quick recommendation if you can back up to at least two separate storage devices (i.e NAS + external SSD or External SSD + Cloud) and keep the capture media if possible at least until you have validated that everything transferred as expected

While both of these programs will help you with transferring and managing files its important to understand that this does not replace the need for manual validation of your footage as it does not prevent losing the capture cards or just forgetting to transfer that footage (this is why I recommend keeping the capture media to make sure you transferred everything).

TerraCopy - https://www.codesector.com/teracopy

The first piece of software I started out with was TerraCopy as it was originally a windows utility, but does have a Mac utility now. Its great for transferring files and creating hash files for validating files, however there is some cost associated for commercial use. I was not able to get TerraCopy to create folders to reorganize my capture media -> storage in an automated fashion. However it was great for being able to pause and resume transfers, and only transfer updated or new files.


(To be fully transparent I did have some significant issues with the Mac version of the utility crashing so not something I would recommend there)

Hedge - https://hedge.video/hedge

This is a Multiplatform utility that is part of the Netflix Technology Alliance that has worked great for me as it has a majority of the features I need to automatically dump files and then organize them in my desired folder structure and create hashes of all the files so that I can validate them later. The current folder structure I use for this is

- YEAR (Manually Created)
|
- Project Name (Manually Created)
||
- Month-Day (i.e 07-28)
|||
- Camera Name (i.e A7SIII)
||||
-File Extension (i.e MP4)
||||
-File Extension (I.e XML)
||||
-File Extension (I.e ARW)
|||
- Camera Name (i.e A1)
||||
-File Extension (i.e MP4)
||||
-File Extension (I.e XML)
||||
-File Extension (I.e ARW)


There is plenty of folder structure alternatives such as Netflix’s minimal standard of “Show Title/Shoot Date/Media/Roll” and even creating your own that works for you and your workflow. Its more about consistency and keeping the same structure over time so that even when you go back to your archived footage its easier to find the shot you want.

The other great feature of Hedge is that you can copy data from a single source over to multiple destinations. For example sending one copy of your SD card to your external SSD for editing and one to your NAS for archiving. This has really made the keeping two copies easier for us.

(To be transparent with this as well; I paid for a license for hedge for both Kayla and I.  I have not been asked or paid to write this; this is just my opinion of the software and how it has worked for us.)

Summary

There is plenty of software out there that will help manage media and content that you create. This style of organization has really helped me when shooting with multiple cameras such as iPhones and action cameras to keep everything organized.

I have attached my config file for Hedge below if you want to use this style or even use it as a starting point. Just add it to your hedge preset folder and then select it as the default when transferring.