Talking about data storage seems kind of boring, but it is a necessity for photographers, videographers and any other digital creators. I was hoping to help those who are getting into this type of work or taking on a more serious commitment in 2022 and making sure you are prepared for your future storage requirements. None of this Blog or Video is sponsored
There are basically three options (and recently a fourth) for storage:
- Network Attached Storage (NAS)
Local storage is where a physical HDD (Hard Disk Drive) or SSD (solid State Drive) is connected directly to your motherboard/system. This provides the most direct connection resulting in the quickest access times and often the quickest read and write speeds. This is the way to go if you are looking for performance.
HDDs VS SSDs:
- HDDs are significantly cheaper for larger capacities (HDDs are Cheap and SSDs are Expensive)
- SSDs are significantly quicker for read/write
- SSDs have a set life where as HDDs are more open ended
- HDDs often fail slowly where SSDs can fail suddenly
- HDDs perform well with sequential writes/reads (large files)
- SSDs can handle random reads and writes of small files
- fastest access times
- fastest read/write speeds
- no need to mount always connected
- possible to upgrade and add drives on most desktops
- harder to expand later (often requires copying of data from one drive to another)
- limited space (as compared to RAID arrays or NAS)
- no redundancy
- if the drive fails you lose all the data
This is similar to local storage however, all of the data will flow through USB ports or thunderbolt. One thing to keep in mind is that you are also limited by the performance of these ports. This is not a problem for HDDs as they often don't reach the read or write speeds of USB 3.x ports, however this is an issue for SSD, especially high end SSDs. There are some benefits with external drives that are unmatched by any other category.
- fast access-fast read/write speeds with SSDs
- easy to move between computers
- limited to USB speeds (Not a problem with HDD and USB 3.0 and above)
- if drive fails or is lost all data is lost-no redundancy
Raid arrays are basically external drives with the added benefit of redundancy, with the downside increased cost and computing overhead. These have been mostly replaced with Network Attached Storage (NAS) but they are still made and still have their purpose, especially if you only edit on one computer. There is also some compatibility issues depending on if the RAID is controlled by your OS (Operating System) or by dedicated RAID Array hardware. This is not something that is always clear and can have significant impacts on ease of use and management of the drives.
- Has redundancy (with RAID 1,5,6,10, etc.)
- Faster than just one external drive
- Can be used with SSDs or HDDs
- Needs Multiple Drives (cost)
- Only connected to one computer
- Admin/Maintenance overhead
Network Attached Storage
NAS's basically have all of the same benefits of RAID Arrays (as it uses RAID arrays internally), with the added component of being connected to your network. This allows them to be accessed over the network by multiple computers at the same time. This comes with some positives as everything can be centrally managed, and everyone is working with the same files, however there is some technical skill required for the administration and maintenance. This downside has been drastically reduced over the last few years with commercial options becoming more home user and small business friendly.
The other down side is most home networks are 1 Gb/s which is significantly slower than USB 3.0 (5Gb/s). (Home WIFI routers often promise high throughput over 1Gb/s especially with WIFI 6 but most of this is marketing as most of the ethernet ports are still 1 Gb/s on the back. So you can expect your speeds to be around 400-800 Mb/s or 0.4 -0.8 Gb/s).This can be fixed by using 10Gb/s networking to the computers accessing the NAS, however this is expensive. There are some intermediate ethernet standards (2.5Gb/s & 5Gb/s) That are becoming more common, but some of these are just as expensive as 10Gb/s networking equipment. This is why we made the investment and moved to 10Gb/s networking at-least for core components and computers connected to the NAS.
- Centrally managed data (only need one RAID Array)
- Faster than just one Drive
- Other tools like automation (i.e automated creation of proxies)
- Limited to network speeds (can be slow at 1Gb/s for Video)
- Slightly more expensive than RAID arrays
Cloud Storage is one of the newer solutions for storage, however this comes with some trade-offs. One thing to remember about the cloud is "its just another persons computer with an internet connection". There are a few benefits, primarily it can expand with your use and need for storage. The other fringe benefits are more about the simplicity of managing data in the cloud and the ease of use as the software does most of the heavy lifting.
The downside is that you are now limited to your internet speeds and unless you have 1Gb/s symmetrical fiber this is going to be significantly slower than your local network (and if you did upgrade to 10Gb/s, 10x slower). Editing in the cloud is becoming more possible, but there are severe limitations and it is not the same as local computer. This is something that works for photographers but not really for videographers and other video based creators.
- Similar benefits to a NAS but accessible anywhere with internet
- Can expand later
- No hardware management
- Great for backups
- limited to internet speeds
- Expensive (pay per space used)
- requires sending data to others (could be a concern for corp work)
What We Do
Well we kinda have a hybrid approach with our storage. We use primarily External SSDs for editing videos as this allows us to move it between machines/computers and still have high performance editing (1050 MB/s). We also store all of our footage on our NAS with 6x10TB drives and a 10Gb/s network connection. This allows us to centrally manage all of this data and access it from multiple machines. Additionally we automatically back up to Backblaze all of our footage should anything happen to our NAS or external storage.
Local Storage: Sabrent 1TB PCIe 4.0 NVME
External SSD: 2TB Samsung T7
NAS: Home Built TrueNAS 5600x, 64GB ECC, 6x 10TB (WD Gold + Seagate Exos X10)
How do we Transfer Data
Quick tip here always transfer from the source i.e the camera, the SD card, the CF express, etc. whatever you use to initially capture your images or video to your drives. So for us we back up each SD card to the SSD and then to the NAS, This helps ensure that even if something happens in the transfer that corrupts the files, hopefully one of the copies on the NAS or External SSD is good.
SD Card #1 -> External SSDs -> editing -> export
SD Card #1 -> NAS -> Backblaze
SD Card #2 -> External SSDs -> editing -> export
SD Card #2 -> NAS -> Backblaze
SD Card #x -> External SSDs -> editing -> export
SD Card #x -> NAS -> Backblaze
Format All SD Cards in camera (if its really sensitive ill keep the SD cards for longer)
hopefully this helps you develop a flow for managing your data and give you some ideas for storing your data longer term.
- RAID calculator: http://www.raid-calculator.com/
- Synology RAID Calc: https://www.synology.com/en-us/support/RAID_calculator
- Synology NAS: https://www.synology.com
- QNAP NAS: https://www.qnap.com
- Favorite External SSD (Samsung T7): https://www.samsung.com/semiconductor/minisite/ssd/product/portable/t7/
- WD Gold: https://www.westerndigital.com/products/internal-drives/wd-gold-sata-hdd#WD1005FBYZ
- Seagate Exos: https://www.seagate.com/products/enterprise-drives/