Adventures in Data Storage, Part 1

Back posting again after a holiday weekend here in the USA as well as an adventure to recover damaged data. On Saturday morning, I wanted to check up on a hard crash that had occurred on the main “Frankenserver” that stores a backup of all the Outside In and SV2 Studios files.

Here’s a pic of secondary Frankenserverbeing upgraded last year – I call them that as they are built from the parts of other machines to recycle parts and keep the film’s costs as low as possible.

A Frankenserver being upgraded in 2010

One question I get a lot is how I store and backup data. I’ve made a simple diagram that gives an overview of the design and process. This diagram is simplified but all the key steps are there. I have two primary workstations – the primary is the fastest, most powerful and has a large, fast disk array for photographic storage and the secondary is used when the primary is tied up and also for freelance work as needed.

I have 3 other workstations – an audio/recording workstation and secondary render box plus two low-end boxes: a Linux machine and a Hackintosh. Then there are two Frankenservers, the primary that has 27 TB of storage and is connected to a cloud backup and the secondary provides additional utility storage (workstation backup etc.).

The primary Frankenserver is connected to cloud backup storage (CrashPlan). The overall design of this is to keep the costs of the film 1/10th of what it would normally but create a system that can recover, eventually, from even a catastrophic data loss – say an alien warship destroying my house (but not destroying earth, but then again, that will not be a big worry at that point).

This design also allows me to work on the film on multiple stations although the primary workstation is fastest for the heavy lifting. It consists of low-cost, DIY, home-built computers overclocked to provide near professional workstation speeds at 1/4 of the cost. And the storage and backup costs are a tiny fraction since it relies on the very cheapest hard drives and cloud storage that costs only $5 a month combined with a $20 month upgrade on internet service for fast upload speeds.

The bottom line is all critical files are backed up in at least two locations and 3 to 6 copies of each file. Storage costs on the film are currently 4 cents a gigabyte and offline backup costs are less than 1 cent a gigabyte. Currently I have about 6 Terabytes backed up offline but the number grows everyday.

It’s a good design especially given the cost – the only disadvantage is a slow restore process in the event of large data loss but this past weekend, it was tested.

PART TWO TOMORROW – what happens when the main server motherboard dies and corrupts the main data storage array.

1Comment

Post A Comment