Friday, February 5, 2010

12 Terabytes of Fast File Storage for less than $2000

In my previous article, A Strategy for Data Backup, I committed to outlining what I think is a solid, cost-effective, and reliable solution for wedding and portrait photographers.

Now as a little up-front disclaimer I would like to say that the equipment and software recommendations I list in this article are my own and I am not compensated in any way, shape, or form by any of the manufacturers or sellers of these products and services. I also offer this advice without an guarantees that they will work perfectly for anyone else. This particular setup is basically an updated version of the one we have been using the past three years.

About the system
This system is designed to offer the best compromise between access speed, raw storage space, backup automation, flexibility for rotating backups, and accommodating future growth for file storage needs. It's a bit of a DIY solution, but in my opinion it's preferable to a Drobo, which is essentially a proprietary RAID system.

System components
The basic components of the system are:
• A port multiplier-capable box, used for housing the hard drives
• Eight hard drives
• An eSATA card, for accessing the box at faster-than-USB/Firewire speeds
• Anti-static hard drive cases, for the offsite drives
• Chronosync, automated and configurable backup software

Recommended Components
Burly Port-Multiplier box ($500)
Mac Pro eSATA card ($200) OR MacBook Pro eSATA card ($200)
1.5 TB Hard Drives ($115)
Weibetech protective cases (10 for $55)
Extra Burly drive tray ($30)
Chronosync (Backup software for $40)

Putting it all together
1 x Burly Box @ $500
8 x 1.5 TB drives @ $125 = $920
1 x eSATA card @ $200
10 pack of protective cases = $55
2 x Burly drive tray @ $30 = $60
1 x Chronosync software @ $40
Total = ~$1755

Using the system
Since duplication is key, it's important to note here that this system is not designed to be the only backup. Ideally this system would be one of three total copies that exist of the work a photographer is attempting to back up -- and have it readily available for taking off site as well as restoration (if necessary).

Four of the drives will be in use regularly, the other four drives will be for offsite/emergency purposes, so the idea is to split data across the drives in a way that makes sense.

Storage system drives
• Drive A: 2010 RAW Files
• Drive B: Work In-Progress
• Drive C: Archived Work
• Drive D: More Archived Work + Business Files

Offsite/clone drives
• Drive E: 2010 RAW Files Clone
• Drive F: Work In-Progress Clone
• Drive G: Archived Work Clone
• Drive H: More Archived Work + Business Files Clone

These silos will allow a photographer to separate the work-in-progress files from RAWs and business files. Separating the work also allows for pairing up these backup drives with the other four drives, giving each drive a same-sized clone.

Chronosync, or a similar smart automated backup program, can run on a set schedule to manage the process of backing up to specific drives. Setting the backup program to do nightly dumps of a local work folder to a Work In-Progress drive is a great way to ensure that a drive failure will only mean one day of lost productivity after restoring.

Sample workflow
Here is a sample workflow for using this system to back up work and incorporate an off-site strategy as well:

1. Shoot a job
2. Download the images to the internal drive of the main editing workstation
3. Copy the RAW images to the 2010 RAW Files drive
4. Burn a DVD of the RAW images (optional but recommended)
5. Process the RAWs in Lightroom and export the corrected Jpegs to a working folder on the main editing workstation
6. Make additional adjustments to the work-in-progress images at the main editing workstation
7. Nightly backup process (Chronosync) store the work folder in the Work In-Progress drive automatically (set up folders & schedules for this up beforehand)
8. Insert the clone drives one by one and sync the contents from the storage system drive to the offsite/clone drive (on a regular basis; weekly, monthly, or after each job)
9. Take the clone drives (in the Weibetech boxes) to another location

What about the cloud?
As of February 2010, the cost of storing data legitimately* with a cloud-based service is still a little high for the average photographer to handle, especially as data grows. I have linked a spreadsheet that estimates the cost of cloud storage for a year.

*After some research I found that, unfortunately, webhosts offering "unlimited" storage only mean unlimited for files that are actually there in support of the website. For example, Dreamhost's terms of service say: "The customer agrees to make use of DreamHost servers primarily for the purpose of hosting a website. Data uploaded must be primarily for this purpose. DreamHost Apps servers are not intended as a data backup or archiving service." Go Daddy's hosting agreement says more plainly: "Go Daddy's shared hosting servers are not an archive"

Backup System and Cloud Storage Calculator

Please feel free to comment and leave your thoughts on this system and alternatives. I'd love to hear/see them!

Damon is the technical (and bag-carrying) assistant to Agnes Lopez, a commercial and wedding photographer who works primarily in Ponte Vedra Beach and Amelia Island, Florida. When he isn't standing still as a lighting test dummy, setting up a c-stand, or holding a reflector, Damon works as an IT Business Analyst, where he gets to solve technical problems on a daily basis.

9 comments:

  1. NOTE: If you tried to post a comment to this thread, please repost!! All comments were accidentally deleted by some stupid girl named Anne.

    ReplyDelete
  2. If I only buy 1 Burly box, how are the other 4 drives cloning?

    ReplyDelete
  3. Thanks Damon! Great info! I'm going to try and implement this workflow but on a 1/2 scale as I don't have a high volume of work or enough $$ to implement this rig. Hopefully next season!

    ReplyDelete
  4. @anon: It's a manual process. Basically you'll have to pull the drives out one by one and clone them, thus the extra trays. If you're already rotating drives offsite then it could just be part of the workflow.

    Chronosync will do it, as will programs like SuperDuper! and Carbon Copy Cloner (freeware).

    ReplyDelete
  5. @John: I'm glad it makes sense. I got a little worried that it was getting a bit too complex to be "simple".

    ReplyDelete
  6. Just as an FYI for readers of this post, I'm investigating CrashPlan. Since we don't currently use this service I don't feel comfortable recommending it to anyone, but I'll post updates when I find out more.

    ReplyDelete
  7. You're awesome. Just in case you didn't know.

    ReplyDelete
  8. Wow, thanks for this little run down. I've been handling backup the best way I can/know how, but this is exctly the info I've been looking for... a detailed list of exactly what I need. Thanks a lot.

    ReplyDelete
  9. I just came across the post & have to say you've got some great info there.

    I don't think too many people think of the absolute headache of trying to re-build broken RAID arrays or recover data across them.

    I'm presently using Araxis Merge for data backup/comparison (has the option of either comparing time stamps, or file content (bit-bit-bit)...

    Thanks so much for the post

    ReplyDelete