Showing posts with label data backup. Show all posts
Showing posts with label data backup. Show all posts

Friday, February 5, 2010

12 Terabytes of Fast File Storage for less than $2000

In my previous article, A Strategy for Data Backup, I committed to outlining what I think is a solid, cost-effective, and reliable solution for wedding and portrait photographers.

Now as a little up-front disclaimer I would like to say that the equipment and software recommendations I list in this article are my own and I am not compensated in any way, shape, or form by any of the manufacturers or sellers of these products and services. I also offer this advice without an guarantees that they will work perfectly for anyone else. This particular setup is basically an updated version of the one we have been using the past three years.

About the system
This system is designed to offer the best compromise between access speed, raw storage space, backup automation, flexibility for rotating backups, and accommodating future growth for file storage needs. It's a bit of a DIY solution, but in my opinion it's preferable to a Drobo, which is essentially a proprietary RAID system.

System components
The basic components of the system are:
• A port multiplier-capable box, used for housing the hard drives
• Eight hard drives
• An eSATA card, for accessing the box at faster-than-USB/Firewire speeds
• Anti-static hard drive cases, for the offsite drives
• Chronosync, automated and configurable backup software

Recommended Components
Burly Port-Multiplier box ($500)
Mac Pro eSATA card ($200) OR MacBook Pro eSATA card ($200)
1.5 TB Hard Drives ($115)
Weibetech protective cases (10 for $55)
Extra Burly drive tray ($30)
Chronosync (Backup software for $40)

Putting it all together
1 x Burly Box @ $500
8 x 1.5 TB drives @ $125 = $920
1 x eSATA card @ $200
10 pack of protective cases = $55
2 x Burly drive tray @ $30 = $60
1 x Chronosync software @ $40
Total = ~$1755

Using the system
Since duplication is key, it's important to note here that this system is not designed to be the only backup. Ideally this system would be one of three total copies that exist of the work a photographer is attempting to back up -- and have it readily available for taking off site as well as restoration (if necessary).

Four of the drives will be in use regularly, the other four drives will be for offsite/emergency purposes, so the idea is to split data across the drives in a way that makes sense.

Storage system drives
• Drive A: 2010 RAW Files
• Drive B: Work In-Progress
• Drive C: Archived Work
• Drive D: More Archived Work + Business Files

Offsite/clone drives
• Drive E: 2010 RAW Files Clone
• Drive F: Work In-Progress Clone
• Drive G: Archived Work Clone
• Drive H: More Archived Work + Business Files Clone

These silos will allow a photographer to separate the work-in-progress files from RAWs and business files. Separating the work also allows for pairing up these backup drives with the other four drives, giving each drive a same-sized clone.

Chronosync, or a similar smart automated backup program, can run on a set schedule to manage the process of backing up to specific drives. Setting the backup program to do nightly dumps of a local work folder to a Work In-Progress drive is a great way to ensure that a drive failure will only mean one day of lost productivity after restoring.

Sample workflow
Here is a sample workflow for using this system to back up work and incorporate an off-site strategy as well:

1. Shoot a job
2. Download the images to the internal drive of the main editing workstation
3. Copy the RAW images to the 2010 RAW Files drive
4. Burn a DVD of the RAW images (optional but recommended)
5. Process the RAWs in Lightroom and export the corrected Jpegs to a working folder on the main editing workstation
6. Make additional adjustments to the work-in-progress images at the main editing workstation
7. Nightly backup process (Chronosync) store the work folder in the Work In-Progress drive automatically (set up folders & schedules for this up beforehand)
8. Insert the clone drives one by one and sync the contents from the storage system drive to the offsite/clone drive (on a regular basis; weekly, monthly, or after each job)
9. Take the clone drives (in the Weibetech boxes) to another location

What about the cloud?
As of February 2010, the cost of storing data legitimately* with a cloud-based service is still a little high for the average photographer to handle, especially as data grows. I have linked a spreadsheet that estimates the cost of cloud storage for a year.

*After some research I found that, unfortunately, webhosts offering "unlimited" storage only mean unlimited for files that are actually there in support of the website. For example, Dreamhost's terms of service say: "The customer agrees to make use of DreamHost servers primarily for the purpose of hosting a website. Data uploaded must be primarily for this purpose. DreamHost Apps servers are not intended as a data backup or archiving service." Go Daddy's hosting agreement says more plainly: "Go Daddy's shared hosting servers are not an archive"

Backup System and Cloud Storage Calculator

Please feel free to comment and leave your thoughts on this system and alternatives. I'd love to hear/see them!

Damon is the technical (and bag-carrying) assistant to Agnes Lopez, a commercial and wedding photographer who works primarily in Ponte Vedra Beach and Amelia Island, Florida. When he isn't standing still as a lighting test dummy, setting up a c-stand, or holding a reflector, Damon works as an IT Business Analyst, where he gets to solve technical problems on a daily basis.

Thursday, January 14, 2010

A Strategy for Data Backup

Backing up your data is a must. Since image files are the lifeblood of any photographer's business it only makes sense to invest in a backup solution that protects the business' most important assets.

Q: What kind of solution is the best?

A: It depends!

For a commercial photographer who maintains a library of images that can be sold for stock, licensed, etc., a backup solution has to be about as bulletproof as possible. On the other hand, a wedding or portrait photographer is unlikely to do any work on a finished session from two years ago and would only need to keep the finished jpegs and album design files from a client job. If they want to be a little more thorough they could keep the RAW files and Lightroom catalogs.

Keep in mind that the cost of any solution increases exponentially with the amount of complexity involved. So if you want a mirrored system that includes offsite backup, RAID servers (never just one), and the ability to restore in a few hours -- you're talking serious dough.

Over this post and a follow up I'll outline what I think is a solid, cost-effective, and reliable solution for wedding and portrait photographers.

First I'd like to offer the tenets of my backup strategy:

  • Figure out what you need
  • Redundancy is not a bad word
  • Storage is cheap.
  • Duplication is key.
  • Complexity is the enemy.
  • Keep your data moving.
  • Take it off site.
  • It's cheaper to buy a lot of drives than to recover one
  • Test your backups

Figure out what you need

Answering some basic questions about your business will shape your backup strategy and help you prioritize your efforts:

What kind of photographer are you?

How likely are you to use the images again for licensing or sale?

What is the data retention policy for your business?

What is the retention policy in your contract?

What is your shooting format (RAW, Jpeg, DNG)?

What is your output format (Jpeg, PSD, Tiff)?

Redundancy is not a bad word

The desire of any good backup system is to be redundant. You want to have your data in a pristine state in multiple locations, preferably on site and some off site.

Storage is cheap

Hard drives are ridiculously large, very reliable, and incredibly cheap. There is no excuse that you can make for not having backups. Hard drives and physical media (DVD, Blu-ray, Flash drives) are getting cheaper, smaller, and more dense all the time. As of January 2010 it's possible to get a 2 terabyte (2 x 1000 gigabytes) drive for less than $200. Blu-ray discs are about $30 for a 50 GB per disc, which is easily enough space to back up a single job or RAW files. This low cost leads to my next tenet...

Duplication is key

I don't think I know anyone who hasn't had a hard drive fail or accidentally deleted an important file, so having a *relatively current* backup of important things is a must. These aren't the old days, where you tucked the film negatives into protective sleeves and put them in a file cabinet. In the digital age there is no quality loss from making copies of your files, so buy multiple hard drives and make copies of those important files! It's a great way to guard against catastrophic data loss. Adding in Blu-ray discs or traditional DVDs is a great way to diversify the duplication process for even more redundancy.

Complexity is the enemy

You may or may not have heard the phrase "RAID is NOT a backup" (It's worth a Google search) That argument about RAID notwithstanding, the important point is that most photographers are not IT people with years of experience working in data centers, troubleshooting hardware issues, and restoring mission-critical data from failures. In general,dealing with proprietary RAID controller cards, striping, configurations, and the like is almost guaranteed to end badly. Software RAID is also a bad idea from a performance and reliability standpoint.

If you're a wedding or portrait photographer it's unlikely that you're dealing with large databases or massive single files. Image files just need to be catalogued decently (so you can find the images you're looking for if you have to restore) and stored in multiple places (redundancy). Keeping it simple may mean having more drives but it also means you'll have a much better chance of having the files you need when that day comes.

Keep your data moving

When new technology comes out and hits a reasonable price, like the newer hard drives or Blu-ray discs, it's a good idea to move your old backup data into the new format. That way you've moved your data to a newer location, effectively increasing redundancy and the life expectancy of that data.

Take it off site

Off site backup just means not in one physical location. It doesn't need to be at some data center or in "the cloud" necessarily. If you can afford to do the Enterprise-class off site backup and your needs are heavy duty, then by all means go for it. At the same time the combination of large RAW file sizes and the bandwidth limitations of even broadband internet limit the effectiveness of going off site over the Internet.

If you have a studio separate from your home then it can be as simple as just having copies of your data at your home. If you run a home-based studio then simply storing a stack of hard drives in a safe location, say your in-laws or a friend's house, can be an effective solution. There's also the option of safe deposit boxes at your bank as well.

It's cheaper to buy a lot of drives than to recover one

The question is: Pay now or pay later?

If you don't have a backup strategy it's likely you'll pay quite a bit at one time when things do go wrong. The high cost of data recovery services is a reality you should be able to avoid (most of the time) with a good backup strategy. I've seen the cost of restoring one drive of client files be over $2,000 -- and that was for a drive that the company couldn't fully restore.

And then there's the awful possibility of litigation for losing important client files if they haven't yet been delivered.

For $2,000 a photographer could easily purchase a bunch of large hard drives, protective cases, a Blu-ray burner, a stack of Blu-ray discs, and DVDs. Considering the value of your data it's foolish to trust a single hard drive or single disc when there is no downside to having additional copies of your work saved in multiple locations.

Test your backups

A backup is only good if it can actually be used. Even though it's a boring exercise it's absolutely crucial to test your backups to make sure they actually work.

It's worth it to simulate a disaster scenario every few months (or at least once a year) to ensure all that backing up is actually effective.


Damon is the technical (and bag-carrying) assistant to Agnes Lopez, a commercial and wedding photographer who works primarily in Ponte Vedra Beach and Amelia Island, Florida. When he isn't standing still as a lighting test dummy, setting up a c-stand, or holding a reflector, Damon works as an IT Business Analyst, where he gets to solve technical problems on a daily basis.