I had a much needed, but not at all desired, chance to think through backup, disaster recovery and business continuity (Archival wasn't really a subject of much thought at the time).
I provide Tier 2+ technical support to an engineering firm of about 40 peole. The Boot Sector on a drive which hosts all of their Autocad files was corrupted today. They backup to their Charlotte office, and so a restore would have taken a full day. For reasons I'll go into in a technical article I'll post later today, we determined a backup of the data on the drive needed to be made before attempting a restore (disk utilities are cool, they can back-up drives you can't even see). The backup took most of the day to complete and so the majority of their engineers were unable to work.
As I think about this I am sickened. I architected their network with new servers, daily backups, multiple domain controllers, etc just so this sort of thing wouldn't happen. It did, and they lost a significant amount of money. How could I have let this happen?
Well, I put lots of thought into disaster recovery, and lots of thought into backups, but not enough thought into business continuity. How do they keep working in the event of a crisis of this sort? Well, for $500 I could have purchard a couple external drive, and utilized sync toy to do a nightly one-way sync to it of all their critical data. They could even rotate the drives so one is always off-site. In a situation like this we could have immediately redirected the engineers to that drive and they would have been up and running in under an hour rather than having to be off-line for a full day. It's not an elegant solution, but it's an example of one of many solutions that would have prevented this disaster (DFS/FRS is another with a slightly higher price tag).
The off-site backup to Charlotte is great for Backup and great for disaster recovery and even takes care of the archival needs as well. It even addresses one aspect of the business continuity concern. Should the building burn, should the office be leveled by a tornado, they could temporarily relocate their office staff to charlotte office and be back to work within a day or two. However, this was not an issue I thought through. What happens if a drive is corrupted, or a server crashes or ??? How quick can we recover from the minor crisis?
Many of you have already thought through your Backup, Disaster Recovery, Business Continuity and Archival. But have you? Have you dug deep? Have you considered the minor and the major disasters? Are their opporunities to spend a little bit of money to add additional layers of responsiveness and robustness to your plan? I'll close with a few random backup tips that may be of interest:
- Never make a domain controller your backup server.
- Never make your exchange server your backup server.
- If your backing up large amounts of data off-site, consider if there are means of quickly getting data back on-line when needed.
- As long as I've got the space and time for it, I'd take a Differential over an incremental any day.
- Brick level back-ups of Exchange are wonderfully helpful. If your Exchange backup require you to dismount the Exchange store to perform a restore, you may want to consider adding a brick level backup to your routine.
That's all I've got. I'd love to hear any additional thoughts or ideas you may have on this subject as it's an extremely complex question with no clean solutions.
One last note, if you're looking for a hosted solution, Fairway Consulting is considering a hosted solution for the church market (http://www.Fairwayconsulting.com) that is very exciting and ACS Technologies already has one (http://www.acstechnologies.com/).