November 30, 2015 by Kenneth Fisher
You have an on call, you have a business continuity plan, you have a disaster recovery plan but what happens when your developer, DBA or whoever is hit by a bus?
User: Oh noes! The web sites down!
One quick call later
Developer: Hmm, it looks like the database connection is down.
One more quick call later
DBA’s spouse: I’m sorry DBA isn’t available right now. He/she was hit by a bus on the way home and is in the hospital right now. Don’t worry, it was nothing major, just an amputated left arm. They re-attached it. DBA expects to be in the office tomorrow.
No backup plan
Developer: That’s not good. No one else knows how to fix the database stuff. What do we do?
DBA’s spouse: Wait until tomorrow?
With a backup plan
Developer: No problem. I’ll call our alternate.
Yet another quick call later
Backup DBA: No problem. It’s an easy fix. I’ll have you up in 5 minutes.
In a moderate to large company this is easy enough. Typically you will have several people who do the same job and one is on-call (or whatever) and another is the backup.
Things can be a little bit more difficult with a small company but it’s still possible. Cross training can be helpful but isn’t always going to be a full solution. When I commented on this on twitter someone jokingly said Just have Brent Ozar (b/t) on speed dial. Nice idea but that has the same problem. What if he was hit by the same bus? Much better to have a multi person company on speed dial. Brent Ozar Unlimited (b/t), The Midnight DBAs and Lobsterpot solutions are a few that I know off the top of my head and there are certainly a lot more. I doubt it would be difficult to find one willing to be on speed dial in case of a bus. They might want a few hours or even a day or two to become familiar with your systems and confirm connectivity but that’s better than losing hours or days of business because your DBA (or whoever) is out of commission right?
Look, you back up your data right? You have a plan in place your hardware goes south right? You need a plan in case one of your people is unavailable.