You should always have an “I got hit by a bus” backup plan
8November 30, 2015 by Kenneth Fisher
You have an on call, you have a business continuity plan, you have a disaster recovery plan but what happens when your developer, DBA or whoever is hit by a bus?
The situation
User: Oh noes! The web sites down!
One quick call later
Developer: Hmm, it looks like the database connection is down.
One more quick call later
DBA’s spouse: I’m sorry DBA isn’t available right now. He/she was hit by a bus on the way home and is in the hospital right now. Don’t worry, it was nothing major, just an amputated left arm. They re-attached it. DBA expects to be in the office tomorrow.
No backup plan
Developer: That’s not good. No one else knows how to fix the database stuff. What do we do?
DBA’s spouse: Wait until tomorrow?
With a backup plan
Developer: No problem. I’ll call our alternate.
Yet another quick call later
Backup DBA: No problem. It’s an easy fix. I’ll have you up in 5 minutes.
In a moderate to large company this is easy enough. Typically you will have several people who do the same job and one is on-call (or whatever) and another is the backup.
Things can be a little bit more difficult with a small company but it’s still possible. Cross training can be helpful but isn’t always going to be a full solution. When I commented on this on twitter someone jokingly said Just have Brent Ozar (b/t) on speed dial. Nice idea but that has the same problem. What if he was hit by the same bus? Much better to have a multi person company on speed dial. Brent Ozar Unlimited (b/t), The Midnight DBAs and Lobsterpot solutions are a few that I know off the top of my head and there are certainly a lot more. I doubt it would be difficult to find one willing to be on speed dial in case of a bus. They might want a few hours or even a day or two to become familiar with your systems and confirm connectivity but that’s better than losing hours or days of business because your DBA (or whoever) is out of commission right?
Look, you back up your data right? You have a plan in place your hardware goes south right? You need a plan in case one of your people is unavailable.
SQL Studies 
I don’t think a fire is a good time to establish a relationship with a consultant. Would Brent or Sean or Rob really drop everything to help you immediately? I am not going to shill for any particular company, but there are companies out there who act as your remote DBA and it’s probably better to get them setup and familiar with your systems before your primary DBA gets hit by a bus.
Well, my point was to set up the relationship ahead of time. That way you know what to expect. They might not be able to get to you within your normal expected response time but they are likely to be quicker than your DBA who is having a personal emergency. The idea here is to plan ahead. Be aware that DBAs (and developers) are human and in an emergency may not be available when you need them.
Too many places I’ve worked don’t even think they need a DBA, let alone a backup DBA (consultant or not). You should never have one person with all the knowledge. Even if they don’t get hit by a bus, they might resign and there are always things that get missed during a 2-week transition period.
Yep. I’ve heard lots of stories about companies like that. I’ve also heard about what happens to them. Frequently on StackExchange as they try to resolve the problems caused by not having that knowledge in house.
I’ve always preferred “won the lottery” to “hit by a bus”. It makes planning discussions a little less morbid since the person being hit by the bus is sitting there in the room!
That is an excellent way to put it. I have a somewhat morbid sense of humor so I’ve always used the bus 🙂
sometime is it easy to use example like this “Hit by bus” or “won the lottery” for management to realize that Cross training and knowledge transfer is very importance within team not only to make team strong but build confidence among the team..
Yep. I agree. Having a strong understanding of what other members of the team is doing can be a huge asset all the time. Not just when you have an emergency.