You should always have an “I got hit by a bus” backup plan


November 30, 2015 by Kenneth Fisher

You have an on call, you have a business continuity plan, you have a disaster recovery plan but what happens when your developer, DBA or whoever is hit by a bus?

The situation

User: Oh noes! The web sites down!

One quick call later

Developer: Hmm, it looks like the database connection is down.

One more quick call later

DBA’s spouse: I’m sorry DBA isn’t available right now. He/she was hit by a bus on the way home and is in the hospital right now. Don’t worry, it was nothing major, just an amputated left arm. They re-attached it. DBA expects to be in the office tomorrow.

No backup plan

Developer: That’s not good. No one else knows how to fix the database stuff. What do we do?

DBA’s spouse: Wait until tomorrow?

With a backup plan

Developer: No problem. I’ll call our alternate.

Yet another quick call later

Backup DBA: No problem. It’s an easy fix. I’ll have you up in 5 minutes.

In a moderate to large company this is easy enough. Typically you will have several people who do the same job and one is on-call (or whatever) and another is the backup.

Things can be a little bit more difficult with a small company but it’s still possible. Cross training can be helpful but isn’t always going to be a full solution. When I commented on this on twitter someone jokingly said Just have Brent Ozar (b/t) on speed dial. Nice idea but that has the same problem. What if he was hit by the same bus? Much better to have a multi person company on speed dial. Brent Ozar Unlimited (b/t), The Midnight DBAs and Lobsterpot solutions are a few that I know off the top of my head and there are certainly a lot more. I doubt it would be difficult to find one willing to be on speed dial in case of a bus. They might want a few hours or even a day or two to become familiar with your systems and confirm connectivity but that’s better than losing hours or days of business because your DBA (or whoever) is out of commission right?

Look, you back up your data right? You have a plan in place your hardware goes south right? You need a plan in case one of your people is unavailable.

8 thoughts on “You should always have an “I got hit by a bus” backup plan

  1. notarian says:

    I don’t think a fire is a good time to establish a relationship with a consultant. Would Brent or Sean or Rob really drop everything to help you immediately? I am not going to shill for any particular company, but there are companies out there who act as your remote DBA and it’s probably better to get them setup and familiar with your systems before your primary DBA gets hit by a bus.

    • Well, my point was to set up the relationship ahead of time. That way you know what to expect. They might not be able to get to you within your normal expected response time but they are likely to be quicker than your DBA who is having a personal emergency. The idea here is to plan ahead. Be aware that DBAs (and developers) are human and in an emergency may not be available when you need them.

      • notarian says:

        Too many places I’ve worked don’t even think they need a DBA, let alone a backup DBA (consultant or not). You should never have one person with all the knowledge. Even if they don’t get hit by a bus, they might resign and there are always things that get missed during a 2-week transition period.

        • Yep. I’ve heard lots of stories about companies like that. I’ve also heard about what happens to them. Frequently on StackExchange as they try to resolve the problems caused by not having that knowledge in house.

  2. Jim says:

    I’ve always preferred “won the lottery” to “hit by a bus”. It makes planning discussions a little less morbid since the person being hit by the bus is sitting there in the room!

  3. H G says:

    sometime is it easy to use example like this “Hit by bus” or “won the lottery” for management to realize that Cross training and knowledge transfer is very importance within team not only to make team strong but build confidence among the team..

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 3,755 other subscribers

Follow me on Twitter

ToadWorld Pro of the Month November 2013
%d bloggers like this: