Comprehensive guide to preparing staff for a disaster recovery

Tom Kiblin, VP of strategic solutions at ServerCentral Turing Group, shares his top tips for getting  staff prepared for a disaster

We all know that testing a disaster recovery (DR) plan is important. It’s the equivalent of testing your smoke detector twice a year – we know we should, but many of us (like, 23 percent of us) don’t get around to actually doing it. Unlike the smoke detector, though, you won’t run an accidental test of your DR plan next time you cook.

Handling a disaster recovery takes time and coordinated effort from people across your organisation – and there’s a good chance most of them don’t understand what role they’re supposed to play.

The good news: with some training, you can prepare your coworkers for a disaster event and make sure your organisation is capable of weathering this storm. Not sure where to start? Try these eight tips.

Include staff from across the organisation

While it may be the IT team’s job to strategize and build backup environments, DR is not a job for IT alone. In a real disaster, your entire organisation will be affected, so in training, you’ll need to work with representatives from every department and understand their use of business-critical apps.

Start by talking with leaders of each department to figure out which business-critical apps their department uses, identify dependencies on non-critical apps, and determine who should participate in the test.

Once you’ve identified participants, take time to train them in how to communicate their findings, especially when they notice things that aren’t working properly. Be sure to emphasize that the goal of a test is not to verify that everything is just fine but to discover areas that aren’t working as expected so that the IT team can adjust the DR plan accordingly.

Emphasize the importance of internal buy-in

The most common reason organisations give for not testing a DR plan is that they don’t have the time. But DR is a team effort and it can’t succeed without practice. If you want people from around your organisation to be willing to dedicate the necessary time and resources to preparing for a disaster, you have to get buy-in from C-suite leaders. Otherwise, your colleagues will brush off your requests to participate in training.

What’s important to note here is that, even if you’ve already convinced the CFO that everyone needs to be disaster ready, they won’t be your CFO forever (average tenure for a CFO is just over five years).

Besides that, budgets change. Priorities change. Critical applications change. Help every member of your organisation understand the importance of ongoing disaster preparedness so that DR testing becomes part of the culture regardless of who’s holding the reins of power (and the strings of the pocketbook).

Facilitate regular updates to your DR plan

Too often, DR plans get created to check a box or satisfy an audit – and then they’re never touched again.

Make it easy to keep your DR plan relevant: send an automated weekly email to members of every team and ask about changes in the last week – new software, new policies, new practices. Then make it part of your weekly routine to determine whether these should lead to a change in your DR plan.

It’s also essential to make sure your DR leader is present at important strategy meetings for the business as a whole (not just for the IT team). As company-wide goals and policies change, your DR plan needs to adapt. It won’t do anyone any good to have a fully functioning DR plan that restores apps your sales team no longer uses but doesn’t include their brand-new CRM.

Schedule regular DR plan tests

I mentioned your DR leader in the section above. If you don’t already have someone who is primarily in charge of DR testing, assign that person now. Make it clear that part of what this person’s job performance is measured on is the execution of DR plans – including tests. This is the single most effective way to ensure that regular testing happens and that a company’s DR plan is as fully aligned with the business as possible.

Test in real life

It may be tempting to let your IT team run remote tests of certain functions. But that’s not good enough for serious DR tests because it doesn’t offer reliable, real-world data about how applications are performing. A few tips for testing “real life” conditions:

Let people who actually use applications test them. The application may technically “work,” but if it can’t function the way people use it every day, it’s not going to work in an actual disaster. Communicate to testers that the goal of the test is not to find manageable workarounds, but to use applications as they would under normal circumstances.

Give testers clear instructions on how to report problems they encounter. Train the IT folks reviewing these comments to follow up and dig deeper as necessary. The goal is to facilitate and reward clear communication – especially when it’s about something that’s not working. This may sound straightforward, but it goes against the way most of us operate day to day. One way to incentivize reporting problems is to create a modest internal “bug bounty” for anyone who finds them (or a lottery where every ticket gets you an entry).

Run the company from the DR environment for an extended period of time (up to a month). This is the rubber-to-the-road test of whether your DR environment actually has what it needs to have to support your business. It’s also very tempting to skip, especially when you’ve just run a successful short-term test. But unless you actually test operating in your DR environment, you can’t be sure you’ll be able to in the event of a real disaster.

Test failing back

Even businesses with the best DR plan sometimes forget to include this step in DR tests and training. They shouldn’t. It’s really important. Your team may know how to operate in a DR environment, but do they know how to roll back to normal functions?

After a successful test, fail back to production systems as you would in a real disaster scenario. As with other parts of your DR test, make sure you communicate to stakeholders what you’re doing and to proactively request feedback. Of course, always be on the lookout for problems to solve.

Plan for disaster communications

You can build an amazing DR plan and train your team to be ready to implement it, but if you don’t have a way to communicate with them that you’re in disaster mode, all that work might be lost – and for sure there will be confusion. How will employees know to log into the DR environment instead of prod? What about remote employees?

Include a plan for letting employees know that you’ve put your DR plan in place, as well as a plan for communicating this plan to your team.

Agree on a physical backup location

If a natural disaster is the reason your system is down, there’s a chance your physical location will also be affected. Given the current rate of severe weather events, every business should have a plan for what to do if their primary location is unusable (flooded, without power, etc.).

Part of training for a disaster recovery should involve running a “fire drill” where employees report to the alternate location (even if it’s their own homes). This will help you identify and iron out any location-based kinks in your DR plan, including remote logins to your backup system and unexpectedly long commute times. Something as simple as knowing what happens when everyone tries to login to the VPN at the same time is critical for a successful DR environment.

Don’t neglect the human side of disaster recovery

Preparing for the human side of disaster recovery is as important as planning for the technical side. Without a trained workforce, even the best-laid DR plans won’t do what you need them to. When you invest in training, though, your organisation as a whole is more likely to understand the realities of preparing for disasters and therefore more likely to handle your next one without serious losses.