8.5

Edit

Sample Disaster Recovery Plan

Abbreviations and acronyms

RPO

Recovery Point Objectives (acceptable data loss)

RTO

Recovery Time Objectives (acceptable downtime)
NAS Network Attached Storage
DR Disaster Recovery
DC Datacenter
DNS Domain Name System
FQDN Fully Qualified Domain Name


Summary

Disaster Recovery is sub-set of Business Continuity, which involves a set of policies, tools and procedures to enable the recovery or continuation of vital technology infrastructure and systems (including applications) following natural or human induced disaster.

Disaster Recovery Plan presents the aspects of the solution relevant to backup and recovery, identifies and describes weaknesses in the system, and describes backup methods and recovery steps.This could include steps for replacing hardware, rebuilding/modifying/replacing the applications, restoring data, or cold backup systems that stand in for a failed solution.

Recovery Point objective and Recovery Time objective

One consistent objective critical to the customer is to ensure reliable solution operations with a minimum of down time with a minimal possible data loss in a cost-effective manner.

Let's consider the following sample Disaster Recovery plan:

  • Redundancy in main data center with sync NAS mirror for continuous data replication
  • Warm Stand By environment in secondary data center with async NAS mirror for continuous data replication
  • Nightly (every 24 hours) backup

The table below defines RPO and RTO (minimum, average, maximum) to recover and resume operations from the moment of incident registration for each type of the solution failure:
Failure TypeRecovery PointRecovery TimeNotes
minavgmaxminavgmax
Single Server is down005m0.5h1h2hPrimary HA scenario: DNS switch to a local backup server, sync NAS mirror as a data recovery provider
Datacenter is down015m30m4h6h12hPrimary DR scenario: DNS switch to Warm site, async NAS mirror as a data recovery provider
Datacenter is down, NAS replica data integrity lost06h24h6h12h24hSecondary DR scenario: DNS swith to Warm site, async NAS mirror and DB backup restore as a data recovery providers

Details

Data replication between the main site and the Warm Site DCs occurs by applying async NAS Mirroring technology. Fail over from a main site to the Warm Site occurs by a global DNS switch approach as the most efficient software DR best practice.

Backup and rRecovery methods

The Backup and Recovery Methods section describes the methods planned to backup the hardware, operating system(s), applications, infrastructure, resources, and data that comprise the solution. The description is included for each of these solution component classes: the type of backup, location of backups, backup procedures, and backup responsibilities. For each backup method, the procedures for using the backup to restart the solution and recover the state of its operations and the solution data is described below.

There are three major data recovery types introduced by the DR architecture:

  1. Hypervisor based replication a technology that automatically creates and maintains replicas of virtual hard disks or entire virtual machines (depending on the platform that is being used)
  2. Asynchronous NAS mirroring between the main site Datacenter and Warm Site DC;
  3. Daily DB backup, automatically performed for the system and customer DBs;

Testing DR plan

In order to ensure Disaster Recovery requirements are fulfilled in regard of the acceptable downtime and data loss objectives (RPO and RTO), the Testing procedure for DR should be regularly performed according to the steps below:

  1. Set up PROD and DR regions.
  2. Stop all services in PROD region.
  3. Execute Disaster Recovery Plan.
  4. Validate that site is operational.

Disaster Recovery Plan prerequisites

  1. Data mirroring between (Asynch, Synch, Hypervisor) the main site and the Warm Site DCs should be configured by the client IT Operations team.
  2. DNS entries mapping with full FQDNs and appropriate IP addresses from both Main Site and Warm Site is created and maintained in up to date state.
  3. Alternative OCR license for a Warm Site infrastructure should be obtained in advance and stored on a suspended system.
  4. Alternative RPA configuration (Selenograph config file) with a list of RPA IPs should be prepared in advance (ideally during the Warm Site installation procedure) and stored on a suspended system.
  5. Alternative WorkFusion system web system and endpoints IP addresses mapped to the main system FQDNs, should be documented and stored in advance (during the Warm Site installation procedure).
  6. Warm Site infrastructure, including OS and Applications, has the same update versions and patches as the main site infrastructure has. The recommended way to meet this requirement is to include Warm Site updates into a main site Operation procedures.
  7. The Warm Site infrastructure is pre-provisioned, all the target apps are pre-installed and pre-configured according to the installation guide.
  8. All the apps in the Warm Site are stopped (in a "stand-by" mode).

Disaster Recovery Plan assumptions

  • Fail over from a main site to the Warm Site instance occurs by the defined configuration steps and a DNS switch to preserve a components FQDNs while an actual endpoint IP address might differ.
  • The main facility of the organization has been destroyed or became nonoperational.
  • Staff are available to perform critical functions defined within the disaster recovery plan.
  • Staff can be notified and can report to the backup site(s) to perform critical processing, recovery and reconstruction activities.
  • Off-site storage facilities and materials survive.
  • An alternate facility is available.
  • A backup site is available for processing the organization's work.
  • The necessary long-distance and local communication lines are available to the organization.
  • Vendors will perform according to their general commitments to support the organization in a disaster.
  • Warm Site should be in active state only the minimal necessary time to recover the Main Site from the disaster.
← PreviousNext →