Section III Business Continuity In This Section Chapter 11: Introduction to Business Continuity Chapter 12: Backup and Recovery Chapter 13: Local Replication Chapter 14: Remote Replication Chapter 11 Introduction to Business Continuity C ontinuous access to information is a must for the smooth functioning of Key Concepts Business Continuity business operations today, as the cost of business disruption could be catastrophic. Information Availability There are many threats to information avail- Disaster Recovery ability, such as natural disasters (e., flood, fire, earthquake), unplanned occurrences (e., Disaster Restart cybercrime, human error, network and com- BC Planning puter failure), and planned occurrences (e., upgrades, backup, restore) that result in the Business Impact Analysis inaccessibility of information. It is critical for businesses to define appropriate plans that can help them overcome these crises. Business continuity is an important process to define and implement these plans.
Business continuity (BC) is an integrated and enterprisewide process that includes all activities (internal and external to IT) that a business must perform to mitigate the impact of planned and unplanned downtime. BC entails pre- paring for, responding to, and recovering from a system outage that adversely affects business operations. It involves proactive measures, such as business impact analysis and risk assessments, data protection, and security, and reac- tive countermeasures, such as disaster recovery and restart, to be invoked in the event of a failure. The goal of a business continuity solution is to ensure the “information availability” required to conduct vital business operations.
229 230 Section III n Business Continuity This chapter describes the factors that affect information availability. It also explains how to create an effective BC plan and design fault-tolerant mecha- nisms to protect against single points of failure.1 Information Availability Information availability (IA) refers to the ability of the infrastructure to func- tion according to business expectations during its specified time of operation. Information availability ensures that people (employees, customers, suppliers, and partners) can access information whenever they need it. Information avail- ability can be defined with the help of reliability, accessibility and timeliness.
■■ Reliability: This reflects a component’s ability to function without failure, under stated conditions, for a specified amount of time. ■■ Accessibility: This is the state within which the required information is accessible at the right place, to the right user. The period of time during which the system is in an accessible state is termed system uptime; when it is not accessible it is termed system downtime. ■■ Timeliness: Defines the exact moment or the time window (a particular time of the day, week, month, and/or year as specified) during which information must be accessible.
For example, if online access to an applica- tion is required between 8:00 am and 10:00 pm each day, any disruptions to data availability outside of this time slot are not considered to affect timeliness.1 Causes of Information Unavailability Various planned and unplanned incidents result in data unavailability. Planned outages include installation/integration/maintenance of new hardware, soft- ware upgrades or patches, taking backups, application and data restores, facility operations (renovation and construction), and refresh/migration of the testing to the production environment. Unplanned outages include failure caused by database corruption, component failure, and human errors. Another type of incident that may cause data unavailability is natural or man-made disasters such as flood, fire, earthquake, and contamination.
As illustrated in Figure 11-1, the majority of outages are planned. Planned outages are expected and scheduled, but still cause data to be unavailable. Statistically, less than 1 percent is likely to be the result of an unforeseen disaster. Chapter 11 n Introduction to Business Continuity 231 Disaster (<1%) Unplanned Outage (20%) Planned Outage (80%) Figure 11-1: Disruptors of data availability 11.2 Measuring Information Availability Information availability relies on the availability of the hardware and software components of a data center.
Failure of these components might disrupt informa- tion availability. A failure is the termination of a component’s ability to perform a required function. The component’s ability can be restored by performing an external corrective action, such as a manual reboot, a repair, or replacement of the failed component(s). Repair involves restoring a component to a condition that enables it to perform a required function within a specified time by using procedures and resources.
Proactive risk analysis performed as part of the BC planning process considers the component failure rate and average repair time, which are measured by MTBF and MTTR: ■■ Mean Time Between Failure (MTBF): It is the average time available for a system or component to perform its normal operations between failures. ■■ Mean Time To Repair (MTTR): It is the average time required to repair a failed component. While calculating MTTR, it is assumed that the fault responsible for the failure is correctly identified and that the required spares and personnel are available. Note that a fault is a physical defect 232 Section III n Business Continuity at the component level, which may result in data unavailability.
MTTR includes the time required to do the following: detect the fault, mobilize the maintenance team, diagnose the fault, obtain the spare parts, repair, test, and resume normal operations. IA is the fraction of a time period that a system is in a condition to perform its intended function upon demand. It can be expressed in terms of system uptime and downtime and measured as the amount or percentage of system uptime: IA = system uptime / (system uptime + system downtime) In terms of MTBF and MTTR, IA could also be expressed as IA = MTBF / (MTBF + MTTR) Uptime per year is based on the exact timeliness requirements of the service, this calculation leads to the number of “9s” representation for availability met- rics. Table 11-1 lists the approximate amount of downtime allowed for a service to achieve certain levels of 9s availability.
For example, a service that is said to be “five 9s available” is available for 99.999 percent of the scheduled time in a year (24 × 7 × 365). Table 11-1: Availability Percentage and Allowable Downtime Downtime Downtime Uptime (%) Downtime (%) per Year per Week 98 2 7.3 days 3 hr 22 minutes 99 1 3.65 days 1 hr 41 minutes 99.2 17 hr 31 minutes 20 minutes 10 sec 99.1 8 hr 45 minutes 10 minutes 5 sec 99.3 Consequences of Downtime Data unavailability, or downtime, results in loss of productivity, loss of rev- enue, poor financial performance, and damages to reputation. Loss of produc- tivity reduces the output per unit of labor, equipment, and capital. Loss of revenue includes direct loss, compensatory payments, future revenue losses, billing losses, and investment losses.
Poor financial performance affects revenue Chapter 11 n Introduction to Business Continuity 233 recognition, cash flow, discounts, payment guarantees, credit rating, and stock price. Damages to reputation may result in a loss of confidence or credibility with customers, suppliers, financial markets, banks, and business partners. Other possible consequences of downtime include the cost of additional equip- ment rental, overtime, and extra shipping. The business impact of downtime is the sum of all losses sustained as a result of a given disruption.
An important metric, average cost of downtime per hour, provides a key estimate in determining the appropriate BC solutions. It is calculated as follows: Average cost of downtime per hour = average productivity loss per hour + average revenue loss per hour Where: Productivity loss per hour = (total salaries and benefits of all employees per week) / (average number of working hours per week) Average revenue loss per hour = (total revenue of an organization per week) / (average number of hours per week that an organization is open for business) The average downtime cost per hour may also include estimates of projected revenue loss due to other consequences such as damaged reputations and the additional cost of repairing the system.2 BC Terminology This section introduces and defines common terms related to BC operations and are used in the next few chapters to explain advanced concepts: ■■ Disaster recovery: This is the coordinated process of restoring systems, data, and the infrastructure required to support key ongoing business operations in the event of a disaster. It is the process of restoring a previ- ous copy of the data and applying logs or other necessary processes to that copy to bring it to a known point of consistency. Once all recoveries are completed, the data is validated to ensure that it is correct.
■■ Disaster restart: This is the process of restarting business operations with mirrored consistent copies of data and applications. ■■ Recovery-Point Objective (RPO): This is the point in time to which sys- tems and data must be recovered after an outage. It defines the amount of data loss that a business can endure. A large RPO signifies high tolerance to information loss in a business.
Based on the RPO, organizations plan for the minimum frequency with which a backup or replica must be made. For 234 Section III n Business Continuity example, if the RPO is six hours, backups or replicas must be made at least once in 6 hours. Figure 11-2 shows various RPOs and their corresponding ideal recovery strategies. An organization can plan for an appropriate BC technology solution on the basis of the RPO it sets.
For example: ■■ RPO of 24 hours: This ensures that backups are created on an offsite tape drive every midnight. The corresponding recovery strategy is to restore data from the set of last backup tapes. ■■ RPO of 1 hour: This ships database logs to the remote site every hour. The corresponding recovery strategy is to recover the database at the point of the last log shipment.
■■ RPO of zero: This mirrors mission-critical data synchronously to a remote site. Weeks Weeks Tape Backup Tape Restore Days Days P eriodic Replication Disk Restore Hours Hours Asynchronous Replication M anual M igration Minutes Minutes Synchronous Replication Global Cluster Seconds Seconds (a) Recovery-point objective (b) Recovery-time objective Figure 11-2: Strategies to meet RPO and RTO targets ■■ Recovery-Time Objective (RTO): The time within which systems, appli- cations, or functions must be recovered after an outage. It defines the amount of downtime that a business can endure and survive. Businesses can optimize disaster recovery plans after defining the RTO for a given data center or network.
For example, if the RTO is two hours, then use a disk backup because it enables a faster restore than a tape backup. However, for an RTO of one week, tape backup will likely meet require- ments. Some examples of RTOs and the recovery strategies to ensure data availability are listed below (refer to Figure 11-2): ■■ RTO of 72 hours: Restore from backup tapes at a cold site. ■■ RTO of 12 hours: Restore from tapes at a hot site.
■■ RTO of 4 hours: Use a data vault to a hot site. Chapter 11 n Introduction to Business Continuity 235 ■■ RTO of 1 hour: Cluster production servers with controller-based disk mirroring. ■■ RTO of a few seconds: Cluster production servers with bidirec- tional mirroring, enabling the applications to run at both sites simultaneously. ■■ Data vault: A repository at a remote site where data can be periodically or continuously copied (either to tape drives or disks), so that there is always a copy at another site.
■■ Hot site: A site where an enterprise’s operations can be moved in the event of disaster. It is a site with the required hardware, operat- ing system, application, and network support to perform business operations, where the equipment is available and running at all times. ■■ Cold site: A site where an enterprise’s operations can be moved in the event of disaster, with minimum IT infrastructure and environmental facilities in place, but not activated. ■■ Cluster: A group of servers and other necessary resources, coupled to oper- ate as a single system.
Clusters can ensure high availability and load balanc- ing.