Disaster Recovery in the Cloud, or DRaaS: Revisited

Randy Weis

By Randy Weis, Consulting Architect, LogicsOne

The idea of offering Disaster Recovery services has been around as long as SunGard or IBM BCRS (Business Continuity & Resiliency Services). Disclaimer: I worked for the company that became IBM Information Protection Services in 2008, a part of BCRS.

It seems inevitable that Cloud Computing and Cloud Storage should have an impact on the kinds of solutions that small, medium and large companies would find attractive and would fit their requirements. Those cloud-based DR services are not taking the world by storm, however. Why is that?

Cloud infrastructure seems perfectly suited for economical DR solutions, yet I would bet that none of the people reading this blog has found a reasonable selection of cloud-based DR services in the market. That is not to say that there aren’t DR “As a Service” companies, but the offerings are limited. Again, why is that?

Much like Cloud Computing in general, the recent emergence of enabling technologies was preceded by a relatively long period of commercial product development. In other words, virtualization of computing resources promised “cloud” long before we actually could make it work commercially. I use the term “we” loosely…Seriously, GreenPages announced a cloud-centric solutions approach more than a year before vCloud Director was even released. Why? We saw the potential, but we had to watch for, evaluate, and observe real-world performance in the emerging commercial implementations of self-service computing tools in a virtualized datacenter marketplace. We are now doing the same thing in the evolving solutions marketplace around derivative applications such as DR and archiving.

I looked into helping put together a DR solution leveraging cloud computing and cloud storage offered by one of our technology partners that provides IaaS (Infrastructure as a Service). I had operational and engineering support from all parties in this project and we ran into a couple of significant obstacles that do not seem to be resolved in the industry.

Bottom line:

  1. A DR solution in the cloud, involving recovering virtual servers in a cloud computing infrastructure, requires administrative access to the storage as well as the virtual computing environment (like being in vCenter).
  2. Equally important, if the solution involves recovering data from backups, is the requirement that there be a high speed, low latency (I call this “back-end”) connection between the cloud storage where the backups are kept and the cloud computing environment. This is only present in Amazon at last check (a couple of months ago), and you pay extra for that connection. I also call this “locality.”
  3. The Service Provider needs the operational workflow to do this. Everything I worked out with our IaaS partners was a manual process that went way outside normal workflow and ticketing. The interfaces for the customer to access computing and storage were separate and radically different. You couldn’t even see the capacity you consumed in cloud storage without opening a ticket. From the SP side, notification of DR tasks they would need to do, required by the customer, didn’t exist. When you get to billing, forget it. Everyone admitted that this was not planned for at all in the cloud computing and operational support design.

Let me break this down:

  • Cloud Computing typically has high speed storage to host the guest servers.
  • Cloud Storage typically has “slow” storage, on separate systems and sometimes separate locations from a cloud computing infrastructure. This is true with most IaaS providers, although some Amazon sites have S3 and EC2 in the same building and they built a network to connect them (LOCALITY).

Scenario 1: Recovering virtual machines and data from backup images

Scenario 2: Replication based on virtual server-based tools (e.g. Veeam Backup & Replication) or host-based replication

Scenario 3: SRM, array or host replication

Scenario 1: Backup Recovery. I worked hard on this with a partner. This is how it would go:

  1. Back up VMs at customer site; send backup or copy of it to cloud storage.
  2. Set up a cloud computing account with an AD server and a backup server.
  3. Connect the backup server to the cloud storage backup repository (first problem)
    • Unless the cloud computing system has a back end connection at LAN speed to the cloud storage, this is a showstopper. It would take days to do this without a high degree of locality.
    • Provider solution when asked about this.
      • Open a trouble ticket to have the backups dumped to USB drives, shipped or carried to the cloud computing area and connected into the customer workspace. Yikes.
      • We will build a back end connection where we have both cloud storage and cloud computing in the same building—not possible in every location, so the “access anywhere” part of a cloud wouldn’t apply.

4. Restore the data to the cloud computing environment (second problem)

    • What is the “restore target”? If the DR site were a typical hosted or colo site, the customer backup server would have the connection and authorization to recover the guest server images to the datastores, and the ability to create additional datastores. In vCenter, the Veeam server would have the vCenter credentials and access to the vCenter storage plugins to provision the datastores as needed and to start up the VMs after restoring/importing the files. In a Cloud Computing service, your backup server does NOT have that connection or authorization.
    • How can the customer backup server get the rights to import VMs directly into the virtual VMware cluster? The process to provision VMs in most cloud computing environments is to use your templates, their templates, or “upload” an OVF or other type of file format. This won’t work with a backup product such as Veeam or CommVault.

5. Recover the restored images as running VMs in the cloud computing environment (third problem), tied to item #4.

    • Administrative access to provision datastores on the fly and to turn on and configure the machines is not there. The customer (or GreenPages) doesn’t own the multitenant architecture.
    • The use of vCloud Director ought to be an enabler, but the storage plugins, and rights to import into storage, don’t really exist for vCloud. Networking changes need to be accounted for and scripted if possible.

Scenario 2: Replication by VM. This has cost issues more than anything else.

    • If you want to replicate directly into a cloud, you will need to provision the VMs and pay for their resources as if they were “hot.” It would be nice if there was a lower “DR Tier” for pricing—if the VMs are for DR, you don’t get charged full rates until you turn them on and use for production.
      • How do you negotiate that?
      •  How does the SP know when they get turned on?
      • How does this fit into their billing cycle?
    • If it is treated as a hot site (or warm), then the cost of the DR site equals that of production until you solve these issues.
    • Networking is an issue, too, since you don’t want to turn that on until you declare a disaster.
      • Does the SP allow you to turn up networking without a ticket?
      • How do you handle DNS updates if your external access depends on root server DNS records being updated—really short TTL? Yikes, again.
    • Host-based replication (e.g. WANsync, VMware)—you need a host you can replicate to. Your own host. The issues are cost and scalability.

Scenario 3: SRM. This should be baked into any serious DR solution, from a carrier or service provider, but many of the same issues apply.

    • SRM based on host array replication has complications. Technically, this can be solved by the provider by putting (for example) EMC VPLEX and RecoverPoint appliances at every customer production site so that you can replicate from dissimilar storage to the SP IDC. But, they need to set up this many-to-one relationship on arrays that are part of the cloud computing solution, or at least a DR cloud computing cluster. Most SPs don’t have this. There are other brands/technologies to do this, but the basic configuration challenge remains—many-to-one replication into a multi-tenant storage array.
    • SRM based on VMware host replication has administrative access issues as well. SRM at the DR site has to either accommodate multi-tenancy, or each customer gets their own SRM target. Also, you need a host target. Do you rent it all the time? You have to, since you can’t do that in a multi-tenant environment. Cost, scalability, again!
    • Either way, now the big red button gets pushed. Now what?
      • All the protection groups exist on storage and in cloud computing. You are now paying for a duplicate environment in the cloud, not an economically sustainable approach unless you have a “DR Tier” of pricing (see Scenario 2).
      • All the SRM scripts kick in—VMs are coming up in order in protection groups, IP addresses and DNS are being updated, CPU loads and network traffic climb…what impact is this?
      • How does that button get pushed? Does the SP need to push it? Can the customer do it?

These are the main issues as I see it, and there is still more to it. Using vCloud Director is not the same as using vCenter. Everything I’ve described was designed to be used in a vCenter-managed system, not a multi-tenant system with fenced-in rights and networks, with shared storage infrastructure. The APIs are not there, and if they were, imagine the chaos and impact on random DR tests on production cloud computing systems, not managed and controlled by the service provider. What if a real disaster hit in New England, and a hundred customers needed to spin up all their VMs in a few hours? They aren’t all in one datacenter, but if one provider that set this up had dozens, that is a huge hit. They need to have all the capacity in reserve, or syndicate it like IBM or SunGard do. That is the equivalent of thin-provisioning your datacenter.

This conversation, as many I’ve had in the last two years, ends somewhat unsatisfactorily with the conclusion that there is no clear solution—today. The journey to discovering or designing a DRaaS is important, and it needs to be documented, as we have done here with this blog and in other presentations and meetings. The industry will overcome these obstacles, but the customer must remain informed and persistent. The goal of an economically sustainable DRaaS solution can only be achieved by market pressure and creative vendors. We will do our part by being your vigilant and dedicated cloud services broker and solution services provider.

*Editors Note: This blog post ended up inspiring a new offering from GreenPages! Click here to learn more about GreenPages DRaaS vStream offering.

 

 

 

 

 

 

 

 

 

 

  • http://twitter.com/richardjferrara Richard J. Ferrara

    Hey Randy. DR to the Cloud, DRaaS, yes, been hearing more about this. Sounds fabulous in theory. I guess it depends on what you consider DR. Consider the following questions:

    1. Do you have ALL of your systems virtualized?
    2. If not, would you convert those few systems to a virtual environment?
    3. Would you run all of your systems, both critical and non-critical, in the cloud?
    4. Do you need to run at production speeds in a DR scenario?

    If your answer is yes to 1 and 3 or all 3, then yes, you are well on your way to DR in the cloud or even DRaaS. For #4, please keep that one in mind. You may think you can run degraded, but trust me, when you have to run, you want that speed.
    Here’s where I question the Cloud….the performance of it. Especially on my most critical systems. So if yes on #4, DR cloud may not be for you and you may want to invest in your own DR datacenter.

    Next set questions

    1. Do you need to get ALL your systems up quickly in a DR panic? Yes, I said panic.
    2. Do you have short RPO and RTOs?

    Depending on the answer will depend on the scenario that test fits you? If it’s yes to both
    questions, then you need a good form of replication ready to make that switch at any time and scenario 1 will not fit your needs. Scenario 2 would be your best fit. Scenario 3 could work, but SRM will only give you 5 minute RPO at best. Don’t ignore those RPO/RTOs!

    If you’re not in a rush to get your systems up and running, scenario 1 should work for you.

    Finally, whatever solution that is decided, you must use it. Commit to it. If you don’t, you will never know if your solution will work when you need it.

    • http://twitter.com/weis_randy Randy Weis

      Richard, you make some very good points for people that are trying to develop DR strategy, not just “DR in the Cloud”. And your wrap-up is perhaps one of the most significant points – committing to a DR strategy that requires testing and validation – not once, but over and over. I know that your organization’s DR strategy is much more than “just” DR – you actually are set up to use two sites interchangeably to correspond with Hurricane season. Performance matters, as does regular maintenance of your solution!

      So, to point #4 – performance in the cloud can be difficult to guarantee. I don’t know of any IaaS provider that supplies Service Level Agreements around anything but Availability, and usually only 3 or 4 nines (99.9% or 99.99%). I don’t know when this will change, but it has to.

      There are many applications not suited to a “public” (I prefer “privileged”) cloud – if the application has very stringent server-to-server latency & performance requirements within the application stack, based on operating systems that aren’t supported in hypervisors (generally UNIX) or requiring physical connectivity to certain types of devices. These sorts of applications (banking, financial, life sciences, etc.) are not suitable for any service provider cloud that I know of, and should be handled as an on-premise solution, or more traditional off-premise solution (colocation facility) where the organization builds and maintains their own hardware & software stack. This doesn’t rule out the use of virtualization technology, just the use of “public” or multi-tenant infrastructure that we are now referring to as “cloud” or IaaS.

      Some service providers are building out dedicated solutions for this sort of thing, but some of the features of “cloud” (access anywhere, scale-up/scale down elasticity and self-provisioning) are absent or restricted. If a service provider is willing to build a dedicated vCloud or OpenStack based solution that does allow for many of the cloud attributes, and is affordable (the formula of risk and cost to mitigate…), then perhaps a DR solution of this sort based on IaaS can work.

      My goal for this blog is to provoke great dialog with vendors and customers (both have succeeded here), and to provide a framework for continuing dialog and investigation for the interested parties. One vendor (well known to Richard) has contacted me to say how they have addressed the SRM replication issue, but they can’t do anything about integration of storage APIs and privileged authentication/access to storage that is required to automate the provisioning of VMs in that emergency (panic) where you need dozens of VMs spun up in a “DR as a Service” scenario. But, like has happened with cloud computing in general, or virtual desktops, we (the marketplace) chip away at the obstacles, roadblocks and business paradigms to help the solution take form in a usable way.

      Thanks for your detailed and useful comment.

  • Pingback: Disaster Recovery in the Cloud, or DRaaS: Revisited « Quick Disaster Recovery.com

  • Pingback: Cloud Computing Backup Solution