How long can your business survive without your data? How long can your business survive if your people can’t access your business via the internet?

Can you do disaster recovery to the Cloud, so that if your datacenter is cut off, you can failover to a virtual infrastructure, if only for a short while? Is this possible? Is it affordable? If it is, and you can do it and it works, then…why are you in the business of building and running a datacenter at your headquarters?

How can you recover your servers in the cloud? There are many technologies that are now within reach, including host and virtual machine replication (not storage array replication, $$$).

Here is a run-down of those applications and tools that enable you to achieve a recovery time objective of half a day or less, and a data loss of about the same.

Rapidly Changing Technology

I started writing this series a few months back, after a major earthquake took down systems in DC, Virginia and Maryland. Back then, I was exploring many options that have already changed, evolved or been superseded. This is one of the challenges of this area of technology.

We have also put into place a few hybrid backup solutions that have shown some of the challenges that face this area of IT solutions.

I designed a backup and archive system for a customer that uses a major backup software suite, CommVault, to back up their virtual servers and write them to local disk. This is working very well. They also are using a virtual appliance that presents cloud storage as iSCSI LUNs. This is from TwinStrata, and it is called CloudArray Controller. The customer likes this very much. They are using the CloudArray Controller to present a new drive to their file server that they can drag old files and directories to, a manual archiving process. This works great – for uploading. They just got back to me and said that the download speeds are painfully slow, around 200 kbs. That seems to be a glitch, and it is being worked on by the cloud storage provider (using Atmos as the platform). Hopefully we’ll find a throttle that was turned on by mistake. If not, that could present a major challenge to adoption. Other cloud storage platforms might have much better download speeds, but we eliminated the ISP, the cloud array controller software, and CommVault as the culprit. A Speed Test can verify this. (Update: different cloud storage platforms have different “object sizes” – think of the standard IO or block size of storage platforms and file systems – that aren’t published. TwinStrata has been independently testing some of them and has a little black book they use in helping people set up their appliance for best performance. They measure performance as they change the cache block size until they see a significant spike in performance and note that for future tuning reference. Pretty slick, but old-school engineering. I like that.)

The implications of extracting and recovering IT applications of any size are obvious, if you have to download. This is where recovery in the cloud becomes important, but only if there is a high bandwidth, low latency connection between the storage and the computing environment. This is not a widely available capability. I know that Amazon has such connections in certain zones. So architecting a solution requires an understanding of how the computing and storage solutions are interrelated.

Cloud Storage and Cloud Computing

The relationship of these two technologies is poorly understood, and this has to change. This market is ripe for maturing, and until it does, you need someone who has hacked a path through this jungle already. Backup to the cloud – fine. Archive to the cloud – fine. Replication of virtual servers from “home” to the “cloud” – fine, but can be expensive.

Cloud Services is too broad a term, as well.

I can recommend cloud storage. Use cases are for backup and archiving but also regular file serving. The customer can provision storage on their own in conjunction with CommVault backups (Symantec and others are adding “cloud storage APIs” to their software now, but CommVault was the leader here) or TwinStrata use cases (see below) or can engage an online backup service provider.

Online Backup Providers are a Cloud Storage service, specifically around backing up data from customer premises to an undefined but secure location “in the cloud.” This is a managed service in the true sense, and cloud services in the broad sense: scalable, on demand, pay as you go. It can be elastic (shrink if not needed) depending on the contract. Asigra is a major managed service provider software vendor that many services have built their products around. Some of the cloud backup vendors have their own software and their own data centers and some license something like Asigra and buy cloud storage in volume from some provider such as Amazon or Rackspace.

We have another cloud storage technology partner, Nasuni, which provides a Cloud File System. The use case here is a shared file system for multiple offices or locations, with local performance. Backup is NOT a use case. The cloud storage service provider is pre-selected and costs are built into the solution. This has an annual fee. The beauty of this solution is that you get local performance due to caching on local disk. But even more important, this solution provides that single location for file storage that has no file locking or replication issues (other than updating local cache). I don’t know of any solution that does this as elegantly as Nasuni. They have even gone so far as to load test various cloud storage providers. They don’t want to necessarily reveal their current choice because they may change as the market develops and they find a better partner in terms of profit or performance or both. I can tell you that one of the major players asked them to stop the testing because it was bringing down their storage systems! (Define that, you ask – performance was being substantially degraded, is the response, so that customer latency soared.)

We have another cloud storage technology partner, Twin Strata. They offer a virtual or physical appliance that connects to any number of cloud storage providers (not preselected) and presents an iSCSI LUN (block level storage) to the customer servers. TwinStrata has proven to be stable, deliver high performance, and has great support. Use cases for TwinStrata’s technology is broader than just file systems, and may turn out to be one of the most significant cloud storage approaches. The cost of the appliance is by subscription or purchase, and the cloud storage needs to be provisioned separately. Provisioning of cloud storage is the hang-up for the resellers out there – the reseller model for cloud storage is very immature at the moment. Most cloud storage providers want us (the VAR) to buy storage capacity and then resell it. We want to sell it without buying it ourselves. Amazon has a good reseller model that they just rolled out. I hope others will do so soon.

Cloud Computing is another matter, and we have Terremark as a Cloud Service Provider (CSP) partner. They do not have a storage-only offering. Amazon, Rackspace, Nirvanix, iLand – all have computing services offerings of differing natures, but the fundamental nature of this service is essentially Managed Hosting. Instead of putting in a ticket to AT&T, for example, where a team racks and configures a server, its storage, and networking, now a couple of guys sit at a console and do the same thing with a few keystrokes. There is a ton of innovation and new technology behind this, but the bottom line for the CSPs is better margin since the labor and time to provision is reduced, and this should or could be passed along to the customers. The customers also can self-provision, but it is just like an online catalog – fill up your cart with CPU cycles and disk space and pay at the cashier. SLAs and performance expectations must be set. Until then, it is unlikely, as noted in another post, that enterprises will put the mission critical stuff in a “cloud” service, nor will Unix platforms ever be virtualized and managed in the same way.

Conclusion

I intend to focus future postings on point solutions and use case scenarios for storage in the Cloud. I hope that this three part introduction to Highly Available data has prompted some thought and given a background to this emerging technology space.

How long can your business survive without your data? How long can your business survive if your people can’t access your business via the internet?

What kinds of data have to be “always on” for your customers and your business? How will you make sure that those data and applications are highly available? What has to be done to your computing, storage and networking infrastructure to keep your business running? Back in the twentieth century, the answers were quite a bit different, and if you didn’t have a million dollar budget, you couldn’t really make sure that your business was accessible even if the power went out at corporate headquarters.

So, what about the Cloud? Can you do disaster recovery to the Cloud, so that if your datacenter is cut off, you can failover to a virtual infrastructure, if only for a short while? Is this possible? Is it affordable? If it is, and you can do it and it works, then…why are you in the business of building and running a datacenter at your headquarters?

 “Disaster Recovery In The Cloud”!

Like many things “Cloud”, this is a concept and not a product. The “productization” of Cloud DR is proceeding apace, though. Here are some of the things to keep in mind as you try to figure out a DR strategy in the 21^st century.

• Cloud Computing platforms, Infrastructure as a Service (Iaas) is the new Managed Hosting for data center operations. This technology allows the data center operator economies of scale for server and networking deployment for customers – a few guys are provisioning for customers on demand without running around slapping servers and switches into racks.
• Cloud Computing Portals, done right, allow the customer to provision their own servers, based on their own builds (hardening,patches, security, apps), without requiring datacenter staff to actually be “remote hands”.
• Cloud Computing provides network, computing and storage ON DEMAND, meaning not only “standing up” servers, but also adding capacity in all three respects.
• Cloud Computing cannot host all critical applications, depending on the platform: AS400, Solaris, AIX and HPUX all do not fit into any Cloud platform that I know of, although they all have their own virtualization strategy (zones, LPARs, workloads, etc.).
• Multi-tenant storage and computing platforms raise security concerns.

So where does this leave the cloud as a DR or even a primary datacenter “location”?

You, the business representative, have to figure out what applications are suitable for the Cloud computing platform, or Managed Hosting/Collocation infrastructure. Nothing has changed in that regard. The only thing that has changed is the method and speed of provisioning certain aspects of your business technology platform.

A true Business Continuity solution will take into account much more than a DR strategy (typically confined to recovering the IT infrastructure and applications). Your “DR In The Cloud” strategy will need to confine itself to the Windows and Linux applications and data that are supported in the Cloud platform of your choice. Typically, this includes the basic IT infrastructure and many business applications, so this may work just fine for you: AD, Exchange, SQL (in most cases), Web and middle-ware, file serving. As was pointed out to me at a recent speaking engagement, enterprises have much more complex requirements (e.g. mainframe, Exadata, etc.) and more resources, so they are excused from class now….

Ok, now you commercial and SMB companies, now that those big guys have left the room, this solution is FOR YOU. As I said before, major replication and HA technologies have been, and still are, out of reach and scope for many companies, and this new kind of solution is for you – not Them.

How can you recover your servers in the cloud? There are many technologies that are now within reach, including host and virtual machine replication (not storage array replication, $$$).

Next week, a run-down of those applications and tools that enable you to achieve a recovery time objective of half a day or less, and a data loss of about the same. I promise.

Email and file system backup and archive retention spans, when not bound by specific business or regulatory compliance requirements, can still vary considerably. The cost of longer term retention, as well as its usefulness or relevance to the business as the backup (or archive) ages, are the main factors in deciding on a policy.

In my experience as the lead engineer for a managed backup service which handled backups for all AT&T hosted data centers in the United States (over 3 PB/year in volume and several hundred corporate customers), 99% of the backup retention requirements were 4 weeks. That’s it, 4 weeks. It is a known statistic in the backup industry that over 90% of restore requests are made within one month of the data’s creation (file or email). Many customers will add a few layers of backups, known as the Grandfather/Father/Son scheme, as follows:

• Daily: retained 2 to 4 weeks
• Weekly: retained 1 to 3 months
• Monthly: retained 6 months to 1 year

The main purpose of the Grandfather backup, the long term retention of a monthly backup, is for archiving. Archiving is long term data storage of older data, which (strictly speaking) has been removed from the primary storage. This is necessary to keep the primary data storage medium growth to manageable and affordable levels. This often comes into play in user data (files, spreadsheets, PDFs) and in email, which can quickly grow out of control when users treat their inbox as a defacto file system.

Email archiving has additional useful characteristics such as Single Instance Storage (removing all duplicate attachments from group emails, keeping one copy), mailbox storage management (archiving/removing old emails from the mailbox), journaling (keeping a copy of every email sent or received or sent throughout the day) and compliance (searching those journaled copies for key words or addresses for legal discovery). Retention of the archived messages and attachments is often no more than 2 to 3 months but can be as long as 6 months. If journaling and legal discovery are an important consideration for the business, the retention of the journaled emails could be up to a year. If there are no compliance or regulatory requirements, email does not need to be retained for any longer than 6 months to a year. The cost of the storage can become financially unsustainable for longer retention periods.

In conclusion: backups do not need to be retained for longer than 4 to 8 weeks, in most cases. Archived backups or emails can be kept for longer periods, especially if deduplicated, but most companies do not keep such data for longer than one year.

Retention of electronic data should be consistent with written record keeping policies and should be documented for use in the event of a legal discovery. Any requests for data outside of the documented retention period can then be safely denied, reducing the costs and risk incurred with legal discovery events.

These opinions and best practices are based on over 10 years of storage and backup administration experience, several of which were at a managed storage service provider handling thousands of backups daily for major internet data center managed hosting and collocation facilities. These standards are broadly applied and observed in many commercial sector organizations retention policies for electronic data.

How long can your business survive if your people can’t access the internet? How long can you go without cable or satellite TV at home, especially in football season or <insert your sport here>?

How long can you go without a mobile phone signal, where you can’t text, call or browse with your BB/Android/iPhone?

Recently, I  “survived” an earthquake near DC, where I live, and I was not able to access the internet, call my wife at work or on her mobile phone (she has AT&T, I have Verizon), or use my home phone (Comcast) or my work phone (Cisco IP phone, connected via the cable broadband service). That outage lasted just a short while (an hour, maybe), but it disrupted my personal life and my work life. I was on a conference call that did not get dropped, with people in three different states up and down the Eastern Seaboard, and we all got to share in my personal earthquake experience as I ran out of the house with my kids and dogs and waited for the earthquake to end. Businesses and many government agencies closed (did you notice?), traffic was snarled, and communications and data access were disrupted. More than an inconvenience, this cost some serious money, but the data and communications infrastructure rebounded very quickly, thanks to some serious data availability planning and design.

I just got back from vacation on Cape Cod, near Falmouth, probably 100 miles as the seagull flies from downtown Boston. I had a cell signal at my rented cottage, but it was so weak (probably 2G) that I could not use my cellular modem (Verizon MiFi Access Point – great product, rush out and get one!) to access the internet! Disaster! Calamity! Frustration! I waited 15 minutes for Google to load! Well, you say, you were on vacation, so get over it. Sorry, I had to approve time cards and expense reports. I had to download & evaluate some data to help a team meet a deadline for a consulting engagement, which could not be viewed on my BlackBerry – something that would have taken 10 minutes total back in the connected world, but was impossible for me. I had to drive to town to find a Starbucks (actually used the free broadband at the public library) to do this seemingly simple task.

Do you see where we are headed? We have to be plugged in, at home and at work. What can you do about the cable company, or the backhoe operator that just cut the cable? Not much. What can you do about terrible phone reception, or no service at all? Not much. What can you do about making sure that your customers can always access your business on the Internet, and your employees can always access their applications and data? Quite a bit, actually.

So, back to my first question: What kinds of data have to be “always on” for your customers and your business? Do you know? I’ll wait while you think it over…

Done? OK, now how will you make sure that this is highly available data and applications? What has to be done to your computing, storage and networking infrastructure to keep your business running? Back in the twentieth century, the answers were quite a bit different, and if you didn’t have a million dollar budget, you couldn’t really make sure that your business was accessible even if the power went out at corporate headquarters. Oh, you could protect the data from loss and corruption, but no one could access it from “outside.” Clustering, mirroring, replication, tape backup, generators – do these solutions sound familiar? What about moving your computing and storage infrastructure to a hosted or collocation data center? Expensive and painful, this solution was out of reach for many, and all applications couldn’t be accessed this way.

So, what about the Journey to the Cloud? Will this take care of it? Can you do disaster recovery to the Cloud, so that if your datacenter is cut off, you can failover to a virtual infrastructure, if only for a short while? Is this possible? Is it affordable? If it is, and you can do it and it works, then…why are you in the business of building and running a datacenter at your headquarters?

Stay tuned for the answers to these and other questions, and some possible solutions, with “Disaster Recovery in the Cloud”!

Document your own security requirements and audit yourself (do you live up to your own standards?), and publish your security requirements to see if a “cloud,” or even hosted solution, will be able to meet your requirements.

I think that virtualized networking and security tools should allow one to build a self-contained virtual infrastructure that can be portable. You should be able to move it from one site to another and preserve a secure perimeter, regardless of the physical infrastructure provider (another private datacenter, a managed hosting operation, or a “cloud”). If such is the case, the security SLA of the cloud provider becomes less of an issue.

An example is a new technology for storage that encrypts the data written to the storage, regardless of the encryption offered by the storage provider (e.g. What if the provider key is compromised? No problem if the data is encrypted as it is written to “cloud” disk…).

I am thinking about a vision of a fully virtualized datacenter, including networking and security, which would allow it to be moved from site to site and preserve all elements no matter where it was moved to. This would be a sort of virtualized “pod” or “datacenter on wheels.” The wheels, in this case, would be the internet.

The portable (or virtualized) datacenter has its routing, switching, firewalls, IDS, load balancing, servers and storage self-contained in the “trailer.” All the owner of this mobile datacenter would need is a place to park it (the Cloud), a connection to the internet, and power.

So, if Altor (or something competitive), in combination with Cisco/Juniper virtual switches and VMware tools (vShield?), can provide that capability, does it matter (as much) about some of the details about the Cloud hosting provider? When I deployed infrastructure building blocks to one particular large telecommunications company’s datacenters, I built my own network and firewall, and didn’t rely on the company to do that for me. What is the difference here? All I was concerned with in this certain instance was physical security. All datacenters meet that basic requirement, and all the tenants use the POP at the datacenter, but they all have separate self-contained networks for the applications.

Agree, disagree?

To add insult to injury, you are running out of space on your file servers or NetApp or Celerra, and you need to buy more disk shelves and hard drives. Wait, isn’t that array coming off warranty causing you to have to pay $20,000 for the next YEAR?? Someone just called and said the Windows file server that the corporate office is using is giving a warning that it is about to run out of space, but there is no downtime allowed and DON’T ARCHIVE ANYTHING. Did we forget to mention that you haven’t finished a full back up on time in a month? Budget numbers are due in the next two weeks. What will you do?

• What if you could find a product to help you get rid of the clutter and junk, reducing the use of your storage by 50%, without the end users even being aware that their old data is now on a SATA array stuffed with 1 TB drives?
• What if you could now complete your backups with hours to spare?
• What if you can finally prove that Marketing has been storing multiple copies of those huge PowerPoint presentations with wanton disregard for the impact on overall storage utilization?
• What if you could put off the storage expansion indefinitely?
• What if you could migrate the data off of that old array without any downtime, and push that boat anchor out to the loading dock and save $20,000 in your budget?

What if?

It will only get worse. According to IDC, way back in 2005, average growth rate of corporate data was 60% annually. If you need 10 TB today, you will need 16 TB next year, unless you take action. Your storage services model will NOT BE SUSTAINABLE without a new way of looking at managing storage.

File System Virtualization

We have virtualized servers, and we have virtualized storage. Now the time has come to tackle the messiest and biggest storage problem of all: cluttered, congested, sprawling file storage systems. All of the scenarios I described above are based on real life situations, with real GreenPages customers. What they all have in common that file system virtualization can help them with, is files (lots and lots of them) being served to their end users via Windows Shares (CIFS) or NFS exports, on Windows, Linux, Unix, NetApp or EMC Celerra file servers. (Sorry, Mac and Novell users, we have to talk to you separately…).

What does “File System Virtualization” mean? It means that you separate the physical location of your files from the means to access them. Put in place a proxy to the file systems without disruption to your business. No agents, no forklift upgrades, no sweeping changes of your client drive mappings. This can be done at network speeds. The end user requests a file from his home directory, sending the request to the proxy appliance which has mapped out the location of all of the files on every filer and server in the network. The request is terminated, and a new one is created to fetch the file and return it to the end user, with no one the wiser. This is done largely in hardware, at RAM speeds. No latency.

Take it a step further. You scan those file systems, and move everything that hasn’t been touched or changed in a year off of your expensive/old/overloaded filer to some new inexpensive iSCSI storage (about $2/GB!!!), and you have freed up 65% of your storage! Back up the stale data once a month, slam shut that big backup window, call your storage vendor and tell them to cancel that order for the new storage array, or better yet, trade in the old Chevy for a new BMW and show off the latest in thin provisioning and built-in storage virtualization features, all without a “maintenance window.”
Now you have a real archive system that reduces the file system utilization without “stubs” or pointers! Migrations for replacing hardware or load balancing take place without the pain of Robocopy or Rsync.

What Can Do All This? What Is the Solution?

What can do this? F5 Acopia ARX Appliances, in combination with any number of second tier or nearline storage devices that your Friendly Neighborhood Storage Architect can recommend.

• Archive to Enterprise Vault FSA? Done.
• Archive to Data Domain? Done. (Oh, and this solution REPLICATES your archived data to an undisclosed offsite location.)
• Tier your gluttonous Art Department’s image files to some inexpensive EMC iSCSI storage (yes, they make inexpensive storage….sheesh!)
• Trade in that old SDLT tape library for a sleek little LTO4 autoloader.