During my college years, Margo Seltzer rewarded her research group with a nice dinner at a lovely Spanish restaurant. Unfortunately, my dish required me to cook my own food – raw chicken, shrimp and fish – on a hot rock. As you may have already surmised, I didn’t cook my food long enough. And so the following 24 hours culminated in a moment of true clarity: “Do It Yourself” is often not good enough.

WARNING: “Your Mileage May Vary”
There is no phrase in the technical lexicon that infuriates me more. When I buy technology, I want to know how much load it can handle in my environment. Anything less and I feel like I’m getting some raw products and hot stone. And that’s why I am a true believer in the concept of an appliance. 

For those of you who haven’t read about my background on this site, I joined NetApp straight out of school in 1997, when it was still Network Appliance. I loved the idea of being able to buy a pre-baked NAS solution instead of trying to “cook your own answer” with host-based file system software, volume managers and storage infrastructure. With an appliance, you knew the pieces would interoperate and you knew how they would perform, which meant you knew “your mileage wouldn’t vary much.”

However, as with so many IT terms (e.g., “social media,” “platform,” “cloud,” etc.), the meaning of “appliance” was diluted over time (often by clueless companies and well-intentioned media). Even more frustrating, the technology itself became so complex that you really didn’t know what you were getting from an appliance anymore. “Your mileage did vary.” And backup was one of the main causes. In fact, I still have nightmares…

Sales Rep (at deal close): “Of course you can store 1 billion files on this box.”

Customer: “Why don’t your backups run faster?”

Young Stephen Manley: “No, there’s no viable way to back up those 1 billion files via NDMP.”

Customer (clearly not listening): “Why don’t your backups run faster?”

Young Stephen Manley (aging quickly): “Because you’re running network backups over a 10-Mbit Ethernet to a DLT4000.”

Customer: “Well, we didn’t have enough money to pay for upgrading our backup environment.”

Young Stephen Manley (to himself): “Aaaauuuugggghhhh!!!!! Then what did you expect?!?”

The Rebirth of the Appliance
The introduction of converged infrastructure offerings like Vblock and FlexPod rekindled my faith in appliances; it was the first evolutionary step in years. As one customer put it, “Now I’m not just buying files in a box… I can buy a chunk of data center in a box.” Network, compute, storage and protection – proven to interoperate with qualified performance levels.

While Vblock represented the new wave of appliances that focused on simplifying your environment, it’s not a fit for everyone. We want to spread the “next generation appliance” to a wider audience.

Therefore, I’m very excited about our VSPEX launch. While VSPEX is not the sheet-metal-enclosed appliance form factor of my youth (things have evolved since then), it does deliver the appliance value proposition:

1. Certified interoperability
2. Validated to deliver specific performance/scale

Of course, the part I like best about VSPEX is that the appliance includes backup – Avamar or Data Domain, depending on the configuration. This is a huge leap forward for appliances. No longer is backup an afterthought to the solution. No longer are you condemned to deploy an environment that has only enough horsepower to serve the application, but not enough to protect it. And that sure beats trying to cook your own over a hot stone!

These passengers got me thinking, not about the state of U.S. healthcare, but about the insidiousness of comfortable, long-standing debates. “Tape vs. disk backup” is one of those comfortable debates, as our own Heidi Biggar pointed out earlier this week. After following the thread of the various posts out there , I’ve got to say it does feel like I’m in the Year 2000 again.   Some of the claims are so extreme that they collapse under their own absurdity, even without refutation. Never mind that disk backup has become such an accepted practice that it’s not worth revisiting “tape vs. disk” again and again, and yet one more time. Despite my best intentions, I can’t help but rejoin the “tape vs. disk” fracas. And I know better.

There are three main use cases for tape – but all three use cases are disappearing:

1. Legacy (a.k.a. “Don’t touch it, you’ll break it.”): I meet customers with environments that are neither growing, or subject to more aggressive requirements. Their tape backup infrastructure is working fine, so they don’t mess with it.  Reality: Nothing remains stagnant forever. Legacy environments are either being virtualized or “appliance-ized,” or the requirements are changing. Regardless, things will need to change. For example, when I say “mainframe,” most people immediately think “legacy. ”I’m having multiple conversations about leveraging virtual tape in mainframe environments (our DLm product line) every week. Nothing stands still.
2. Long-term retention, without need for recovery (a.k.a. “Taking crazy things seriously is a serious waste of time.”): I recently met a Venezuelan customer who explained that Hugo Chavez decreed that his company needs to retain backups for 30 years. Interestingly, there is no requirement to recover that data. (Note: I am not trolling Hugo Chavez; I suspect his definition of a “flame war” is more intense than mine.) Reality: Most companies with irrationally long backup retention periods have never thought about what long-term retention of backups means to them – legally, organizationally, and functionally. Backup teams are using Data Domain’s 10-year backup retention as a forcing function to begin concrete discussions with their legal, compliance and business teams. These are not easy conversations, but they result in rational retention periods (unless, of course, your boss is Hugo Chavez or Fidel Casto).
3. Offsite Disaster Recovery (a.k.a. “My other site is a Honda.”): For many smaller companies, their offsite backup policy is putting backup tapes in the trunk of their car or under their beds. Reality: With more “cloud” offerings for offsite DR, this issue is fading quickly. We have customers who replicate their backups to everybody from a trusted reseller to prominent cloud providers. They now have more room under their beds for their obsolete Dell servers.

If, after all that’s happened in the last 10 years, you still believe that tape is integral to the future of backup, my arguments, or those of anyone else, won’t change your mind. And that’s the danger with comfortable arguments. They tempt us to wallow in the mindless muddiness of irrelevant posturing, instead of engaging in productive discussions. The saddest part of the “train debaters”? Every day they complain about the lack of time to do things that really matter to them.

So, if you’re looking to improve backup and recovery, accelerate IT transformation and deliver immediate and long-term business value, not only is tape not the answer, it’s not even part of the conversation.

Sometimes, though, competition finds you. It started with a simple text message from an old friend at NetApp – “Why does EMC bother showing up to FAST?” FAST refers to the USENIX Conference on File and Storage Technologies, one of the pre-eminent research conferences for storage. In my last couple of years at NetApp, we published 6 papers (2008) and 3 papers (2009). While NetApp’s production has fallen off (2 papers in 2010 and 1 paper in 2011), they were still the silverback gorillas at the conference. This year, the EMC Backup Recovery Systems (BRS) team decided it was time to compete.

The BRS team exceeded my wildest expectations – 3 of the 26 published papers (from a pool of 137 applicants) were ours. This led to an entire session dedicated to the papers, appropriately called “Back It Up.” And, importantly, we picked up bragging rights from NetApp, which had only 1 paper accepted. After all, the only thing more fun than winning is beating your best friends to do it. Or so a competitive person would claim!

Our papers covered a variety of BRS’ innovative research and analysis:

• Characteristics of Backup Workloads in Production Systems – Have you ever asked yourself – “What do all those Data Domain systems do every day?” This paper analyzes the auto-support information of over 10,000 systems to answer just that question. It then deeply explores some real-world systems and their behavior with different types of data/workloads. In a nutshell, backup workloads are very different than primary storage workloads. There’s a reason why purpose built backup appliances (PBBAs) are so popular.
• WAN-Optimized Replication of Backup Datasets Using Stream-Informed Delta Compression – When it comes to saving space, what comes after deduplication and basic compression? Delta compression. Imagine finding two similar, but not duplicate pieces of data (e.g., hoppy and happy) and storing only the unique differences between the two. In this paper, the team explored the efficient implementation of delta compression for replication of backups. Don’t worry, this isn’t just “ivory tower” research; it’s been a Data Domain replication option for years.
• Power Consumption in Enterprise-Scale Backup Storage Systems – As more customers adopt Data Domain for long-term backup retention, power consumption has become an increasingly common discussion. This paper analyzes the various components of your Data Domain system that consume power, the advances we’ve made in power efficiency, and some possible next steps. If you want to understand the future of power consumption, this is a must read.

There were some fascinating non-BRS papers, as well. I encourage you to explore all of the freely available papers. As expected, flash and cloud were on virtually everybody’s mind. Also of interest was Revisiting Storage for Smartphones, awarded Best Paper by the conference. While it has little to do with your backup window, I recommend reading through it; it may change how you view your phone.

In case you have forgotten, though, EMC BRS delivered a strong showing at USENIX FAST 2012, and I appreciate all of the team’s hard work – especially that of Data Domain CTO Windsor Hsu. And when you get your own session at a big conference, you need to take advantage of it. That’s why we ran an internal contest for engineers to design t-shirts just for the conference. There really is nothing more satisfying than watching dozens of engineers roam the hallways of a fancy hotel, showing off their backup pride. I’m pretty sure that these shirts will be flying off the shelves faster than any other t-shirts we’ve ever made. Not that I’m competitive about things like that, though.

For some of you, the trends of 2011 may be your 2012 future. For those who have already taken those steps, we have some predictions about what’s awaiting you in 2012.

1)     We’re all backup admins now. Over the last decade, we became an increasingly data-centric world. Without data, the virtual machines, applications, and storage systems don’t do much. Since data is so important, everyone – application, virtual server, storage and backup administrators – wants more control over the protection of their data. With versioned replication becoming available to each of these administrators (Trend #1 from 2011), each team can run its own backups. Disk democratizes backup. Versioned replication delivers the performance that revolutionizes the backup process. The change will apply pressure to two areas:

• Backup software will have to more fully embrace federations (Trend #3 from 2011). The backup application may not always touch the data since the application or hypervisor team may be running their own backups to the centralized backup storage. The backup application, however, must provide centralized visibility and control over all the protection processes in the company.

• Storage-based versioned replication will have to evolve. While storage-based versioned replication was once on the cutting edge of data protection, it needs to innovate to remain relevant. Users demand protection policies at a VM or application granularity – not at a storage container (LUN, file system, volume) granularity. If the storage array cannot meet their versioned replication needs, their backup software, application or hypervisor can.

2)     Backup appliances – quality is job #1. In the past few years, the backup appliance market has focused on speeds and feeds. How fast? How big? While performance and scale will continue to be important, the mantra in 2012 will be “storage of last resort.” As you eliminate tape (Trend #2 from 2011), there is no safety net underneath your backup appliance (and if you do have tape, I’m not sure that’s the safety net you want to trust your business to). When you think about quality of the storage of last resort, remember to ask one thing: “When I need to restore – 5 minutes, 5 months or 5 years from now, will my data be there?” If you’re not sure, why are you wasting your money?

3)     Backup onsite, DR to the cloud. As small and medium businesses and mid-sized enterprises embrace disk (Trend #2 from 2011), their processes must change. They can’t stash their backup appliance in the trunk of their car or under their bed. They need that system to handle today’s restores and tonight’s backups. Meanwhile, they don’t have the bandwidth or faith in the cloud to eliminate all local backup copies. This is the perfect opportunity to add more powerful disaster recovery (DR) to their arsenal. They can efficiently replicate backups offsite to the cloud. Most often, the cloud will be a service offered by their local reseller, to help cope with trust and bandwidth concerns. Regardless of the provider, deduplicated backups and versioned replication will converge backup and DR – and accelerate some organizations’ journey to the cloud – even as they keep their most recent backup copies onsite.

Even a CTO isn’t reckless enough to predict that, in 150 years, there will be multiple magazines debating the battles that transformed the data protection industry in 2012 (but only because I predict that all magazines will be about the American Civil War by then). The world of data protection is changing in profound ways. Over the last 10 years, we have built up the necessary deduplicated disk infrastructure for the revolution. For the first time in memory, you have the chance to regain control, instead of trying to just hang on. You have the opportunity to shape your destiny, instead of merely reacting to others’ actions. You can truly lead the way to a better future. 2012 can be your year.

For those of us who have watched customers flock to disk-based data protection solutions like a phototropic plant bending toward light, 2011 marked the end of a remarkable 10-year run. The shift to disk started slowly with low-cost ATA disk arrays and burst into hyper-drive with deduplication. Within the last 10 years, the question shifted from – “Why would I put disk in my backup infrastructure?” to “You don’t have a dedupe backup appliance?!?”

As deduplicated disk accelerated its inexorable march (would that now make it a relentless double march?) to displace other backup media, we tracked three key emerging trends.

1. Versioned replication – it ain’t just for storage anymore. For the last decade, storage vendors have waxed eloquently about versioned replication between two of their storage arrays. “You can back up only changed data and storing each version as a space optimized full!” In 2011, non-storage vendors embraced versioned replication, to customer acclaim. Versioned replication has finally moved from niche to mainstream. Users aren’t asking “should they implement it” but “how or where they should implement it” – at the backup, storage, application or virtual sever layer?
2. Going tapeless – less talking and more tossing. We’ve had discontent with tape, catchy slogans and explosive growth in backup appliances. Still, prior to 2011, most customers saw truly “going tapeless” as an aspiration, not a plan. And, in 2011, many companies crossed the tipping point. They proved the stability of dedupe disk, bought new cost-effective disk-based solutions for long-term archival (e.g., Data Domain Archiver) and drove pragmatic negotiations to set sensible retention policies (“keep it forever” being the business equivalent of renting multiple apartments to hold all your old People magazines). The result? Many companies have rolled tape straight out the door (see video below).
3. One ring will not rule them all; a federation will unite them. Backup teams stopped fantasizing about exercising absolute control by consolidating onto one centralized backup solution that they own. With complex legacy environments, mergers/acquisitions, a broad constituency of stakeholders (e.g., storage, application, and hypervisor admins) and new workloads (e.g., big data), the goal isn’t one-size-fits-all homogeneity. Instead, successful groups provide centralized backup storage services that can serve their broad and distributed backup constituency. Once they’ve regained control over the actual backup data, they use data protection management tools (e.g., EMC Data Protection Adviser) to gain visibility into their federated protection environments. This insight enables them to provide even better services for their customers.

For some of you, the trends of 2011 may be your 2012 future. For those who have already taken those steps, we have some predictions about what’s awaiting you in 2012. But you’ll have to tune in next week to find out. By then, we will have – “Chumbawamba – Where are They Now?” for you. (We would do “Tape – Where is it Now?” as well, but I made a New Year’s Resolution to avoid dark, depressing places.)

The three customer stories in this series demonstrate the disparity I see in our customer base. People are choosing different approaches based on their organization’s current challenges – technical, business or organizational. While everybody’s road will be different, I do believe the ultimate destination will be a versioned replication solution. Thus, as you plan your approach, I encourage you to explore the trends driving backup and recovery to chart your own course:

• Deduplicated disk remains the first step. It enables you to optimize traditional backups, adopt tools like Avamar, build brute-force versioned replication and evolve to versioned replication solutions, without changing your storage infrastructure.

• Backup and recovery windows will continue to put pressure on data protection solutions. The best way to scale will be versioned replication. You need to decide the layer on which to build the data movement intelligence: backup client, storage, application, or hypervisor. Each layer offers unique characteristics in granularity of management (per-VM, per-app or per-storage container?), flexibility (do I want to lock in on one storage array for both primary and protection storage?) and functionality (how important are traditional backup features?). If you have not considered versioned replication, now is the time. If you have already drawn some conclusions, the optimizations in the VMware and application segment make this an ideal time to explore this area again.

• More administrators – not just backup – will play an active role in protection. Users demand better service and more visibility into their protection schemes, and the only way to meet expectations will be to embrace technical changes like versioned replication. Also, as one company discovered, technical change will depend on organizational change. Backup teams will need to partner with other IT teams (e.g. VM, application, or storage teams) or these teams will find other ways to meet their protection needs. Monolithic backup solutions and teams will not succeed.

Change isn’t easy, but it is necessary. When you combine your insight of new technology trends with your users’ critical needs – you can lead your organization and deliver value. Or, you can be trampled by the wave of new technology. Server virtualization is leading a new wave of technology for backup and recovery. The choices you make today will define your future. They matter.

As for my daughter (for those of you who have been following this series since the start), she eventually put on a pink dress with black polka dots and reveled in the adoration of the masses: “Oh aren’t you just the cutest thing ever?” Clearly, the right choice.

For the first five years of my career, I subscribed to the Will Rogers’ school of problem solving. If customers would wipe out their legacy infrastructure and immediately deploy our newest products across their environments, their problems would disappear. But that was before I met real customers with real data centers and, well, in the immortal words of Keanu Reeves, “Whoa!”

In this blog series, we’ve talked about the variety of mechanisms to protect VM environments and met two mid-sized customers that have deployed innovative solutions to solve their business problems. In this post, you’re going to meet a large customer that described its biggest challenge with one word – “change.” Its complex legacy infrastructure, organizational silos and users’ unwillingness to embrace new initiatives have made it difficult for the company to successfully change anything.

The company, which prides itself on minimizing exposure to cutting-edge technology (its IT team slowly rolls out new technology, allowing it to demonstrate stability and business value and gather user demand before rolling it out more broadly), has successfully deployed thousands of VMs. The company began by virtualizing its test/dev environment and lower-tier applications. But once manufacturing discovered that it could spin up a new application on a VM within 48 hours (versus a 90-day procurement cycle on physical servers), the IT team had its first advocate and the company’s path to virtualization was paved. The company is now virtualizing its business-critical applications.

Following the IT team’s standard technology deployment model, the backup team at this company has deployed new backup technology on an “as needed” basis. Backup and recovery performance has been the forcing function for change. Initially, it protected VMs just like it did physical servers – with a traditional backup to tape. However, when that approach failed to scale, the IT team evaluated two alternatives: guest-level VM backup via Avamar and traditional backups via VMware Consolidated Backup (VCB).

Ultimately, the company chose Avamar. While the initial Avamar roll-out addressed only the heavily loaded ESX server running the Tier-3 apps, the main driver for the remainder of the company was the network. The cost of local network bandwidth made it too expensive to push thousands of full backups across the LAN. Nearly two years later, it uses Avamar for 95% of its VM backups.

However, the company faces new backup and recovery window challenges. With even more heavily loaded ESX servers, Avamar guest-level backups struggle to complete. While it is backing up only the unique, new data, it’s expensive to search through the VMs to find that data. Therefore, as before, it has selected a new technology solution and is selectively deploying it.

The solution: Avamar-driven VMware Changed Block Tracking (CBT)-based versioned replication on the same heavily loaded ESX servers that first deployed the Avamar clients two years ago. While the company is concerned about the scalability of CBT and VM performance, change – specifically, organizational change – is the real issue. How can it trust a backup solution that depends on technology and administrators who are in different groups?

Fortunately, the company has a safety net. Avamar enables it to implement and manage both solutions – Avamar and CBT – from one interface. (“And if this VMware CBT thing fails, we can quietly switch back to the existing approach and not look stupid.”)

Rapid recovery, not rapid backup, has created a groundswell of demand in the organization. With Avamar’s Changed Block Recovery, on three separate occasions, the IT team has recovered applications within minutes. In each case, the data suffered a logical (read: user-driven) corruption. In the past, the company would either have had to run a full restore to an alternate VM or restore all of the files that comprised the application, but with VMware CBT-based versioned replicas, it simply rolled back the changes to the point of the last backup – moving a fraction of the data. (“It was like having a high-end snapshot storage system for all our apps – even the so-called tier-3 apps.”)

Currently, fewer than 10% of this customer’s VMs are protected by VMware CBT-based versioned replication. And while the backup team continues to roll out the new technology as deliberately as possible – incrementally building a relationship with the VM team – it acknowledges the power of VMware CBT-based versioned replication. The company has solved backup and recovery performance problems that seemed intractable before, and users (“for once”) are leading the drive for change in the company.

We all know that boiling the ocean does not solve problems in real customer environments. For most people, change doesn’t start with a clean sheet of paper, it starts with a mess. Therefore, like this customer, you need to set a clear goal and make a path that moves you in that direction. Along the way, you’ll need to embrace new technology, new techniques, and new organizational structures. You likely won’t overcome your VM backup challenges with one grand proclamation – you need to win hundreds of individual battles. In a more pragmatic moment, Will Rogers also said, “Even if you’re on the right track, you’ll get run over if you just sit there.”

While many companies deploy traditional storage-based replication for critical applications and data, the disaster recovery (DR) “solution” for much of their data is still “putting tape in trucks” – that is, if they’re doing offsite DR at all. Not surprisingly, many customers are searching for DR options between “creating a duplicate data center, connected by expensive networks” and “loading the tape and praying.” Fortunately, a clever combination of server virtualization with deduplicated storage can deliver a unified backup and DR solution, as the following mid-sized enterprise case story details.

This particular customer was a mid-sized enterprise with two locations – primary and archive. Initially, the company ran nightly and weekly tape backups, which it shipped to its archive location. While this company wanted a better DR solution, they could not afford to duplicate its server and storage, pay for sufficient network bandwidth to replicate their data, and manage complex DR processes, so it delivered joint backup and DR with VMware and Data Domain.

This company’s approach has brute force simplicity to it:

1. Move all the applications into VMs. Even if the application owner demanded a dedicated server, they placed the application inside a VM.
2. Copy the VMDK files, via NFS, to the Data Domain every night. (The company has a very fast network in the primary site.)
3. Replicate the Data Domain to the archive location, using very little bandwidth.

Initially, this approach merely maintained their backup functionality… but the next step was inspired.

1. Put one standby ESX server and one standby primary storage array in the archive location.
2. In the event of a single-file recovery, boot a VM and mount the VMDK off the Data Domain.
3. In the event of a disaster recovery, boot a VM and mount the VMDK off the Data Domain and storage VMotion the VMDKs associated with the performance-sensitive applications to the standby primary storage.

With this simple and inexpensive approach, this company has delivered (relatively) high-performance DR for critical apps and low performance DR for second-tier apps without compromising its traditional backup/recovery support. In particular, the company prioritizes its Exchange environment and a handful of business-critical databases. For its lower-end databases and file-serving VMs, it is content to run them on the Data Domain. This approach works because the bandwidth to the remote site is its bottleneck – not the Data Domain.

The company is tracking VMware CBT-based solutions, but currently has enough primary-site bandwidth and performance to do brute-force local versioned replication while utilizing deduplication to minimize the bandwidth for offsite DR replication. 

Each company sets different priorities to address in their environment. In the first customer story in this series, the customer needed high-performance VM backups without the cost and limitation of storage-based versioned replication. Here, the customer needed a viable DR solution without spending more money. Over time, each is likely to encounter the challenge that the other has already solved. Fortunately, they won’t be forced to choose between the two priorities: With VMware CBT and Data Domain, they’ll be able to have full-featured high-performance backup with cost-effective DR.

Next time, we’ll explore how a 3^rd customer is balancing its priorities and willingness to deploy new technology.

In prior posts, I talked about the importance of making the right choices, the challenges of providing scalable and efficient protection of VMs as well as some of the technologies (e.g., VMware Changed Block Tracking, versioned replicas, etc.) that IT folks need to be thinking about.  But enough theory – what are companies really doing? Not surprisingly, the customers I’ve met have all chosen different approaches to protecting their VMs. However, despite the variations, there are two common themes: displacing tape with deduplicated disk and innovating to meet backup/recovery windows.

Over the next three posts, I will cover three different customer deployments.

Customer Story #1: From Storage-Based Replication to VMware CBT-Based Versioned Replication

Prior to joining EMC, I spent over a decade helping NetApp build its storage-based versioned replication suite (e.g., SnapMirror, SnapVault, Open Systems SnapVault, etc.). I worked with a variety of exceptional customers.  At a recent conference, one of my more excitable former customers saw me, and shouted, “I replaced all your old stuff (i.e., SnapVault) with all your new stuff (i.e., EMC backup and recovery technology). It’s so much better!”  To be honest, I wasn’t sure how to react. On the one hand, I was pleased my new company was able to improve his backup environment, but on the other hand, I had written much of the Snap code and none of the EMC code. (I quelled this threat to my ego by focusing on my wisdom in joining a group of people much smarter than me.)

Even without this riveting blog series, this customer had realized the potential of VMware CBT. Originally, he had provisioned VMs on FC-LUNs on FAS systems. He retained one week of nightly snapshots on the primary system and one month of nightly snapshots on a SnapVault secondary. He was thrilled with his backup performance and reliability, and had only three complaints about his environment:

1. “Wasting” nearly 50% of space on the primary flexvol because he would not implement thin provisioning (fractional reservation < 100) and could not support automatically deleting snapshots (they were his backups, after all). He feared that he would run out of capacity and bring down his VMs. Without thin provisioning, his first snapshot reserved half the space on his flexvols to ensure that he could overwrite every block in the LUN without failure.
2. Single VM recovery was complex and inefficient, since he could not put each VM in its own LUN, much less its own flexvol.
3. Granular recovery from the VMs was complex and inefficient.

With the release of VMware’s Changed Block Tracking (CBT), his deployment strategy changed. He opted to use Avamar 6.0 to run raw VM backups with VMware CBT directly to Data Domain systems, where he retains each nightly backup for a month. His entire environment backs up within 15 minutes with stellar success rates. (He told me, “I ain’t gonna jinx myself by saying 100% success rates. You can’t make me do it. So stop trying. Just stop.” Of course, at this point, I hadn’t said a word in 10 minutes.)

He has successfully rolled back a VM to a prior night’s backup, using Avamar’s Changed Block Recovery, restored a VM to a new location and extracted a variety of spreadsheets and documents from the backups. While he has been thrilled with the performance, simplicity, and security of the Avamar solution, its cost-savings capability was clearly the highlight. “It’s saved me more money than anything else I’ve done all year,” he proclaimed.

He was able to:

1. Regain 50% of his FAS capacity by embracing thin provisioning. He still uses snapshots, but since they’re not “official” backups, he’s enabled automatic snapshot deletion to free up space for the VMs, if necessary.
2. Repurpose the SnapVault secondary FAS as a primary storage system.
3. Reduce the size of the backups by 3x (the additional space savings provided by Data Domain’s compression and variable-length deduplication vs. NetApp’s snapshots and deduplication).
4. Reduce network bandwidth consumption by 1.5x – less data means less bandwidth required.

Not only did the Data Domain systems pay for themselves, but the savings funded a separate project to protect his mainframe environment (and this isn’t the first time I’ve seen VM optimizations fund mainframes – who knew?). He now has backup storage for 3 years, primary storage for 2 years, and bandwidth for the next 12 months. This was not a three-year ROI – it was immediate.

In the next post, we’ll examine a customer implementing VMware-versioned replication in a way that blends brute force with subtle elegance (think either of the dancing hippos in Fantasia or Chris Farley as a Chippendale on SNL).

However, despite its importance, CBT hasn’t generated the hype of Kim Kardashian’s wedding. (Google hits: VMware CBT – 2.2 million; Kim Kardashian Wedding – 18.5 million. I see you pretending not to know who Kim Kardashian is. Stop. You’re not fooling anybody. ) VMware CBT promotes versioned replication to become a first-class backup citizen. With VMware identifying the changed data, we can now build solutions that combine the best of storage-based-versioned replication with the best of the backup functionality.

• Minimal load on the client. The server tracks the changed blocks through the day; the client then reads only the changed blocks at backup time.
• Minimal load on the storage array. Reading only the changed data.
• Light network load. Deduplicating changed data further reduces the network load.
• Vendor lock-in. CBT works independent of storage array. Customers can back up VMs on any primary storage to any backup system that supports versioned replicas.
• Full functionality. The backup application or VM-specific protection tool provides the full backup workflows and functionality you have come to expect.
• Granularity of backup management. CBT works on a per-VM basis, so you can apply independent backup schedules, retention periods, security domains, etc. on each VM. More importantly, you can change those settings at any point, without penalty.

In short, you get the performance of versioned replication with the functionality, control and heterogeneity of backup.

What’s required to transform a technology like VMware CBT into a backup solution?
It’s bad enough trying to run recoveries from incremental file-based tape backups. Can you imagine the agony associated with trying to piece together something coherent from random series of blocks on tape? (Actually, I can – I watched the Tron sequel on cable this weekend. Yikes.)

Any viable CBT solution depends on a backup infrastructure built to manage changed-block versioned replicas, specifically:

• Dedupe backup software. Rapidly transform, catalog and index changed data into full backups to optimize for recovery reliability, simplicity and performance. You do not want software with “bolted-on” virtual synthetics to become a bottleneck or point of failure.
• Dedupe storage system. To store tens, hundreds or thousands of full backups, you need a storage system that can efficiently create and store those backups.  As with any backup system, make sure that you can trust it as the storage of last resort and that you can manage each backup version independently.

By combining VMware CBT with the appropriate backup infrastructure, customers have not only been able to reduce their backup pain but also increase the amount and type of production load on their ESX servers. Of course, I don’t want to be a shameless product shill, so I’ll let you figure out what types of backup solutions can properly leverage VMware CBT. (Editor’s note: EMC is not too shameless to point out that Avamar 6.0 uses VMware CBT to deliver this ideal versioned replication solution to either an Avamar Data Store or Data Domain target.)

While both I and the rest of The Backup Window have focused on the value of VMware CBT, wise readers observe that this is merely one instance of this technique. Oracle Block Change Tracking, Microsoft filter drivers and storage-based versioned replicas highlight a broader industry trend. Backup software needs help from the primary data owners to scale. You want to bet on solutions that demonstrate a commitment to integrating with all those data owners. (Because you’d like your backup solution to last longer than a Kardashian marriage.)

In my next post, I will share some customer deployments in which the use of backup solutions with VMware CBT helped remove roadblocks to VM deployments while bringing storage and backup teams together, ultimately showing why making the right backup infrastructure choices does make a difference. So, tune in to the next blog post – same blog site, shorter backup window!