During World War I, Will Rogers, noted American humorist, was asked by a young reporter how he would stop the U-boat attacks on ships, to which Rogers responded, “Boil the ocean.” But when pressed for how he would do that, Rogers quickly replied, “I’m just the idea man here. Get someone else to work out the details.”
For the first five years of my career, I subscribed to the Will Rogers’ school of problem solving. If customers would wipe out their legacy infrastructure and immediately deploy our newest products across their environments, their problems would disappear. But that was before I met real customers with real data centers and, well, in the immortal words of Keanu Reeves, “Whoa!”
In this blog series, we’ve talked about the variety of mechanisms to protect VM environments and met two mid-sized customers that have deployed innovative solutions to solve their business problems. In this post, you’re going to meet a large customer that described its biggest challenge with one word – “change.” Its complex legacy infrastructure, organizational silos and users’ unwillingness to embrace new initiatives have made it difficult for the company to successfully change anything.
The company, which prides itself on minimizing exposure to cutting-edge technology (its IT team slowly rolls out new technology, allowing it to demonstrate stability and business value and gather user demand before rolling it out more broadly), has successfully deployed thousands of VMs. The company began by virtualizing its test/dev environment and lower-tier applications. But once manufacturing discovered that it could spin up a new application on a VM within 48 hours (versus a 90-day procurement cycle on physical servers), the IT team had its first advocate and the company’s path to virtualization was paved. The company is now virtualizing its business-critical applications.
Following the IT team’s standard technology deployment model, the backup team at this company has deployed new backup technology on an “as needed” basis. Backup and recovery performance has been the forcing function for change. Initially, it protected VMs just like it did physical servers – with a traditional backup to tape. However, when that approach failed to scale, the IT team evaluated two alternatives: guest-level VM backup via Avamar and traditional backups via VMware Consolidated Backup (VCB).
Ultimately, the company chose Avamar. While the initial Avamar roll-out addressed only the heavily loaded ESX server running the Tier-3 apps, the main driver for the remainder of the company was the network. The cost of local network bandwidth made it too expensive to push thousands of full backups across the LAN. Nearly two years later, it uses Avamar for 95% of its VM backups.
However, the company faces new backup and recovery window challenges. With even more heavily loaded ESX servers, Avamar guest-level backups struggle to complete. While it is backing up only the unique, new data, it’s expensive to search through the VMs to find that data. Therefore, as before, it has selected a new technology solution and is selectively deploying it.
The solution: Avamar-driven VMware Changed Block Tracking (CBT)-based versioned replication on the same heavily loaded ESX servers that first deployed the Avamar clients two years ago. While the company is concerned about the scalability of CBT and VM performance, change – specifically, organizational change – is the real issue. How can it trust a backup solution that depends on technology and administrators who are in different groups?
Fortunately, the company has a safety net. Avamar enables it to implement and manage both solutions – Avamar and CBT – from one interface. (“And if this VMware CBT thing fails, we can quietly switch back to the existing approach and not look stupid.”)
Rapid recovery, not rapid backup, has created a groundswell of demand in the organization. With Avamar’s Changed Block Recovery, on three separate occasions, the IT team has recovered applications within minutes. In each case, the data suffered a logical (read: user-driven) corruption. In the past, the company would either have had to run a full restore to an alternate VM or restore all of the files that comprised the application, but with VMware CBT-based versioned replicas, it simply rolled back the changes to the point of the last backup – moving a fraction of the data. (“It was like having a high-end snapshot storage system for all our apps – even the so-called tier-3 apps.”)
Currently, fewer than 10% of this customer’s VMs are protected by VMware CBT-based versioned replication. And while the backup team continues to roll out the new technology as deliberately as possible – incrementally building a relationship with the VM team – it acknowledges the power of VMware CBT-based versioned replication. The company has solved backup and recovery performance problems that seemed intractable before, and users (“for once”) are leading the drive for change in the company.
We all know that boiling the ocean does not solve problems in real customer environments. For most people, change doesn’t start with a clean sheet of paper, it starts with a mess. Therefore, like this customer, you need to set a clear goal and make a path that moves you in that direction. Along the way, you’ll need to embrace new technology, new techniques, and new organizational structures. You likely won’t overcome your VM backup challenges with one grand proclamation – you need to win hundreds of individual battles. In a more pragmatic moment, Will Rogers also said, “Even if you’re on the right track, you’ll get run over if you just sit there.”
