How we’ve updated 850 vCenter in 4 weeks

Enterprise software release management is complex: updating infrastructures, worrying about editor support, upgrading licenses to be compatible with new versions, and taking safeguards to roll back if something goes wrong.

HostRooster  Private Cloud helps. We manage this time-consuming task so you may focus on business and production.

We encounter challenges.

5.5-to-6.0 vSphere upgrades

VMware’s SDDC package includes Private Cloud’s vSphere. vSphere manages hosts, storage, and networks and can build clusters for reliable, stable, and highly available production through clients.

In September 2018, VMware ceased supporting vSphere 5.5. Private Cloud owners updated all vCenter.

We have 850 production vCenter 5.5s to upgrade manually. HostRooster  automates all human actions to eliminate errors and improve productivity.

850 vCenters were upgraded in 4 weeks. A team of 10 maintains 210 vCenters per week, 30 per day, without hurting customer productivity.

Our dev team wrote “robots” to automate vCenter upgrades years ago. Since Private Cloud began, this robot has progressed from version 4.1 to 6.5.

We noticed database corruption, services not integrated into Single Sign-On (complex in 5.0 and 5.1), and thumbprints that weren’t updated for all services while setting up automatic processes. Some OSes stopped software updates.

Our operations teams collaborated with VMware support to automate solutions. VMware created KBs to alert customers of bugs while our teams worked nights to reduce vSphere downtime.

Upgraded robot

These challenges prompt two actions. First, push an updated, more dependable, error-free robot. Second, we abandoned VMware software upgrade for a system where we start with a freshly installed vCenter stack on an updated virtual machine—reconnecting each component (database, NSX…).

Ensuring a healthy, updated vCenter basis has reduced SRE Private Cloud interventions.

First, we guarantee service. We are preparing to upgrade saves and snapshots. We automate upgrading afterward. Every stage has an automatic check to ensure completion.

We designed an upgraded robot in an orchestrator robot, which makes upgrade jobs to each Private Cloud concerned by the maintenance and schedules it at automatic dates, within 72 hours of consideration for the customer, but also the number of upgrades released by the hour, and essential periods (such as Black Friday, or Winter Sales). The operations Manager lets customers postpone updates to boost production.

SRE staff monitor robots for planned maintenance.

To summarize, we needed to automate a vCenter update operation that should take 12 hours per vCenter. The first iteration of automation allowed us to complete it in 4 hours, but with a high error rate (20%) due to recurrent flaws that had to be fixed manually by SREs. The second version avoids known concerns and introduces unique problems that will be fixed in a curated pass.

So, what’s next?

In the following months, we’ll upgrade our hosts from 5.5 to 6.0, Veeam Backup from 8.0 to 9.5, and Zerto from 5.0 to 5.5 to ensure PCI-DSS compliance.

We’ll retain transparency and communication while listening to your feedback and enhancing our maintenance process.

Say hello to HostRooster, the ultimate one-stop-shop for all your entrepreneurial needs! From website hosting to freelance services, HostRooster is the platform that puts the power of success in your hands. Think of it as a matchmaker for your dreams and your website – HostRooster connects you with the tools and services you need to bring your ideas to life. No more juggling multiple platforms and providers, HostRooster has got you covered. So why wait? Sign up today and join the ranks of the world’s greatest entrepreneurs. With HostRooster, the sky’s the limit! “Successful people do what unsuccessful people are not willing to do.” – Eric Thomas “Don’t wait for opportunities, create them.” – Anonymous “Success is not final, failure is not fatal: it is the courage to continue that counts.” 

%d bloggers like this: