By Joe Smith II, Sr. Systems Admin, Town of Castle Rock
At the Town of Castle Rock my team provides water, fire, police and development services to more than 53,000 residents, a population that has skyrocketed from its total of 15,000 in 1999. Despite the city’s growth, and the digitalization of public services over the last decade, I’m one of just two system administrators to oversee more than 180 virtual servers, 100 applications and 70 TB of usable storage. We wear a lot of hats around here, to say the least.
The majority of our data consists of user shares and profiles, video surveillance, in-car camera footage, VMware datastores, Microsoft Exchange email and SQL databases. Like many of you, we often found ourselves trying to piece together a complete picture of what was going on in our environment from multiple tools and siloed reports of all the separate IT environments, including VMware vSphere and NetApp SANtricity.
Castle Rock rolled out Blue Medora True Visibility Suite for VMware vROps to turn their existing VMware vRealize Suite into a full-stack, centralized IT operations platform. This improved storage management, monitoring and troubleshooting to reduce the time it took to manage these diverse environments and solutions.
Through this collaboration, here’s what they were able to achieve:
Predict and prepare for future IOPs needs
When we added the Blue Medora True Visibility Suite to our VMware vROps platform, we finally got end-to-end visibility, analytics and capacity planning for workloads running on NetApp storage. Our team can now access automated analysis to expose performance and capacity issues before they become a problem. If a department introduces a new workload, we’ll quickly know if we need to change our QoS policy to set up more or fewer IOPs.
With the True Visibility Suite for vROps, Castle Rock’s I can understand the exact cause of an issue and know which team to contact to resolve it. By moving from a reactive to proactive state, I’ve finally got more time to devote elsewhere–and believe me there is plenty more to do.
Tackle bully- victim VM scenarios
Once we knew we needed a centralized management tool with access to health and performance at a more granular level, the VMware vRealize Suite seemed to be an obvious choice. Especially since we could get native tool-like detail from NetApp and vSphere in a single pane of glass, without adding another tool for our team.
One thing we regularly struggled with in particular with was identifying VMs that were bullying others for resources. The bully-victim scenario is a term coined by NetApp when virtual environments share and compete for resources. When a machine becomes stuck on a process where it is running a large report against a database, it hits the hard drives harder than other VMs. Those smaller VMs with smaller workloads become more affected as they wait for other processes to finish. In this scenario, performance becomes unequally distributed. Blue Medora helps on the storage side, allowing us to determine which VM was causing issues and get a better understanding of spreading the load.
At the end of the day, the True Visibility Suite saves our sysadmins 60-90 minutes a day, which allowed our Division of Innovation and Technology to focus on more strategic projects, like our Splunk implementation and vCenter upgrade.
To read more about this collaboration, check out Town of Castle Rock’s case study with Blue Medora here.