Handle disk IOPS when deploying/deleting or update a virtual machine

A couple of weeks ago I needed to build a solution where disk IOPS are allocated when a new virtual machine is deployed. The scenario was that virtualization administrators wanted to make sure no disks was over allocated looking at disk performance. When Virtual Machine Manager deploy a new virtual machine it doesn’t look at what kind of virtual machines that are already deployed at the disk, only at the current disk load. In worst case this could result in very poor disk performance if all virtual machines start working heavily with the disk at the same time. In the virtual environment we had three virtual machine templates, small, medium and large. There was a estimate of disk IOPS required by each template. Virtualization administrators needed to make sure that not more than one virtual machine based on the large template was deployed to a disk and that new virtual machines was deployed to the disk with most free IOPS. Also make sure that no disk was over allocated when looking at disk IOPS.

In Service Manager disks will show up as CI if you have a connect to Operations Manager. But default properties of the disk class was not enough, we needed to store information about disk IOPS and didn’t want to affect anything else, so we created a new configure item class, that would only be used for this purpose. The new class was created with Service Manager authoring tool, example here.

We could now create CI objects for each disk that the virtual environment was using. Disk CI include max, allocated and free IOPS. We then published an offering in the Service Manager self-service to request new virtual machines. We used a runbook to deploy the new virtual machine. As you can see in the runbook below we have a main runbook building the virtual machines, then we invoke another runbook to figure out which disk to use. The runbook also write a register value to the new machine with information of the template used. Operations Manager will later discover this information and start monitor the new virtual machine based on the virtual machine template. Virtual machines are monitored with different thresholds and in different ways based on which template that was used during deployment.

The “Get disk” figure out which disk to use. It use a temp database to store disk information in, so the first step in the runbook is to clean this database to make sure it runs with fresh data. It then adds data to the temp database about each virtual machine template, and query Service Manager to find all disks that have enough free IOPS based on the virtual machine template. After the junction activity the runbook runs a SQL query to find out which disk to use, if we are about to deploy a new virtual machine based on the large template we also check if there is a free disk with enough IOPS that don’t already host a large virtual machine.   In the end the runbook updates the disk CI in Service Manager, allocate the disk IOPS,then returns which disk to use to the “main” runbook that will continue to build the virtual machine.

We also created a runbook to handle deletion of a virtual machine, including giving back allocated disk IOPS to the correct disk CI. We created one portal offering also around update a virtual machine, for example if a machine is running based on the medium template but need more RAM we can update it to a large template. Portal offerings for deletes and updates virtual machines use a query result to show only virtual machines there the portal user are owner, to make sure no one deletes or updates the wrong virtual machine.

example of the runbook that update of virtual machines

example of the runbook that delete virtual machines

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *