Home » Articles posted by Anders Bengtsson (Page 12)
Author Archives: Anders Bengtsson
Estimate the Orchestrator database size
Audit Collection and Gateway Server
Show number of hours a computer has been sending heartbeats, per dayOrchestrator dashboard in Operations Manager 2012
When you start utilize Orchestrator to integrate between services and execute workflows you soon realize that you need to get an overview of what Orchestrator is actually doing. With the Orchestrator management pack for Operations Manager 2012 you get a good foundation of monitoring the Orchestrator infrastructure, but not that much about what Orchestrator is really doing. If Orchestrator is integrated with Service Manager most runbooks will run as an activity in a work item in Service Manager and then we can use Service Manager Reports to review what has been executed. In this example I will show you how you can build a dashboard in Operations Manager 2012 to show what is going on in Orchestrator.
With Operations Manager we can run a VB script and return the result as performance data. We can then use reports, performance views or dashboards to look at the performance data. In this example I have created a number of rules that runs VB scripts every 15 minute. Each script query the Orchestrator database for some information and sends the result back as performance data to Operations Manager. Some of the rules could be merge together to one SQL query, but as this is only an example and not complete management pack I did not re-wrote that. In Operations Manager I have created a dashboard to show the data.
Each script have an override controlled parameter, Script Arguments, which input Orchestrator database server and Orchestrator database name to the script. My example rules use a run as profile named “Contoso – Orchestrator – DB read account†to configure which account to use when query the Orchestrator database. With default settings, in this example, each query runs every 15 minutes and ask for data for the last hour.
My example dashboard includes five widgets, each widget show a number of performance instances.
- Queue
- Pending Jobs, show number of runbooks with pending status, meaning they are waiting to start
- Top minutes in queue, show number of minutes top 1 job have been in the queue.
- Runbook Results
- Success, show number of runbooks that have ended with success result
- Warning, show number of runbooks that have ended with success result
- Failed, show number of runbooks that have ended with success result
- Runbook Jobs. This widget show number of times each runbook have run with success result. You can easy see which runbook that most often executed. The names you see is the name of the runbook.
- Orchestrator Server Status, show status of my Orchestrator roles. In this sandbox all roles are on the same server, SCO01.
- Orchestrator Alerts show alerts generated by my Orchestrator machine.
You can download my example MP here, NOT SUPPORTED – Contoso.Orchestrator – v2 . Note that this is provided “AS-IS†with no warranties at all. This is not a production ready management pack or solution for your production environment, just a idea and an example.
As always, big thanks to Patrik for support and good discussions around System Center.
Other examples around scripts in rules, generating performance data, can be found here
Job Concurrency in Orchestrator
At each runbook you can configure Job Concurrency, default setting is “1”. The job concurrency setting lets you configure the maximum number of simultaneous jobs, for an individual runbook, so that you can carry out multiple requests for the same runbook at the same time. With the default setting of “1” only one instance of the runbook will run at the same time.
There are some scenario where you need to modify this setting, also a couple of scenario where you should not modify it. For example if you have a runbook working with a counter you should not have multiple runbooks instances running, as they will all affect the same counter. Another example are runbooks that starts with a Monitoring activity, they cannot run multiple instances. Before modifying this setting it is also important to look into what the runbook is doing, if the runbook for example creates a new unique computer name it sometimes will result with multiple computers with the same name.
I did a test with two runbooks, the first runbook trigger the second runbook. 5.1 trigger a new instance every 30 seconds, and invoke the 5.2 runbook. When 5.1 invoke runbook 5.2 it is also pass a timestamp to runbook 5.2. Runbook 5.2 writes the timestamp from runbook 5.1 and a new timestamp to a text file. Runbook 5.2 then sleeps for 3 minutes and ends.
As runbook 5.1 trigger a new instance of runbook 5.2 every 30 seconds, and runbook 5.2 only runs one instances at the same time, there will be a queue. You can check the queue with the following SQL query against your Orchestrator database
SELECT [Microsoft.SystemCenter.Orchestrator.Runtime.Internal].Jobs.Id, [Microsoft.SystemCenter.Orchestrator.Runtime.Internal].Jobs.ParentId,
[Microsoft.SystemCenter.Orchestrator.Runtime.Internal].Jobs.ParentIsWaiting, [Microsoft.SystemCenter.Orchestrator.Runtime.Internal].Jobs.StatusId,
[Microsoft.SystemCenter.Orchestrator.Runtime.Internal].Jobs.CreationTime, POLICIES.Name, DATEDIFF(MINUTE,
[Microsoft.SystemCenter.Orchestrator.Runtime.Internal].Jobs.CreationTime, GETUTCDATE()) AS QueueTime
FROM [Microsoft.SystemCenter.Orchestrator.Runtime.Internal].Jobs INNER JOIN
POLICIES ON [Microsoft.SystemCenter.Orchestrator.Runtime.Internal].Jobs.RunbookId = POLICIES.UniqueID
WHERE ([Microsoft.SystemCenter.Orchestrator.Runtime.Internal].Jobs.StatusId NOT LIKE ‘4’)
AND ([Microsoft.SystemCenter.Orchestrator.Runtime.Internal].Jobs.StatusId NOT LIKE ‘3’)
AND ([Microsoft.SystemCenter.Orchestrator.Runtime.Internal].Jobs.StatusId NOT LIKE ‘2’)
This is a example of the result from the SQL query
- ID is the runbook job instance ID
- ParentID is the job that started this job. As you can see in the image all jobs have a parentID that starts with 45723… that is the 5.1 runbook, the runbook with the Invoke Runbook activity
- ParentIsWaiting is set to “1” if the runbook that trigger this job is waiting until this job is complete. When you configure a Invoke Runbook activity there is “Wait to completion” option, if you check it the parent runbook will wait until the runbook it trigger is completed
- StatusId is current stats of the job.
- 0 = Pending
- 1 = Running
- 2 = Failed
- 3 = Canceled
- 4 = Completed
- CreationTime is when the job was created in the database
- Name is the name of the runbook running
- QueueTime is number of minutes between “NOW” and CreationTime, number of minutes it has been in pending state
Next image is an example of the log file that the 5.2 runbook writes to. As you can see the delay between the runbook was triggered and the time it executed is growing fast when only one runbook instance is allowed to run simultaneous.
Pass information between runbook activity
In Service Manager 2012 we can use runbooks from Orchestrator as activity in for example a service request template. This brings a lot of cool possibilities for automation in your data center. I have blogged a number of examples around this that you can find here on the blog. But in most of the examples I use only one runbook activity, and if I need to use multiple runbooks they are all started from one “master” runbook, not as individual activities in the service request template.  I came across a scenario where we needed to use multiple runbook activities in the same service request template and we needed to pass information between the runbook activities. One benefit with splitting multiple runbooks into multiple runbook activities is that it is easier to track the result in Service Manager, when each step in a complex workflow is its own runbook activity. This also gives the benefit that a Service Manager operator can re-start/re-run a step if needed.
In this blog I will show how you can use two runbook activities in a service request and pass information from runbook activity one to runbook activity two. We will also see how the result from both runbook activities can be published in the service request and self-service portal. In my demo example the first runbook generates an “account name” and the second runbook generates a “phone number”.
Each instance of a runbook activity have a number of “generic/blank” fields that we will use to store the result, they are for example named TEXT1 to TEXT10. Why not store the result in the Service Request? Even if the service request have some fields I could use I don’t want to do that as it would be difficult to remember what result I store in which property. An alternative would be to extend the service request class, but that would result that all kind of service requests would get those new properties, as they use the same service request class.
The first runbook, in this example named 12.1, has one input parameter which is the runbook activity ID. The runbook then generates an account (random 4 characters) and returns account name back to Service Manager.
In Service Manager I have configured that OUT from this runbook should be written to the TEXT1 field of the runbook activity instance. This is done in the service request template, on the runbook activity.
The second runbook, in this example named 12.2, have one input parameter which is the runbook activity ID. The runbook then generates a phone number (random numbers). Then it gets a bit more complicated 🙂
- Get Related Service Request. This activity use the input parameter, the runbook activity instance ID, to get the related Service Request.
- Get Service Request. This activity gets the Service Request that the previous activity found
- Get Related Runbook Activities. This activity finds all related runbook activities to the service request. We need to do this as we need to find the first runbook activity and read the TEXT1 field.
- Get Runbook Activities. The “Get Related Runbook Activities” will return all related runbook activities, in this example there are two runbook activities in the service request. The link after the “Get Runbook Activities” activity is configured with a filter to only continue with the runbook activity that that a specific title. This title is the title you configure in the service request template.
- Update Imp Result. This activity update the service request with both a implementation result and an new description. Implementation result and description are both visible in the Service Manager console. The description field is also visible in the self-service portal.
SQL query to show last result for each runbook
The following SQL query will show you last result for each runbook. The query will also show you when it started and ended, and at which runbook server.
SELECT PI.Status, POLICIES.Name, PI.TimeEnded, PI.TimeStarted, PI.Computer
FROM POLICIES
INNER JOIN
(SELECT PI1.PolicyID, PI1.TimeStarted, PI1.TimeEnded, PI1.Status,
ACTIONSERVERS.Computer
FROM PolicyInstances AS PI1
INNER JOIN ACTIONSERVERS ON PI1.ActionServer = ACTIONSERVERS.UniqueID
WHERE PI1.TimeEnded = (SELECT MAX(PI2.TimeEnded) FROM PolicyInstances AS PI2 WHERE PI2.PolicyID = PI1.PolicyID)
) AS PI ON PI.PolicyID = Policies.UniqueID
WHERE (POLICIES.Deleted = 0) AND (POLICIES.CheckOutUser IS NULL)
a example of the result
Different Service Level Objectives (SLO) for different departments
The following post show how to configure different Service Level Objectives (SLO) for the DEV and HR departments. The scenario is that the two departments have different SLO for the time between incident created and first response. That could also be two different customers or companies.
- Queues are used to group similar work items that meet specified criteria such as all incidents that are classified by analysts as E-mail incidents. Queues membership rules are dynamic and are periodically recalculated to ensure that the queue membership list is current. For this example we need two queues, one for incidents where the affected user is in the HR department and one for incidents where the affected user is in the DEV department. Navigate to Library/Queues to create these
- Queue name: Contoso – Queue – User dep eq HR
- Work item type: Incident (typical)
- Criteria: Affected User [User] Department contains HR
- We need to configure a calender to control when the SLO should be active. In the Service Manager console navigate to Administration/Service Level Management/Calendar. In my example I have configured the calender to be always on, all days are checked and start and end time is 12:00:01 AM to 11:59:59 PM.
- We need a Metric to configure what time we want to measure, in this scenario the time between the incident was creates to the first response. In the Service Manager console navigate to Administration/Service Level Management/Metric
- Last thing to configure is Service Level Objectives, in the Service Manager console navigate to Administration/Service Level Management/Service Level Objectives. Configure the Service Level Objective to use the calendar and metric created before. As we will use different target and warning thresholds between the two departments we need to create two Service Level Objectives, one for the HR department and one for the DEV department.
When a new incident is created with a affected user from either the DEV or the HR department it will be included in the HR queue or the DEV queue. If you want to verify that you can use the following script. Thanks to Anton Gritsenko who wrote this script. When the incident is in the queue the SLO will be applied too. It can take a couple of minutes, but you should then see the SLO on the Service Level tab of a incident.
Reporting have changed a lot with Service Manager 2012. We now have cubes to analyze the data and we use Excel to easy drag and drop fields to build table.  You can use the Service Manager WorkItems Cube to analyze this data in Excel. If you for example want to see SLO information related to each incident, grouped by department, configure Excel like the image below. Note that it can take some time, hours, before all data has been transferred to the data warehouse and that the Cube has been updated. As you can see there is one row above DEV, that one list all incidents where the affected user don’t have a department configured.
Deploy a new service instance with Powershell in VMM 2012
I read on Technet that Microsoft recommend to use service templates in VMM 2012 even for a single server. Instead of using a VM template we should use a service template to deploy a new virtual machine. In Orchestrator there is an activity in the Virtual Machine Manager (VMM) integration pack that can create a new virtual machine from a template. But there is no activity to create a new instance from a service template. So I copy/paste “wrote” a script that deploys a new instance of a service, based on a service template. This is a very basic and simple example script that you can use as a foundation. The script will ask for five input parameters
- CloudName = Target cloud in VMM for the new service
- SvcName = Name of the new service. As I only deploy a single server I use the computer name as service name too
- ComputerANDvmName = The name that will be used both for the virtual machine and the computer name
- SvcTemplateName = The service template to use
- Description = A description that will be added to the new service instance
Param(
[parameter(Mandatory=$true)]
$CloudName,
[parameter(Mandatory=$true)]
$SvcName,
[parameter(Mandatory=$true)]
$ComputerANDvmName,
[parameter(Mandatory=$true)]
$SvcTemplateName,
[parameter(Mandatory=$true)]
$Description)
Import-Module ‘C:\Program Files\Microsoft System Center 2012\Virtual Machine Manager\bin\psModules\virtualmachinemanager\virtualmachinemanager.psd1’$cloud = Get-SCCloud -Name $CloudName
$SvcTemplate = get-scServicetemplate -Name $SvcTemplateName
$SvcConfig = New-SCServiceConfiguration -ServiceTemplate $SvcTemplate -Name $SvcName -Cloud $cloud -Description $Description
$WinSrvtierConfig = Get-SCComputerTierConfiguration -ServiceConfiguration $SvcConfig | where { $_.name -like “Windows*” }
$vmConfig = Get-SCVMConfiguration -ComputerTierConfiguration $WinSrvtierConfig
Set-SCVMConfiguration -VMConfiguration $VMConfig -name $ComputerANDvmName -computername $ComputerANDvmName
Update-SCserviceConfiguration -ServiceConfiguration $Svcconfig
$newSvcInstance = New-SCService -ServiceConfiguration $Svcconfig
The script first create and configure a service deployment configuration. The service deployment configuration is an object that is stored in the VMM Library that describes the new service instance, but it is not running. The last two lines in the script will pick up that service deployment configuration and deploy it to. All settings of the new service instance is stored in the service deployment configuration. In my service template, named “Contoso Small”, I have a tier named “Windows Server 2008R2 Enterprise – Machine Tier 1” that is why the script search for a tier with a name like “Windows*”.
When we have a PowerShell script we can easy use it from a runbook in Orchestrator to deploy new instances of services.
Note that the script is provided “AS-IS†with no warranties.
Handle disk IOPS when deploying/deleting or update a virtual machine
A couple of weeks ago I needed to build a solution where disk IOPS are allocated when a new virtual machine is deployed. The scenario was that virtualization administrators wanted to make sure no disks was over allocated looking at disk performance. When Virtual Machine Manager deploy a new virtual machine it doesn’t look at what kind of virtual machines that are already deployed at the disk, only at the current disk load. In worst case this could result in very poor disk performance if all virtual machines start working heavily with the disk at the same time. In the virtual environment we had three virtual machine templates, small, medium and large. There was a estimate of disk IOPS required by each template. Virtualization administrators needed to make sure that not more than one virtual machine based on the large template was deployed to a disk and that new virtual machines was deployed to the disk with most free IOPS. Also make sure that no disk was over allocated when looking at disk IOPS.
In Service Manager disks will show up as CI if you have a connect to Operations Manager. But default properties of the disk class was not enough, we needed to store information about disk IOPS and didn’t want to affect anything else, so we created a new configure item class, that would only be used for this purpose. The new class was created with Service Manager authoring tool, example here.
We could now create CI objects for each disk that the virtual environment was using. Disk CI include max, allocated and free IOPS. We then published an offering in the Service Manager self-service to request new virtual machines. We used a runbook to deploy the new virtual machine. As you can see in the runbook below we have a main runbook building the virtual machines, then we invoke another runbook to figure out which disk to use. The runbook also write a register value to the new machine with information of the template used. Operations Manager will later discover this information and start monitor the new virtual machine based on the virtual machine template. Virtual machines are monitored with different thresholds and in different ways based on which template that was used during deployment.
The “Get disk” figure out which disk to use. It use a temp database to store disk information in, so the first step in the runbook is to clean this database to make sure it runs with fresh data. It then adds data to the temp database about each virtual machine template, and query Service Manager to find all disks that have enough free IOPS based on the virtual machine template. After the junction activity the runbook runs a SQL query to find out which disk to use, if we are about to deploy a new virtual machine based on the large template we also check if there is a free disk with enough IOPS that don’t already host a large virtual machine.   In the end the runbook updates the disk CI in Service Manager, allocate the disk IOPS,then returns which disk to use to the “main” runbook that will continue to build the virtual machine.
We also created a runbook to handle deletion of a virtual machine, including giving back allocated disk IOPS to the correct disk CI. We created one portal offering also around update a virtual machine, for example if a machine is running based on the medium template but need more RAM we can update it to a large template. Portal offerings for deletes and updates virtual machines use a query result to show only virtual machines there the portal user are owner, to make sure no one deletes or updates the wrong virtual machine.
example of the runbook that update of virtual machines
example of the runbook that delete virtual machines
Find the Orchestrator Administrators group
There are two security Groups after a default installation of Orchestrator, Orchestrator User Group (OrchestratorUserGroup) and Orchestrator System Group (OrchestratorSystemGroup). The OrchestratorUserGroup is the default administrators group in a Orchestrator environment. After installation, members of this group can run the Runbook Designer console and Deployment Manager. With default settings, members of this group have the authority to perform tasks like create new runbook, deploy new runbook servers, deploy new designer consoles, administrate integration packs and global settings. By default this is a local security group in your management server. From time to time I see the questions about how to find this group if you don’t know which one you specified during installation. You can do that in two steps, first get the SID from the Orchestrator database and then run a WMI Query.
Run the following SQL Query against your Orchestrator database
SELECT Name FROM [Orchestrator].[Microsoft.SystemCenter.Orchestrator.Internal].Parties WHERE ID = ‘1’
You will then get a result like
Copy the SID and run the following VB script
You will then get a pop-up showing you the group. With a default installation, using a local security group, on a server named SCORCH the pop-up will look like
Get-FileAttachments – Download attached files from Service Manager
In the Integration Pack for Service Manager 2012 there is a activity to upload attachment, you can also look at attachments with the Get Object activity. But what you are looking is just the object, not the context of the file. For example you can see that incident IR123 has a related file (System.FileAttachment), you can see some properties of the file, like name and size, but you cant read it. This has been a challenge in a number of scenario. A couple of days ago I asked Patrik if we couldn’t do something about it and today he uploaded the first public version of a Powershell script. This Powershell script, Get-FileAttachments, will dump all file attachments to a folder. It works with both work items like incidents and configuration items like Windows computers.
One scenario that I read about in the forum a couple of days ago is the fact that attachments are not moved to the data warehouse in Service Manager. Instead you need to archive them before the work item or config item is moved to the data warehouse. There is no built in feature for this. But with a simple runbook in Orchestrator and the get-fileattachments script it is solved 🙂
This example monitor Service Manager for incidents that is updated with a new status that is equal to closed. The runbook then checks if there are any related file attachments, if there are, the script is triggered. The script that I run as a command in this example gets the SC Object GUID from the Monitor Closed Incidents activity as input. C:\SCSM_Archive is my archive folder where all attachments are stored. The scripts creates a folder named as the incident and stores all files in the folder.
Some time ago I wrote a blog post about converting incidents to service requests. That is a scenario where you also would like to include this get-fileattachments script, to make sure all attached files are moved to the other work item too. You can build activity and a integration pack on the script too, see this blog post for a example
You can download the script here.
Consider Invoke Runbook activities when doing a runbook update
The recommended way to build runbooks is to build them in a isolated environment, then move them to a test environment and after that move them into production. In the production environment no modifications should be done, only import new or updated runbooks. When working with the Invoke Runbook activity there is a setting named “Invoke by path”. By default you invoke a runbook by GUID, meaning that you can move around a runbook between folders but it will still be triggered as it has the same GUID.
When the “Invoke by Path” setting is enable you will always invoke the runbook with the same folder path and name as configured, in this example /Prod/C. Even if update or re-create runbook C in the Prod folder it will be invoked. The challenge is when you invoke, with the invoke by name enabled, another runbook that requires parameters. In the image above we invoke a runbook named C with parameter named msg, the value is “Hello World!”. Looking at this from the database it looks like this
As you can see it will pass “Hello World!” to a parameter with GUID “87BFA0D3-741E-4E9A-9F84-7AD0AEC0FB75”. If we now import a new runbook named C and replace/overwrite the current version of runbook C, the first runbook will still try to trigger runbook C (the path is the same) and still a parameter with GUID “87BFA0D3-741E-4E9A-9F84-7AD0AEC0FB75”. But if we look in the Runbook Designer console, in runbook A we see that the value field is empty on the Invoke Runbook activity
This happened when we imported a new version of the C runbook. Why? It is because when we imported the new version of runbook C it has a new parameter. Even if the parameter has the same name and the runbook looks the same, same activities, the parameter GUID is not the same. The GUID is generated when the activity is created and it follows the activity. If you for example build a runbook in a test environment and then export it to production, it is the same GUID in both, the one from the test environment where you created it the first time.
With the following SQL query you can see all Invoke Runbook activities that have a NULL/blank  the value fields
SELECT OBJECTS.Name AS [Activity Name], TRIGGER_POLICY.TriggerByPolicyPath, TRIGGER_POLICY.PolicyPath AS Runbook, TRIGGER_POLICY_PARAMETERS.Value,
TRIGGER_POLICY_PARAMETERS.ParameterName
FROM OBJECTS INNER JOIN
TRIGGER_POLICY ON OBJECTS.UniqueID = TRIGGER_POLICY.UniqueID INNER JOIN
TRIGGER_POLICY_PARAMETERS ON OBJECTS.UniqueID = TRIGGER_POLICY_PARAMETERS.ParentID
WHERE (OBJECTS.ObjectType = ‘9C1BF9B4-515A-4FD2-A753-87D235D8BA1F’) AND (OBJECTS.Deleted = 0) AND (TRIGGER_POLICY_PARAMETERS.Value IS NULL)
To handle this, avoiding to manual update all Invoke Runbook activities when you implement a new version of the runbook they invoke, you need to work with updates. Always update the runbook, don’t create a brand new runbook. You can import your archive file, or even export from production, edit the runbook in another environment and then re-import it again. That will keep the same GUID for everything you don’t re-create. If you re-create any of the initialize start parameters you need to update runbooks that pass parameters to it.
Recent Comments