Home » Orchestrator (Page 3)
Category Archives: Orchestrator
From time to time there are questions around using Orchestrator as a batch robot, for example run a set of file copy jobs every night. In general Orchestrator was not designed as a job scheduler, even it if would work in small scale. Challenges you can run into when trying to this are
- There is not a single view to show when you have schedule which runbooks. For example if you want to check all runbooks configured to run every night you have to browse through all your runbooks in Runbook Designer.
- There is not a single view to show all the calendars you have configured and how they are configured.
- There is not a single view to see if all runbooks was successfully last ran, for example if you have a number of runbooks running every night you want to have one view in the morning to verify that everything worked.
- There is no easy way to import a calendar from another tool
- By default a runbook server can run 50 jobs at the same time. You can reconfigure this (http://technet.microsoft.com/en-us/library/hh420378.aspx) but you should consider the resource requirements of the runbooks on a particular server. Lets say you start one runbook that checks for new files in a folder and then for each new file you invoke another runbook. What if there are 1500 new files in that folder? 1500 running runbooks would require a lot of runbook servers to run all that at the same time. If you don’t have another runbook servers a queue would be built up, which could result in long delays and affect other runbooks.
If you still want to schedule a runbook, for example to run every 12 hours, you can use a Monitor Date/Time activity. A disadvantage is that the runbook will be running all the time, monitoring the schedule. As the runbook is running all the time, it will use one of the 50 (by default) slots on your runbook server.
If you want to run your runbook 01.00 on Sundays only you can also use the Monitor Date/Time activity, but you need to combine it with a runbook level schedule. Create a new schedule, check only Sundays, and configure your runbook to use the new schedule. You configure the runbook to use the schedule by right-click on the runbook tab, select properties and then click schedule.
After you have configured a runbook to run for example every Sunday at 1 am you start it. The runbook will get the green “running” icon and in the database it will be marked as pending. Once it gets Sunday at 1 am the runbook will run. After the schedule time window the runbook will stay in running status, until next schedule time window, even if it is not allowed to execute between the schedule time windows. If the runbook tries to run other times a platform event will be generated.
Instead of using Orchestrator to trigger runbooks based on a schedule you can use Windows Schedule Tasks, see this blog post from the Orchestrator Engineering team. I have two related blog posts around this, one about execute runbook later and one about runbook servers in different time zones.
Note that this is provided “AS-IS” with no warranties at all. This is not a production ready management pack or solution, just an idea and an example
Let’s say you have multiple manual activities in a service request. Each of these manual activities need the output of previous manual activities. For example the first manual activity creates a user account for a new hire. Then the second manual activity is adding this user account to a HR system. The first engineer can input a comment, the user account, when the activity is marked as completed in the self-service portal. But the second engineer need to click a lot to see the note from the first engineer.
- Parent work item
- See details
- Related Items
- Contains Activity
- Expand Manual Activity
- Find the correct activity and click it
- Scroll down and find the correct property
As you can understand it is not easy to find the information, special for some non-technical colleges that don’t understand parent work item and activities.
An alternative to this is to use a runbook between your manual activities. The runbook copy the information between your manual activities, engineers can then see all information from previous manual activities on the first page in the self-service portal. It is the comment/note that previous engineers have input when they mark activities as completed that are copied between manual activities.
In this example I have 3 manual activities in my service request template
It is the same runbook (runbook activity) between each manual activity.
- Initialize Data. Input Service Request work item ID and Runbook Activity work item ID
- Read the Runbook Activity
- Read the Service Request
- Get Related Manual Activities for the Service Request
- Read each related manual activity
- Compare Value. Checks if the manual activity sequence number is the one before the runbook activity sequence ID. All activities in the service request has a sequence ID, we need to find the one before the current runbook activity. This Compare Value activity checks if the manual activity sequence ID equals “runbook Activity sequence ID minus 1”
- Reads the manual activity before the runbook activity
- Again, gets all related manual activities
- Again, read each related manual activity
- Again, compare values. Same as previous compare value but this time we look for the manual activity with sequence ID after the runbook activity
- Update the manual activity after the runbook activity with information from the manual activity before the runbook activity
- Return data without any configuration
You can download my example runbook, 20130708_CopyMA_wolf . Note that this is provided “AS-IS” with no warranties at all. This is not a production ready management pack or solution, just an idea and an example
From time to time I get the question if it is possible to pause a SLA in Service Manager. You should write your SLA in a way that you never have to think about pause them. For example if you have a SLA saying you will fix hardware issues in 8 hours, but your hardware vendor has agreement saying they will send you spare parts in 4 days, you need to re-write your SLA. Another example could be a SLA saying an incident will be resolved within 8 hours. If you are dependent on end-users that might be difficult to fulfill, instead write SLA based on first response. Why write a SLA that you can never fulfill?
In this blog post I will show how to modify the SLA endtime with a runbook. This runbook dont pause the SLA, it just adds minutes to the TargetEndDate and the TargetWarningDate for incidents in a “Waiting for end-user” status.
- Monitor Date/Time. The runbook trigger every 5 minutes
- Get Object. Gets all incidents in “Waiting for end-user” status
- Link. If there are more than one incident
- Get Relationship. Gets related Service Level Instance Time Information
- Get Object. Reads the Service Level Instance Time Information instance
- Format Date/Time. Takes the current TargetEndDate as input and adds 5 minutes on Format Result output
- Format Date/Time. Takes the current TargetWarningDate as input and adds 5 minutes on Format Result output
- Update Object. Updates the related Service Level Instance Time Information instance with new TargetEndDate and TargetWarningDate
The result is that every 5 minutes when the incident is in “Waiting for end-user” state the runbook will add 5 minutes to both the SLA Warning time and the SLA End time. You can download my example file, PauseSLA_wolf. Note that this is provided “AS-IS” with no warranties at all. This is not a production ready management pack or solution, just a idea and an example
A friend asked me if it is possible to show the status of a related review activity in the “All Open Service Requests” view? After a while we realized that a requirements was to show the status of the last review activities, so if there were multiple review activities we just wanted to show the last one. The reason why they needed this view was that customer called service desk and asked about service requests. With a view like this the operator could easy see if all the manual review activities was completed, if the service request was still waiting on more approvals or if someone was building/working on the requested service.
We started by extending the Service Request class in Service Manager with two new properties, one to show the last review activity status and one to show when we last updated that status.
Comprehensive description of each activity in the runbook
- Monitor Date/Time. The runbook runs every 30 minutes
- Query Database. Truncates a database that is used to store data temporary. The runbook use a database table to store review activity information temporary.
- Get Object. Gets all open Service Requests, from the extended Service Request class.
- Get Relationship. Get related Review Activities for each open Service Request
- Update Object. If there are no related review activities it will update the “LastStatus” property with “No Review Activities”
- Get Object. If there are related review activities it will get the review activity and write review activitiy and Service Request information to the database
- Junction is used to merge all threads together to one
- Format the current time/date to a format that works with Service Manager
- Query Database. Query the database to get last review activity for each Service Request and status of it
- Update Object. Updates each Service Request that is in the database with review activity status
The result in the Service Manager console can look like this
You can download my example runbook and Service Manager class extension here, 20130616_SRStatus
Note that this is provided “AS-IS” with no warranties at all. This is not a production ready management pack or solution, just a idea and an example.
Orchestrator support a number of data manipulation functions. These functions can be used to manipulate string and convert it into a usable form. You can also perform simple arithmetic operations, such as calculating sums and differences, and performing division and multiplication operations. For a complete list of the functions that you can perform see Technet. One nice thing about these data manipulation functions is that you can combine them.
Example, a simple text file
The result will be
John Connor,Skynet,555-123456John ConnorJohnJoJO555-123456555123456124011
First line is the text line as in the text file. Then it is
String divided on “,” and showing first part
String divided on “,” and showing first part, then dividing that answer on ” ” and showing first part
String divided on “,” and showing first part, then dividing that answer on ” ” and showing first part, then showing two first letters from the left side
String divided on “,” and showing first part, then dividing that answer on ” ” and showing first part, then showing two first letters from the left side, then UPPER means converting that string to uppercase
About the numbers
String divided on “,” and showing third part
String divided on “,” and showing third part, then dividing that answer on “-” and showing first part
String divided on “,” and showing third part, then dividing that answer on “-” and showing second part
String divided on “,” and showing third part, then dividing that answer on “-” and showing first and second parts, then sum the two numbers
With the System Center Integration Pack for System Center 2012 Operations Manager we can integrate with Operations Manager and automate maintenance mode, alert handling, monitor for alerts, and state changes. Unfortunately there are no activities for handling management packs or management pack objects.
Some time ago I posted a blog post around self-service for Operations Manager (http://contoso.se/blog/?p=2764). That idea was built around creating new objects, like a rule, in XML and then import the management pack. While this works fine in some scenarios, there are other scenarios where it would be nice to have a bit more flexibility and reliability.
Russ Slaten (a true Texans with a lot of guns), a PFE colleague and Operations Manager Jedi from the US and I have built a first version of what we call “Operations Manager Admin Integration Pack” for Orchestrator. The purpose of this integration pack is to enable more self-service and automation scenarios in Operations Manager.
Activities in version 1 are
- Create MP. Creates a new management pack
- Create Performance Collection Rule. Creates a rule that collects performance data
- Create Event Alert Rule. Creates a rule that generates an alert based on event viewer event
- Delete Management Pack. Delete a management pack
- Delete Rule. Deletes an rule based on rule ID
- Export Management Pack. Exports a management pack
- Get Management Pack. Lists management packs from the management group
- Get Rule. List rules. With default settings the activity will list all rules, use the Displayname property to filter the search result
- Import Management Pack. Imports a management pack to the management group
These activities doesn’t use global connections, instead you can specify management server in each activity. This integration pack requires Operations Manager PowerShell snap-in on all Runbook servers.
As I think you have already realized, this integration pack enables a lot more self-service scenarios where non-Operations Manager Engineers can order new objects and handle management packs. Please take it for a spin and let us know what you think!
Example configuration for the “Create Rule Collection Performance” activity
|OM Server Name||SCOM-Lit.Litware.com|
|Rule Displayname||Sandbox Test Rule 1|
|Rule Description||Sandbox Rule for testing|
|Counter Name||% Processor Time|
|All Instances||TRUE (used for multi instance counters)|
|Instance Name||If AllInstances is false, then fill this in with either a target variable or fixed value|
|DW Only (Data Warehouse)||TRUE (If you only want to write to the data warehouse)|
|Is Optimized||TRUE (Use if you’re using the optimized data provider)|
|Tolerance||10 (Percentage or absolute. If value changes more than x then collect, otherwise skip)|
|Tolerance Type||Percentage (Percentage or Absolute)|
|Maximum Sample Separation||12 (How many samples can be skipped before forcing collection)|
Example configuration for the “Create Alert Event Rule” activity
|Management Pack ID||Custom.Example.Sandbox1|
|Rule Description||Sandbox Rule for testing|
|Rule Displayname||Sandbox Test Alert Rule 8|
|Event Log Name||Operations Manager|
|Event Source||OpsMgr Scripting Event|
|Alert Name||Sandbox Test Alert Rule 8|
|Alert Priority||High = 2Medium = 1Low = 0|
|Alert Severity||Critical = 2Warning = 1Information = 0|
Example configuration for the Create Management Pack activities
Example configuration for the Delete Management Pack activities
Example configuration for the Delete Rule activities
Example configuration for the Export Management Pack activities
Example configuration for the Get Management Pack activities
Example configuration for the Get Rule activities
Example configuration for the Import Management Pack activities
Big thanks to our colleague Stefan Stranger for Powershell support a very early morning 🙂
Download the IP, OMAdminTasks_20130508-1
Note that this is provided “AS-IS” with no warranties at all. This is not a production ready management pack or solution, just a idea and an example.
I have updated the “runbook validator” again. Added a couple of new checks and also added a table of contents in the beginning of the report. You can download the new version here, 20130419_SCO_CHECK.
Since I started to work with Opalis I have been surprised to see how fast people build their first runbook (policy back in Opalis). When showing Orchestrator for a group of engineers it doesn’t take many minutes before they have a running workflow. A disadvantage when it is too easy, drag-drop-done, is that Orchestrator is some times not used as it should be used. In Orchestrator the challenge is that many engineers quickly drag and drop a couple of activities to the runbook, click Start and then lean backward. Most likely they have a working runbook, but are they really working? Is it built as it should be? Is it tested? I have written a couple of blog posts around best practices and the areas that the “runbook validator” checks
- Fault tolerance in Opalis polices,
- Building a log for your runbooks
- Check in history
- Should this runbook be running
- Job concurrency in Orchestrator
- Consider Invoke Runbook activities when doing a update and dependencies between runbooks
- Activities not using default account and run a runbook with a specific account
- Link delays
The “runbook validator” is a set of runbooks that will check all other runbooks according to a number of best practices. A report file will be generated with the result. All runbooks use a number of variables, like Orchestrator database server name and database name. Don’t forget to update them before you run the check.
If you have any ideas what I should add, please post them as comment to this post or send me an e-mail. Please note that this is provided “as is” with no warranties at all. Also note this is all based on my ideas and is not a “Health check” or Microsoft official guidelines.
Runbooks should be administered using version control. For each version deployed into production, create a package including the runbook and documentation. The documentation should include an explanation of the runbook, including all parameters, global settings and variables. As Orchestrator don’t include version handling it is a good idea to keep earlier runbook versions as export files, making sure you can rollback to earlier versions easy. You should also have a change management process around releasing new runbooks to production to make sure they are tested and signed off correct by the different teams.
As an additional layer of your change process, or maybe in lack of a change process, you can use check in comments in Orchestrator. In the Runbook Designer console you can enable “Prompt for comment on check in”. This is not enabled by default but it is something you should enable at least in your production environment. The result is that each time you click check in in the Runbook Designer console you need to input a comment. It is very easy way to keep track of changes.
You enable check in comments from the Options menu > Configure, in the Runbook Designer console.
All check in comments are written to the Orchestrator database, in the CHECK_IN_HISTORY table. Together with the POLICIES table and the OBJECTS_AUDIT table we can list all comments with account name, runbook name and date. We can also use SQL Server Report Builder to build a report to show this data. The following SQL query will show all check in information.
SELECT CHECK_IN_HISTORY.DateTime AS [Check In Time], POLICIES.Name AS [Runbook Name], CHECK_IN_HISTORY.Comment AS [Check In Comment], SIDS.Account, POLICIES.Version AS [Current Version in database]FROM CHECK_IN_HISTORY INNER JOIN POLICIES ON CHECK_IN_HISTORY.ObjectID = POLICIES.UniqueID INNER JOIN SIDS ON CHECK_IN_HISTORY.CheckInUser = SIDS.SIDWHERE (POLICIES.Deleted = 0) ORDER BY [Check In Time] DESC
“Current Version in database” is a value we get from the POLICIES table. Each time we click check in Orchestrator will add one to that value, so if we check out and check in a runbook 20 times it will show version 20. As we get this value from the current checked in runbook it will show the same “Current Version in database” for all check in for the same runbook, so it is not the version that was checked in at that time.
You can download my example report here, CheckInHistory. Note that this is provided “AS-IS” with no warranties at all
I in this blog post I would like to share an idea around measure how much time you are saving by using Orchestrator. Many organizations spend weeks building cool runbooks, but are they really saving hours, or did the building phase took more hours then they will ever save? My idea is to build a report that show number of hours saved and compare it with number of hours building the runbook. To do a report on this we need some info about each runbook
- How often is the runbook running?
- How long would it take to do the same task manually?
- How much would manually hours cost us?
- How much did it cost to build the runbook?
There is another important aspect on this. Runbooks will always be executed the same way. When someone is doing a task manually there is always a risk that the do something wrong and we need to spend hours/day fixing that. That “risk” is difficult to estimate. If a solution includes multiple runbooks it could be difficult to estimate each runbook, then it might be configure hours on the “parent/root” runbook and not all the child/functions” runbooks.
All this info can be stored in the Service Manager database. I extended the runbook class with extra attributes.
- Runbook owner: Engineer responsible for the runbook
- Runbook Category: a category from a list item, a way to group runbooks
- Manual hours: Number of hours it will take to perform the task manual
- Manual cost: The cost per hour when performing the task manual
- Build cost: The cost to build the runbook
We can query the Orchestrator database to get runbook instance history, from the Microsoft.SystemCenter.Orchestrator.Runtime.RunbookInstances view. We can query the Service Manager database to can get runbook cost and hours, from the MTV_Microsoft$SystemCenter$Orchestrator$RunbookItem table. But out of the box Orchestrator only store 500 entries of log data, we should also try to keep the Orchestrator database small, so it is not a really good data source for this report. Instead I have created a new database where I write a daily summary for each runbook.
I have a runbook that read from the Orchestrator database and writes to the Log database. This runbook gets all runbook objects from Service Manager that have a “Build Cost” specified. Then it query the Orchestrator database for success and failed jobs for previous day, and writes that to the Log database. It also writes a summary of number of hours saved (SM Manual Hours x Number of success jobs) and total amount of money saved (SM Manual Hours x Number of success jobs x manual hours cost).
We can then use SQL Report Builder to build a report to show the data. In my example I have a table and a smiley face icon. The smiley face icon show if I have saved more money then spent on building runbooks. I can also drill down into each runbook category and see data about each runbook. The report includes two parameters, start and end date, to report time frame.
You can download my example files here, OrchestratorScoreCard. It includes the Service Manager extension, the runbook and the SQL report file. Note that this is provided “AS-IS” with no warranties at all. This is not a production ready management pack or solution, just a idea and an example. As always, thanks to Patrik for support and good discussions around System Center.
Last week I was working on private cloud solution that included around 50 runbooks. In the end of the development phase we needed to make sure all runbooks was following best practices, including naming convention and design standard. For example all links between activities that had a name including “IF” should be updated to orange color, and all runbooks named “Link” should be updated to green color. As we had a lot of runbooks I was thinking about a quick way to updated them all, instead of manually updating each link. We also wanted to update some activity names based on the configuration.
The answer was a couple of SQL queries
To update all non-deleted (Deleted = 0) runbooks that have black (default) color and named Link (default) to green link color
Update LINKS SET Color=’65280′ WHERE Color = ‘0’ AND UniqueID IN (select uniqueID from OBJECTS WHERE ObjectType = ‘7A65BD17-9532-4D07-A6DA-E0F89FA0203E’ AND Name = ‘Link’ AND Deleted = ‘0’)
As you can see we set the color to 65280,
- Green is 65280
- Black is 0
- Red is 255
- Orange is 4227327
To updated all non-deleted links that have name that contains “warning” to red link color, run
Update LINKS SET Color=’255′ WHERE Color = ‘0’ AND UniqueID IN (select uniqueID from OBJECTS WHERE ObjectType = ‘7A65BD17-9532-4D07-A6DA-E0F89FA0203E’ AND Name like ‘%Warning%’ AND Deleted = ‘0’)
If you want to update link width for all links named “Link” you can run the following query
Update LINKS SET Width=’3′ WHERE UniqueID IN (select uniqueID from OBJECTS WHERE ObjectType = ‘7A65BD17-9532-4D07-A6DA-E0F89FA0203E’ AND Name = ‘Link’ AND Deleted = ‘0’)
If you want to update all Invoke Runbook activities with a name starting with “Invoke”, and set the name to the runbook it invoke you can run the following query
SET objects.Name = TRIGGER_POLICY.PolicyPath
FROM OBJECTS, TRIGGER_POLICY
WHERE objects.UniqueID = TRIGGER_POLICY.UniqueID
AND objects.name LIKE ‘INVOKE%’ AND Deleted = ‘0’
You don’t need to checkout your runbook to apply the modification, just refresh the console after the update.
Please note that is unsupported as we modify settings in the database direct. This is provided “AS-IS” with no warranties at all. This is not a solution for your production environment, just a idea and an example.