Home » Articles posted by Anders Bengtsson (Page 11)
Author Archives: Anders Bengtsson
Apply runbook best practices (quick and dirty)
Last week I was working on private cloud solution that included around 50 runbooks. In the end of the development phase we needed to make sure all runbooks was following best practices, including naming convention and design standard. For example all links between activities that had a name including “IF” should be updated to orange color, and all runbooks named “Link” should be updated to green color. As we had a lot of runbooks I was thinking about a quick way to updated them all, instead of manually updating each link. We also wanted to update some activity names based on the configuration.
The answer was a couple of SQL queries
To update all non-deleted (Deleted = 0) runbooks that have black (default) color and named Link (default) to green link color
Update LINKS SET Color=’65280′ WHERE Color = ‘0’ AND UniqueID IN (select uniqueID from OBJECTS WHERE ObjectType = ‘7A65BD17-9532-4D07-A6DA-E0F89FA0203E’ AND Name = ‘Link’ AND Deleted = ‘0’)
As you can see we set the color to 65280,
- Green is 65280
- Black is 0
- Red is 255
- Orange is 4227327
To updated all non-deleted links that have name that contains “warning” to red link color, run
Update LINKS SET Color=’255′ WHERE Color = ‘0’ AND UniqueID IN (select uniqueID from OBJECTS WHERE ObjectType = ‘7A65BD17-9532-4D07-A6DA-E0F89FA0203E’ AND Name like ‘%Warning%’ AND Deleted = ‘0’)
If you want to update link width for all links named “Link” you can run the following query
Update LINKS SET Width=’3′ WHERE UniqueID IN (select uniqueID from OBJECTS WHERE ObjectType = ‘7A65BD17-9532-4D07-A6DA-E0F89FA0203E’ AND Name = ‘Link’ AND Deleted = ‘0’)
If you want to update all Invoke Runbook activities with a name starting with “Invoke”, and set the name to the runbook it invoke you can run the following query
UPDATE OBJECTS
SET objects.Name = TRIGGER_POLICY.PolicyPath
FROM OBJECTS, TRIGGER_POLICY
WHERE objects.UniqueID = TRIGGER_POLICY.UniqueID
AND objects.name LIKE ‘INVOKE%’ AND Deleted = ‘0’
GO
You don’t need to checkout your runbook to  apply the modification, just refresh the console after the update.
Please note that is unsupported as we modify settings in the database direct. This is provided “AS-IS†with no warranties at all. This is not a solution for your production environment, just a idea and an example.
SD-B317 Best Practices For Runbook Authoring and Managing Orchestrator @ MMS
At MMS 2013 I will present a session together with Pete Zerger
This session will cover how to author and organize runbooks created in Orchestrator so they can be easily reused, tested, and debugged. It will also cover some of the challenges customers face when managing the environment and best practices to follow.
During the session we will walk through the best practices and lessons learned around Orchestrator runbook design . We will highlight what works and pitfalls to avoid. Join this session to get ideas and tips how to get most out of Orchestrator and success with the automation in your datacenter.
See you there 🙂
Nordic Infrastructure Conference (NIC)
NIC is a premier event for all IT-professionals, offering broad technical education on Microsoft and 3. party products, tools and services. Our two day event will focus on deep-dives and practical knowledge on the most used products. More info about NIC here.
A couple of weeks ago I presented two sessions at NIC in Oslo, both sessions can be seen at the following links
- Best practices and lessons-learned around runbook design, link
- Busting the time wasters in your IT environment, link
Should this runbook be running?
When browsing around in the runbook designer console sometimes the “should this runbook be running” question pops up in the head. The runbook might start with a “monitor” activity, like monitor alert, so most likely it should be running all the time, monitoring a data source. When different teams and different engineers in each team works with Orchestrator is sometimes get difficult to keep track of which runbooks that should be running and which ones that should not. One way to handle this is by including “should be running” in the naming convention. “Should be running” might be a bit to much, so instead use for example “RUN” or “ON” or “MON” (as monitor).
If you name your runbooks including a info if they should be running or not it is easy to monitor this. Lets say we include “ON” in runbook names for all runbooks that should always be running. We can then use a SQL query to verify the status of these runbooks.
SELECT UniqueID, Name
FROM POLICIES
WHERE (Deleted = ‘0’) AND (Name LIKE ‘%-ON-%’)
AND UniqueID NOT IN (SELECT RunbookId FROM [Microsoft.SystemCenter.Orchestrator.Runtime].Jobs WHERE Status = ‘RUNNING’)
The query will show you all runbooks with a name including -ON- which don’t have a running job (status equals 1). If all your monitor runbooks are running, then there should be no result. You can then use a rule or monitor in Operations Manager to monitor this, like I do in this blogpost and you can add this to your Orchestrator dashboard.
Self-service data recovery with Data Protection Manager, Service Manager and Orchestrator
In my sandbox I test a lot of solutions, management packs, integration packs and ideas. They don’t always work out the way expected 🙂 The result is that I often need to restore a database from backup. I use Data Protection Manager to protect my databases, Service Manager to order the restore and Orchestrator as the “doer”. In this blog post I want to share a example how to make restore of database a bit easier.
In the Service Manager self-service portal I have a request offering named Restore Database.
- Restore to Original Location. If this checkbox is enabled the database will be restored to original instance. In most cases I restore to a network folder. Network Folder is used in this integration pack in the same way that it is used in the Data Protection Manager user interface. Choosing Network Folder recovers to a local path on a server that have the DPM agent installed. I have configured my runbook (also included in this blog post) to always recover to C:\RESTORE on the target machine
- Target server. If I select to restore to a network folder, default, I input a server name. For example if I want to restore the Orchestrator database to my Orchestrator database server I input SCO12SP1-SQL01 in the Target Server text box. The database backup will then be restored to C:\RESTORE in the SCO12SP1-SQL01 server.
- Recovery Point to Restore. In this query based list I can select which DPM recovery point to restore. I have a runbook (also included in this blog post) that create CI of each recovery point.
Service Manager invokes the “1.2 Restore” runbook in Orchestrator. The runbook is divided into two tracks depending if restoring to a network folder or to original location. Both Data Source ID and Recovery Source ID, used to recover the SQL database, is stored on the Backup CI in Service Manager so we dont need to get them from DPM within the runbook. In general the runbook restore the database and updates the service request.
Runbook “1.1 Create Backup CIs” is the second runbook in this example. It is used to create backup CI objects in Service Manager. The backup class is a custom class that I have created with the Service Manager authoring tool. The runbook runs every hour and creates new/updates/deletes CIs of the backup class.
- Every hour.  Invokes the runbook every hour
- Get Existing Backup. Gets all objects of the backup class in Service Manager. If there are any objects the “Set Verified to FALSE” activity change the verified property of all the backup CIs to FALSE.
- Junction. Used to merge possible multiple threads to one
- Get Data Source for System Center DBs. In my DPM server I have a protection group named “System Center Databases”, this activity gets all data sources for that protection group
- Get All Recovery Point. This activity gets all recovery points for the data sources returned by the “Get Data Sources for System Center DBs” activity
- Check if Backup Exist. This activity checks in Service Manager if there is a backup CI, with Active status, for the current BackupID. BackupID is a property of the backup class that I use to give all recovery points a unique ID, the backup ID contains of <Protection Group Name>.<Production Server Name>.<Recovery Time Point in Time>.<Data source Name>
- If a backup CI object already exist the runbook change the verified property of the backup CI object to TRUE
- If no backup CI object exist a new backup CI is created and a relationship to the server is created
- Junction. Used to merge possible multiple threads to one
- Get Non Verified Backups. This activity gets all backup CIs that has not been verified (verified property equals FALSE) and deletes them with the “Delete Backup” activity
Backup CIs listed in the Service Manager Console
Backup CI
Relationship between windows server and backup CI
When the runbook is done the service request is updated with some information, which can be read from the Service Manager self-service portal. As you can see the database has been restored to C:\RESTORE on the SCO12SP1SQL-01 server. A very Quick and easy way to roll back a database.
You can download my example files here, 20130103_DPM. Note that this is provided “AS-IS†with no warranties at all. This is not a production ready management pack or solution for your production environment, just a idea and an example.
Runbook servers in different time zones
In this blog post I will try to show how Orchestrator works with schedules and different time zones. In my sandbox I have one runbook server (SCO12SP1-01) in the USA and one runbook server (SCO12SP1-02) in Sweden. I want to configure my runbook to only run between 22 and 23 Monday-Friday. To do this I first create a schedule, in Runbook Designer on my Swedish runbook server where I also have the Runbook Designer console installed. Â Name of the schedule is “22-23 SWE” as I want the runbook to run between 22 and 23 and I created it on my Swedish machine.
Next I have create a small runbook that generates a platform event saying which runbook server it ran on and the local time of that runbook server. I have configured the runbook with the new schedule too.
If we run the following SQL query in the Orchestrator database we can take a look at all schedules
SELECT OBJECTS.Name, SCHEDULES.DaysOfWeek, SCHEDULES.DaysOfMonth, SCHEDULES.Monday, SCHEDULES.Tuesday, SCHEDULES.Wednesday,
SCHEDULES.Thursday, SCHEDULES.Friday, SCHEDULES.Saturday, SCHEDULES.Sunday, SCHEDULES.First, SCHEDULES.Second, SCHEDULES.Third,
SCHEDULES.Fourth, SCHEDULES.Last, SCHEDULES.Days, SCHEDULES.Hours, SCHEDULES.Exceptions
FROM SCHEDULES INNER JOIN
OBJECTS ON SCHEDULES.UniqueID = OBJECTS.UniqueID
WHERE (OBJECTS.Deleted = 0)
For the “22-23 SWE” schedule we can see in the database  the Hours column looks like “000000000040000040000040000040000040000000”. Hours are configured for any day that the schedule permit or deny. The Hours field contains in 7 parts, one for each day of a week. Each part contains of six characters, where the day is divided into 8 hours pieces, so 2 characters represent 8 hours of the day.
- The week = 000000000040000040000040000040000040000000
- Sunday and Saturday =Â 000000 = No hours permitted
- Monday-Friday =Â 000040
- 00 = hours 00-08 (no hours permitted)
- 00 = hours  09-16 (no hours permitted)
- 40 = hours 17-23
- 40 is hex, equals to 01000000 binary = penultimate hour in that time frame is permitted (if it was 23-00 it would look like 10000000)
If I start my runbook 11.32 PM Swedish time it will only run on the Swedish runbook (SCO12SP1-02) server as on the runbook server in the USA local time right now is 1.32 PM.
If I change the schedule to allow only 1-2 PM it will only run on the runbook server in the US.
That means that if you build you schedule for 1-2 AM in Sweden and trigger it on a runbook server in the USA, 1-2 AM is still 1-2 AM local time, no translation between time zones. The evaluation of the schedule is done local on the runbook server. If a runbook server is not allowed to run a runbook due to the schedule the runbook server will still download the runbook and then check the schedule locally.
If a runbook is scheduled to start during an hour that is skipped, due to daylight saving, when the system clock is adjusted forward by one hour, that starting time is skipped, and the runbook starts at the next scheduled time. If a runbook is scheduled to start during an hour that occurs twice because the system clock is adjusted backwards by one hour, the runbook launches twice.
Manage new monitoring by self-service (light MP authoring with a pinch of Orchestrator magic)
A common scenario I often see is that everyone in the IT organisation knows that Operations Manager can monitor everything and fulfill all requirements, but it is to complicated for different expert/administration teams to do anything in Operations Manager. For example if the Exchange team wants to monitor a event they need to ask the Operations Manager team to create the rule. Of course the Operations Manager team don’t have time to do that the same day, instead there is a delay and once the rule is created the Exchange team have already solve it in some other way. The result is that Operations Manager is not used as much as it should be used.
In previous posts I showed how to handle overwrites and groups in Operations Manager with self-service in Service Manager and a bit Orchestrator. In this post I want to share a idea how to handle new monitoring, for example creating new rules from the Service Manager self-service portal.
My example starts with a service request in the Service Manager portal. A engineer goes in a request a new Windows event rule in Operations Manager. The engineer fills in event ID, rule name, which Windows Log, service/system and also alert name. A service request is created and in the service request there is a runbook activity.
The runbook activity trigger a “master runbook”. The runbook first invoke a runbook that will find a suitable management pack, then invoke a runbook to create a new monitor or rule (I have only included the rule part so far), then it invokes a runbook to import the management pack into Operations Manager and finally it invokes a runbook to update the service request.
The 60.3 Find MP runbook will find and return the management pack to use. It use the service parameter from the service request to select management pack. All management packs that are in products are stored in a “production” folder. The “Check if MP exists” activity checks if there is a management pack in that folder for the selected service. If there is, it makes a copy of it to a “archive” folder and returns the file path. If there is not a management pack it will write a new management pack file and return the path of that file. The “Write new MP file” activity will write all the needed XML code to a new XML file, it includes a number of input parameters.
The 60.2 Create Rule runbook will first translate targeting between the service parameter and the target parameter needed in the management pack. In my example I only have one target there, that is Windows 2008 Computer. The runbook then finds the <Rules>,<DisplayStrings> and the <StringResources> sections of the management pack and then adds the new rule. We use “Find” to know where in the management pack, on which line, to insert the new configuration. Each “Add Rule -” activity use input parameters when writing the new rule.
The 60.5 Import MP runs a Powershell script to import the management pack into Operations Manager. The last runbook, 60.4 Update Service Request, will update the service request with some information about the new management pack.
This example shows a way to use the self-service portal in Service Manager to order a new event rule in Operations Manager. Orchestrator builds the new rule in a management pack and import it into Operations Manager. A engineer that don’t know much about Operations Manager can still “author” a new rule and import it into Operations Manager. You could include approval step in the process and you can also include a check on the Orchestrator side to make sure the management pack and the new rule is according to best practices.
You can download my example runbooks here, 60 Create OM Rule , please note that this is provided “as is†with no warranties at all. This is not a production ready management pack or solution for your production environment, just a idea and an example.
vNext of this example could include version handling in each MP, should be easy to build with a couple of counters. Also information about the service request requesting the new management pack version could be included in the management pack description, shown in the Operations Manager console.
Microsoft Certified Solutions Expert: Private Cloud
Yesterday I did the “70-247: Configuring and Deploying a Private Cloud with System Center 2012” exam. This is the second exam I needed to become Microsoft Certified Solutions Expert: Private Cloud. A couple of years ago I did Microsoft Certified Solutions Associate (MCSA): Windows Server 2008 (but I think it was named Certified Administrator back then), those exams together with 70-246 and 70-247 is equal to MCSE: Private Cloud. You could also combine the private cloud exams with Windows Server 2012 exams. You can read more about that at Microsoft Learning website. The exam was quite deep and included most of the System Center 2012 components. I was a bit surprised about the exam, it was really covering most of the System Center components and I think it would be difficult to pass without some real world hands-on experience. The exam measure the following areas 2012
- Design and Deploy System Center
- Configure System Center Infrastructure
- Configure the Fabric
- Configure System Center Integration
- Configure and Deploy Virtual Machines and Services
I recommend everyone working with System Center 2012 and private cloud to take the training and do the exam. Microsoft Certification helps you stand out, you demonstrate your expertise to customers and employers, special these really deep certifications.
Execute a service request at a later date
I received a questions a couple of days ago how to delay a runbook? The scenario was that someone submit a service request in Service Manager which includes a couple of runbook activities, but these runbook activities should not run until two days later. As we don’t want runbooks to be hanging, looping or paused for two days we can’t simple add a “wait two days activity†in a runbook. We also want to see in the service request in Service Manager that we are waiting for Orchestrator, and the service request should not be marked as completed until the runbook activities have run.
There are a number of ways to solve this. In this blog post I will show one where we use multiple runbooks and an external database to store data temporary. The scenario in this example is that you order a server reboot from the self-service portal in Service Manager. In the portal you pick a date and also set a checkbox if the server is an IIS server. Server reboot is only allowed after office hours, in this example around 23:00 every evening. The process is
- You submit a service request from the self-service portal in Service Manager, saying that a server need to be rebooted at a specific day. You also set if this service is a IIS Â or not
- A runbook is triggered and write the service request data into a external database
- The service request moves to next activity which is a manual activity
- Another runbook is running every day at 2300 and checks the external database for reboot jobs that should be executed at that day. If there is a job the runbook reads all the service request details from the external database, restarts the machine and updates the manual activity. If the server is a IIS server some extra steps is executed during the restart
- Monitor Date/Time, trigger the runbook every day at 2300
- Query Database, ask the external database if there are any reboot jobs to execute (SELECT * FROM Reboot WHERE DATE <= GETUTCDATE())
- If there are any rows returned the runbook moves to Restart System. Restart System reboots the target machine, it also sends the service request id to the target machine shutdown tracker.
- Run .Net Script, waits five minutes (Start-Sleep -s 300)
- Get Computer/IP Status, tries to ping the machine. If percentage of packets received is 100 the runbook moves to Query Database (2) else it generates an alert in Operations Manager
- Query Database (2) query the external database if the target machine is a IIS or not. As we use that on the link as condition we need to do the query again. A link can only have a condition based on the previous activity.
- If the target machine is a IIS we check the web server service, if the service is not equals “Service Running” we generate an alert in Operations Manager
- Of the machine is not a IIS or if the web server service is running, we move to Get SR
- Get SR picks up the service request
- Get Relationship gets all related manual activities, in this example we only have one, named “Waiting for Orchestrator to reboot”
- Update Activity, set the “Waiting for Orchestrator to reboot” activity to completed
- The last Query Database activity deletes the reboot job from the external database
Don´t forget to add a couple of activities to handle Operations Manager maintenance mode in your reboot runbooks too 🙂
As a alternative solution you could also create a new custom activity instead of using the default manual activity (example of how to build custom activity can be found here), then have only one runbook that checks every day at 2300 for activities of that class in “ongoing state”. The runbook can read all the settings from the custom activity, for example which server to reboot and when. If “when” is today the runbook reboots the server and marks the activity as completed.
Note that this is provided “AS-IS†with no warranties at all. This is not a production ready management pack or solution for your production environment, just a idea and an example.
Building a change calendar with Orchestrator and Service Manager
A change calendar keeps everyone informed about when changes will be performed and also gives a overview of planned changes in the environment. We handle change requests as work items in Service Manager but there is no change calender out-of-the-box in Service Manager. In this blog post I will show how to build a change calender with a shared calender in Exchange, Service Manager and Orchestrator. The idea is what when a change request is created in Service Manager a appointment is created in a shared Exchange calender. Then all engineers can access that calender and see all planned changes in one view.
The first thing to create is the runbook. The runbook monitor Service Manager for new change request, when there is one, the runbook trigger and create the appointment in Exchange. The runbook is quite small as you can see in the image below.
I use the Exchange Users Integration Pack to create the appointment, download it here. The “Monitor New Change Request” activity is configure to trigger on all new objects of Change Request. The “Create Appointment” activity and the Exchange connector is configured according to the images below.
Below is a image of the result in the shared calender, that every engineer can see from Outlook
Done! It is a very quick and easy solution that will bring a lot of value for many organisations. But what if you want to add something manually? Then you can of course create a appointment manual in Outlook and invite the shared calender, but you could also build a service request and use the self service portal in Service Manager to add that. Include a runbook activity in the service request that create the appointment.
What if you need to delete or update a appointment? The Exchange User integration pack will publish a ID for the item you create, in this case a appointment. Write that ID back to the service request. Then you can create a runbook that monitor change requests for changes, for example status changed to cancelled, and trigger a runbook that delete or update the change request. If you then have the Exchange appointment ID stored on the change request in Service Manager it is easy to pick it up and update/delete the correct appointment, instead of trying to find the correct appointment based on title or start date. In the runbook below I have added a Update Object activity that writes the Exchange ID to the change request.
Recent Comments