Do you have multiple accounts for administrator of your IT environment? Do you have for example one local administrator account, one Active Directory administrator account, one account to administrate some business service and one account that you use to log on to your workstation and read your e-mail? Do you remember all the passwords? I often see customers having a challenge remember all passwords and when to use which account.

Another funny (or scaring) side of it is that everyone is more or less administrator of everything, it is just a question about clicks. For example an engineer is not a domain admin but administrator of Orchestrators that integrates with Active Directory using a domain administrator account. Another example could be an engineer without any administrator accounts except is Operations Manager administrator, with indirect access to all servers as local system, including domain controllers.

One thing that also crossed my mind is why do so many engineers have permissions to do 200 tasks, when you normally only do 20 tasks? Why do you have an account that could delete the entire domain when all you do is reset passwords and add members to groups? There is an imminent risk of human errors.

I am no security expert but I want to show a way to minimize number of administrator accounts floating around, and maybe minimize human errors. The scenario is that no engineers have more than one account. That account is a normal user account, no special administrator permissions at all. When the engineer are about to do something special he or she request permissions and also describes the work that will be done. The engineer also select number of hours permissions are needed.

Figure 1 The Service Request form

The service request templates includes the following activities

1. Runbook, update approval step. This step will look which system the requester has requested permissions to, and then depending on the system configure the manual approval step. If it is for example Active Directory the runbook activity will add a group of people that can approve Active Directory permissions.
2. Review Activity, “managers” or “owners” approve temporary permission
3. Runbook, grant temporary permissions. A runbook grants the requester permission to the system

Figure 2 Service Request Template

 

The first runbook, 2.7.2 in this scenario, is the runbook that will update the review activity based on the request. The review activity is configured in the service request template without any reviewers, it is the 2.7.2 runbook that will populate the list of Reviewers. A dynamic approval step J

The second runbook, 2.7.1 in this scenario, is the runbook that will grant permission to the requester. It gets the user requesting permissions and add that user to a group in Active Directory. The Map System To Group activity will decide which security group to add the user to, depending on system. In the end of the runbook it updates the service request with information that permissions has been granted and when the permissions will be removed. The Format Timestamp activity is used to get expire time for the permissions. But there is one more extra important activity in the runbook. The Write to Temp Permissions database activity. For each permission granted this activity writes a row to a SQL database.

 

Figure 3 Temp Permissions database

The last runbook, 2.7.3 in this scenario, is removing permissions. Every 10 minutes it query the Temp Permissions database for permissions to remove. If there is any permissions to remove, based on the ExpireDate column, the runbook will remove the user from the security group and remove the record in the database. There will still be logs in the technical log database for Orchestrator, more info about logs here. You will also have a work item, a Service Request, in Service Manager which information about the temporary permission, including requester, approver, description and system.

 

You can download the runbooks, 20131204_TempPermissions. Note that this is provided “AS-IS” with no warranties at all. This is not a production ready solution, just an idea and an example.

From time to time you need to compare what software are installed on two computers. For example when using a test environment to test a new release of a software you want to make sure the test environment has same version installed as the production environment. Service Manager includes all this information as it synchronize it from Configuration Manager and Operations Manager. Unfortunately there is no out of the box report or feature to do this. I guess we could do it with Service Manager Powershell but for an Orchestrator geek like me it is much fast done with a runbook J

Figure 1 shows the master runbook. It invokes three other runbooks, one to gather all installed software, one to compare installed software and one to find all software installed on both computers. The solution use a temporary SQL table to store data. The result, a report, is sent to the Service Request “requester” by e-mail. In the start of the master runbook there are three Service Manager Activities, these are used to get the “requester”. The Send Email activity in the end use this information to send the report file.

Figure 2 shows runbook 8.3.2. This runbook first drops the temporary table if exists. The temporary table is named based on the computer name, if we have ran a comparison before, we delete the temp table. The master runbook will also delete temporary table, but if there is a “left over” from another job it is deleted in the 8.3.2 runbook and also we want to make sure no old data is affecting our new report. The 8.3.2 runbook gets all the installed software on one computer and insert it into the temporary SQL table. The runbook insert both software display name and GUID. Two software could be named the same, so then we also compare GUID of the CI. If we were interesting in installed patches we could use Software Updates class instead of the Software Items class. We run runbook 8.3.2 once of each computer.

Figure 3 shows runbook 8.3.2. This runbook use a SQL query to find software installed on one computer but not the other one. We run runbook 8.3.2 once for each computer.

Figure 4 shows runbook 8.3.4. This runbook use a SQL query to find software installed on both computers.

 

Figure 1. Master runbook

 

Figure 2. Runbook 8.3.2

Figure 3Runbook 8.3.3

Figure 4 Runbook 8.3.4

 

I use Service Manager Portal to submit a service request which will start a runbook activity, runbook 8.3.1. Figure 6 shows an example report, delivered by e-mail. The report is very basic, built in HTML. You could re-design it easy by changing HTML code in the runbook.

Figure 5

Figure 6 Report

 

You can download the runbooks, CompareSoftware. Note that this is provided “AS-IS” with no warranties at all. This is not a production ready solution, just an idea and an example.

If you have an Orchestrator environment with multiple runbook servers might have notice that when a runbook failover to secondary runbook server the runbook restarts at the first activity. Let’s say you have a runbook that creates a new user account, a user folder and deploy a computer to a client. If that runbook is interrupted between creating the user folder and deploying the computer it will failover to secondary runbook server and restart at creating the user account. The issues with this is this, in this example, is that you will end up with multiple user accounts for the same user, or the runbook will crash as it tries to create the same user account again. In other scenarios it could cause multiple installation of patches, multiple restarts of a server or multiple deployed of the same VM.

In this blog post I will show a couple of alternatives to handle state tracking. I will also include an example on logging for your runbooks. In my examples I use a SQL table to store logs from all runbooks.

SQL
For all state tracking you need to have a unique ID of the runbook job. If the runbook only runs in one instance you could use the runbook name or activity ID, but often the same runbook can run in multiple instances at the same time.

First alternative is to use a SQL table. I often create a database called OrchestratorTool or something similar to store logs, temp data and so on, in this database I create tables for different runbooks. It is often complicated to use the same table for state tracking from multiple runbooks.

Figure 1 show a “master” or “parent” runbook that invokes three other runbooks. This runbook is used from Service Manager when order a new user account, it creates an account, a user folder and a workstation. The “master/parent” runbook is the runbook that we use as a runbook activity in Service Manager. The Service Request ID (SR) from Service Manager is the unquiet ID that these runbooks use to do state tracking. After each Invoke Runbook there is a check to make sure the runbook returns “something” and not NULL. In the end of the runbook it updates the service request in Service Manager with the result, for example account name. If any of the three Invoke Runbook activities returns NULL I want the runbook to stop, I use the Fail Runbook activity to do this. The Fail Activity runbook is a Query Database activity with incorrect configuration. This will make sure the runbook stops. All the “Write to log…” Invoke Runbook activities invoke a runbook that will write data to a log table in a SQL database. Figure 2 show configuration example of a “Write to log” Invoke Runbook activity. Figure 3 show the runbook that writes to the log table.

Figure 1

Figure 2

Figure 3

Figure 4, 5 and 6 shows the different runbooks that the “master/parent” runbook invokes. Each of these runbooks starts by checks if for example a folder has been created for the service request. If the folder is not created it will create a folder and update the tracking database table with info. If there is already a folder in the tracking database table for this service request it will publish that information back to the “master/parent” runbook. Each runbook also includes a couple of “Write to log” invoke runbook activities to write data to the log table in SQL. If any of these runbooks fails the error will be written to the log table and the “master/parent” runbook will fail, with the “Fail runbook” activity. As you can see in figure 2 the severity is CRITICAL, when a runbook sends CRITICAL severity to the log runbook (figure 3) it will generate an alert in Operations Manager with the info and write it to the log table. This means that each runbook don’t have to include “Create Alert” activities to generate an alert in Operations Manager, they just need to write a CRITICAL event to the log table.

Figure 4

Figure 5

Figure 6

With the SQL alternative the “master/parent” runbook can be restarted multiple times and each “child” runbook will check if has already ran for the service request. We use a table in a SQL database to store the state of each service request. We need another runbook to clean up the state table in SQL, or do it as a last step in the “master/parent” runbook.

Service Manager

The second alternative is to use Service Manager to store the state of the runbook. In this alternative each runbook is independent of each other, we don’t have a “master/parent” runbook, and instead each “child” runbook is its own runbook activity in Service Manager. Servicedesk operators can easy re-start or skip an activity if needed and customer can track each step. If you have only one huge runbook activity, invoke a “master/parent” runbook that then invokes 50 other runbooks, it is very difficult to troubleshoot or restart.

When we build runbooks that handle success, warning and failure from all activities the runbook always ends with a success. Figure 7 show an updated version of the runbook shown in figure 5, to create the account I Active Directory. If for example the Create User activity fails the runbook will go to the Write to log – Can’t Create Account activity. That activity will invoke a runbook and write information to the log. That invoke runbook activity will most likely work, so the runbook will end as a success even if it was not a success, we couldn’t create the account in Active Directory. That is the reason for the “Fail Runbook” activity, which will make sure the runbook crash. The Fail Activity runbook is a Query Database activity with incorrect configuration. This will make sure the runbook stops.

In the end of the runbooks shown in figure 7, 8 and 9 you can see an “Update RB Activity” activity. This activity will write the result of the runbook, for example an account, to TEXT1 of the runbook activity. Each instance of a runbook activity has a number of text fields that you can use to store information. In figure 8 you can see that we use “Get Related RB activities” based on the Service Request to get the runbook activity named something with 9.2.1. 9.2.1 Is the runbook that create the account, shown in figure 7. The runbook in figure 8 then readers TEXT1 from that runbook activity and use it as input when create the folder.

Figure 7

Figure 8

Figure 9

The log table I use for these examples looks like the table in figure 10. The SQL state tracking table I use for the first example looks like the table in figure 11.

Figure 10

 

Figure 11

Download my example runbooks, StateTracking.
Note that this is provided “AS-IS” with no warranties at all. This is not a production ready management pack or solution, just an idea and an example