Building a SLEEP activity for Orchestrator

Posted: 18th January 2012 by Anders Bengtsson in Orchestrator
3

When building runbooks you sometimes need to include a delay or a sleep. You can configure a delay on the links between activities, but it is not always a good solution. In the example in the image below I have a delay on each link. As you can see it looks like the runbook is on the first activity, but it is not, it is on the link, waiting. You don’t see the link as a step in the runbook and you cant see the link delay as a step in the history log either. A better alternative is to use a “wait” or “sleep” activity. In this blog post I will show you how you can build a “sleep” activity with Orchestrator Integration Toolkit. We will use a Powershell cmdlet named start-sleep and input number of seconds we want to sleep.

  1. Download and install the Orchestrator Integration Toolkit
  2. Start the Orchestrator Command-Line Activity Wizard
  3. On the welcome page click Next
  4. Assembly Details. Input a name a assembly file, for example ContosoSleep as name and C:\IP\ContosoSleep.dll as assembly file
  5. Commands. Click Add
  6. Add/Edit Command. Input
    Name: Sleep
    Mode: Run Windows PowerShell
  7. Switch to the Arguments tab, click Add parameter
  8. Add/Edit Parameter. Input
    Name: Number of seconds to sleep
    Usage mode: Command Argument
    Display Style: Text
    Default value: <leave blank>
  9. Click Ok
  10. On the Arguments tab, in the Command Line text field, input
    start-sleep -s
    then click Insert and Number of seconds to sleep
  11. Click Ok
  12. Commands. Click Next
  13. When you see “The wizard completed successfully” click “Build Integration Pack”
  14. Product Details. Fill in the information, for example
    Product Name: Contoso Sleep
    Category Name: Sleep
    Company: Contoso
    You can click Modify and change the icon. Click Next
  15. Activities. Verify that your Sleep activity is listed, then click Next
  16. Dependencies and included files. Our example don’t need any extra files so click Next
  17. Orchestrator Integration Pack File. Input a path where the Integration Pack will be created. Click Next
  18. The integration pack is created. Click Finish to exit the wizard

Now, register and deploy the integration pack with Deployment Manager. After that is successfully you can restart your Runbook Designer console and you will see a new Integration Pack with your new activity.

As you can see in the image below we can now easy see that the runbook is in “sleep mode” as Sleep is its own activity. More examples with the Orchestrator Integration Toolkit can be found here. You can download my sleep IP here. ContosoSleep . Please note that this is provided “as is” with no warranties at all.

The Configuration Manager dashboard with Orchestrator

Posted: 11th January 2012 by Anders Bengtsson in Orchestrator
0

A couple of years ago I wrote a blog post around using the Configuration Manager dashboard with Operations Manager, link. Now days Operations Manager have their own dashboard. But today I needed one for Orchestrator. I tried the Configuration Manager dashboard and it seems to work fine with Orchestrator. As you configure the dashboard with SQL queries you can query any database and show the result in the dashboard.

All SQL queries I used can be found in my “Runbook Validation” blog posts, found here. Instructions how to configure and install the dashboard can be found in my old blog post here.

Automate Operations Manager with Orchestrator and Service Manager

Posted: 27th December 2011 by Anders Bengtsson in Operations Manager 2012, Orchestrator, System Center Service Manager
1

Lately I have been working with a number of automation scenarios where we use Service Manager and Orchestrator to automate Operations Manager. The scenarios have not really been target to end-users, instead to Operations Manager administrators and subject matter experts (SME). For example install agent and uninstall agent, maintenance mode and disable alerts. All this can be done from the Operations Manager console, but that require first knowledge and then of course permission. Another benefit with automate it based on a form is that is will be done the same everytime, independent on which administrator performing the task.

For example if you want to allow subject matter experts, like your Exchange team, to install Operations Manager agent on new Exchange machines, you will need to give them Operations Manager administrator permissions. Alternative they will contact your Operations Manager administrator’s team and request to get the agent installed. With Service Manager 2012 self-service portal you can publish this as a service request. Anyone with enough Service Manager permissions can request an agent in the portal, an operations manager administrator can approve it and then Orchestrator will install the agent.

Another common scenario is that a team gets an alert that they don’t want. For example a service is not running on server X. Instead of forwarding this ticket to the Operations Manager team they could simple disable it for the server themselves, after an Operations Manager administrator have approved it in Service Manager.

In this blog post I want to share with you a runbook that disable a rule or monitor based on ticket ID from Service Manager. The scenario is that a team, for example the Exchange team, gets an incident in Service Manager; the incident is generated from an alert in Operations Manager. After some thinking they realize that they no longer want to monitor that for that server. They browse to the self-service portal and input the incident ID (ticket ID in Operations Manager), after approval, Orchestrator find the rule or monitor generate that alert and disables it for the machine. The Orchestrator runbook also handle override management pack. The Exchange team could also browse to the Orchestrator console and start the runbook.

Another discussion is if this is a service request of a change request. The current build of Service Manager 2012 allows us to publish service requests direct in the portal, not change requests. Therefor I will use Service Requests in this example.

  1. The first activity, gets Ticket Id and target server name as input. Which Operations Manager management server the runbook is using is controlled by a variable. All activities use the same variable to contact the same management group.
  2. This activity is used to format date and time which is used for the management pack backup folder
  3. Powershell script, loads the Operations Manager module and gets the alert with the ticket ID we input when the runbook was started. The script then checks if the alert is from a rule or a monitor. As we need to use different cmdlets to disable a rule or a monitor we need to know if the source is a rule or a monitor. Next step finds out the orginal management pack, the management pack that stores the alert source. The script checks if there is a suitable override management pack. The override management pack name is “Organisation – orginal name” for example “Contoso – Microsoft Unix Library”. “Contoso” is controlled by a variable. All steps will echo to the logfile, default path is C:\TEMP\logfile.txt
  4. If there is no suitable override management pack this activity creates it, as a XML file in the C:\TEMP folder
  5. Imports the override management pack created in step 4
  6. Overrides the rule or the monitor
  7. If there is a override management pack already step 7 makes a backup of the management pack. The backup of the management pack is stored in C:\TEMP by default. The script will create a folder with the current timestamp and then stores the management pack in it
  8. Overrides the rule or the monitor

An example of this could be that first an alert is generated in Operations Manager and syncronized to Service Manager. Service Manager writes back the incident number to Operations Manager

A SME finds out that this rule or monitor can be disabled for this or a machine. The SME can then navigate to the Orchestrator Console or browse to the Service Manager self-service portal and input the ticket ID and the server for which the monitoring should be disabled

Either way the runbook will be executed and create the override

Summary: Self-service can be target to engineers and SME teams, not only end-users. We use self-service with the Service Manager self-service portal or with the Orchestrator Console to execute tasks that the engineer normally dont have deep skills of or permissions to do. In this example disable a rule or monitor in Operations Manager. We save time both for the engineer and the Operations Manager team. With the Orchestrator console we can use Orchestrator as a “proxy” to carry out complex tasks in a very simple way. With Service Manager we can build a approval flow to make sure every change is approved in a correct way.

If you want to build a runbook that deploys agent to untrusted networks or workgroup machines, where you need certificate based authentication, look at, Deploy OPSMGR agent to untrusted zones with Opalis. Also look at the posts around fault tolerance and logging. If you want to build a runbook to uninstall (and install) the Operations Manager agent, that can be triggered from the Orchestrator console and the Service Manager self-service portal, this is an example

You can download the runbook here, 20111227_DisableRuleMonitor. Please note that this is provided “as is” with no warranties at all.

AEM cross-forest

Posted: 18th December 2011 by Anders Bengtsson in Operations Manager 2012
0

Monitoring desktop client hardware, operating system and application faults can be of great value in terms of reducing total cost of ownership (TCO) through identification of widespread faults in the monitored environment. In Operations Manager Agentless Exception Monitoring (AEM) can help you with that. AEM enables you to monitor operating systems and applications for error. Client monitoring configures clients to send error reports to an Operations Manager Management Server. With the reporting function client monitoring mode can then generate reports based on these errors. AEM clients don’t need a agent and all client settings are controlled by a group policy.

This week I tested to run AEM between two un-trusted Active Directory forests and it seems to work fine. The biggest challenge is to get name resolution to work, to find a common share where to write error logs and also the design of Operations Manager. Each computer that reports a application error with AEM will be seen as a Windows Computer, with unknown state. Therefor you might want to use a dedicated management group for AEM and client monitoring in general. In the picture below you see a AEM view from the OM12 console, including errors from two different forests.

More info around AEM here.

OM12: SSL Certificate Error

Posted: 18th December 2011 by Anders Bengtsson in Operations Manager 2012
0

This week when I was working with Linux monitoring from a resource pool in Operations Manager 2012 I got this alert

SSL Certificate Error
Description: The SSL Certificate used by the Agent has a configuration error.

The server certificate on the destination computer (SERVERNAME:1270) has the following errors: The SSL certificate could not be checked for revocation. The server used to check for revocation might be unreachable. The SSL certificate is signed by an unknown certificate authority.

During discovery of Linux/UNIX machines Operations Manager when the discovery wizard deploys an agent, it retrieves the certificate from the agent, signs the certificate, deploys the certificate back to the agent, and then restarts the agent. Operations Manager uses certificates to authenticate access to the agent computer. During this process the certificate is signed by the management server that is doing the discovery. So even if you configure monitoring to be executed by a resource pool, one of those management servers will do the discovery. After discovery any of management server in the resource pool can do the job, for example run a task against the machine.

You get this alert when another management server in the resource pool is trying to communicate with the Linux/UNIX machine. The root cause is that the another management server is not trusting the first management server as certificate authority, the one that sign the certificate. Therefor when the Linux/UNIX machine is trying to communicate the management server is saying it is using a untrusted certificate. The resolution is to copy the certificate from the management server that sign the agent certificate (Linux/UNIX machine in this case) to all other management server in the resource pool.

More info how to copy the certificate between management servers is here.

UNIX/Linux Shell Command as Performance Data in OM12

Posted: 30th November 2011 by Anders Bengtsson in Operations Manager 2012
0

Tonight I created a example on how to use a Linux/UNIX command to return data as performance data in OM12. In this example we build a rule that count files in a folder and returns it as performance data.

  1. In the Operations Manager console, navigate to Author/Management Pack Objects/Rules
  2. Right-click rules and create a new Rule, select Collection Rules/Probe Based/UNIX/Linux Shell Command (Performance) rule
  3. Select a management pack
  4. General, input a name and a rule target. For example “Field – X plat – Number of files” as name and “SUSE Linux Enterprise Computer” as monitor target
  5. Schedule, select how often you want the rule to run the command, for example every 15 minute
  6. Shell Command Details, input the script you want to use. The command can be a path to a binary or script file or a single-line shell command. In this example we can the following command to count files in the /tmp folder
    find /tmp | wc -l
  7. Filter Expression, use default settings and click Next
  8. Performance Mapper, input name of object, counter and instance, for example
    Object: Folder Management
    Counter: Number of files
    Instance: /tmp
  9. Click Create and your rule is ready!

You can now navigate to a performance view and show the data collected by the rule (might need to wait some minutes depending on your rule settings)

Ubuntu Server in Operations Manager

Posted: 29th November 2011 by Anders Bengtsson in Operations Manager 2012, System Center Operations Manager 2007
2

Earlier this week I did some tests around Ubuntu Server and Operations Manager 2012. I did the same in Operations Manager 2007 R2 and the way to get the monitoring to work is almost the same in both products. All challenges that I meet was the same in both products. Before we continue I would like to remind you that Ubuntu is not support by Microsoft in Operations Manager 2007 or 2012. The management pack and the agent I am using is community projects and is not supported either.

I installed a X86 Ubuntu Server version 10.04.3. I configured it with a static IP-number (sudo vi / etc/network/interfaces (you might need to remove DHCP client to get that setting static sudo apt-get remove dhcp-client)), DNS settings (sudo vi / etc/resolv.conf) and restarted networking (sudo / etc/init.d/networking restart). Note there is a space in front of etc, due to some security setting in the blog platform:) )

If you are in a sandbox and don’t care about the firewall you can disable it by running sudo ufw disable. I would not recommend that for production servers but I would not recommend using a un-supported agent either :)

The first discovery result in this error

Second try, after updated the forward and reverse DNS zones, result in this error

As I didn’t had a management pack for Ubuntu or a Ubuntu agent I thought that could be a good next step. There is a Ubuntu agent and a Ubuntu management pack at Codeplex that you can download and extract. You will notice there is two GetOSVersion.sh files, according to instruction at Codeplex you should use these files and replace the default file on your management server (C:\Program Files\System Center Operations Manager 2012\Server\AgentManagement\UnixAgents). Operations Manager copies this file over to the Linux/UNIX machine (/ tmp/scx-username) during discovery and executes the script. The script will get what kind of Linux/UNIX it is and report back to Operations Manager, that decides if it has a management pack or not for the version. The challenge is that we had two files, to decide which one to use you can copy them over to your Ubuntu machine and manually run the, You will then see that only the GetOSVersion.sh that came with the management pack returns valid XML. In other word copy the GetOSVersion.sh from the agent folder to your UnixAgents folder on the management server.

After that I still had some problem with the discovery, so I installed the agent manually on the Ubuntu machine (sudo dpkg -i scx-1.0.4-265.Ubuntu.10.x86.deb) and restarted the server (sudo reboot). After reboot I verified that Microsoft SCX CIM server was running (ps –ef|grep scx).

Then I ran the discovery again and a new error showed up. As you can see in the picture below there seems to be a problem with the certificate that the Ubuntu machine is trying to use. Normally the Linux machine will get a certificate signed by the management server, during the discovery. But in this example we installed the agent manually so the certificate is self-signed by the Ubuntu machine. If you copy the certificate file (/ etc/opt/microsoft/scx/ssl/scx-host-ubuntu02.pem) to a Windows machine and rename it to .cer you can open it and look at it. To solve this certificate issue, copy the certificate from your Linux box to your management server, run scxcertconfig -sign scx-host-<hostname>.pem scx_new.pem. Then rename scx_new.pem to the name of your Linux generated certificate and replace it on your Linux box. Restart the SCX service (sudo scxadmin -restart).

After that the discovery worked fine and the Ubuntu machine showed up healthy in the console. Don’t forget to configure accounts and profiles for your Ubuntu machine

 Please note that this is un-supported by Microsoft and provided “as is” with no warranties at all.

UNIX/Linux Shell Command in Operations Manager 2012

Posted: 26th November 2011 by Anders Bengtsson in Operations Manager 2012
0

On the Operations Manager 2012 Release Candidate download page there is a file named Microsoft.Unix.ShellCommand.Library.exe. The description of this package is

UNIX/Linux Shell Command Template Management Pack
This Management Pack implements authoring templates that allow the creation of rules, tasks, and monitors based on execution of shell commands on UNIX/Linux agents

 

If you download this file, extract it and import the management pack you will see some new features in the console


Of course this is a fantastic tool for us Operations Manager administrators. In March 2009 I wrote a blog post how to monitor a logfile on Linux, Logfile Check on Linux. The scenario was to monitor a file size. The challenge was that we first needed to use a bash script to check the file size, if the file size was above the threshold we wrote a warning to a logfile. The script was triggered by a cron to run every X minute. Then we used default features in OM 2007 R2 to monitor the logfile. That was the solution in 2007 R2. Cron > bash script > logfile > Operations Manager 2007 R2.

Now, with this new feature in Operations Manager 2012 we can run the script direct from a monitor, send back the result and control the status of the monitor. To do the same as we did in the blogpost from March 2009, but with Operations Manager 2012, follow these steps

  1. In the Operations Manager console, navigate to Author/Management Pack Objects/Monitors
  2. Right-click monitors and create a new Unit Monitor, select Scripting/Generic and UNIX/Linux Shell Command Two State Monitor
  3. Select a management pack
  4. General, input a name and a monitor target. For example “Field – X plat – File Size” as name and “SUSE Linux Enterprise Computer” as monitor target
  5. Schedule, select how often you need to monitor to check the file size, for example very 15 minute
  6. Shell Command Details, input the script you want to use. The command can be a path to a binary or script file or a single-line shell command. In this example we are looking at the script.sh file, if it larger than 4
    s=$( stat -c %s /root/script.sh); if [ $s -gt 4 ]; then echo $s Error; else echo “Ok”; fi;
  7. Error Expression, use the default parameter name and configure “Operator Contains Error”
  8. Healthy Expression, use the default parameter name and configure “Operator Contains Ok”
  9. Configure Health, use default settings
  10. Configure Alerts, check “Generate alerts for this monitor”. Input a alert description. In this example I have insert WS-Man data as a variable
  11. Click OK and your monitor is ready!

When the file size is about the threshold, in this script example that is 4, you will get an alert looking like this

As I friend of mine would say, “That is pretty much Ninja!” :)

Update Operations Manager 2012 alerts with Orchestrator

Posted: 20th November 2011 by Anders Bengtsson in Operations Manager 2012, Orchestrator
0

System Center 2012 is soon here and many are already playing with release candidates and beta versions. As you might notice the current Orchestrator integration pack (IP) for Operations Manager only support Operations Manager 2007 R2. But what if we are running Operations Manager 2012 and Orchestrator in a sandbox and want to try connect them? Some of the activities in the Operations Manager 2007 R2 integration pack might work with Operations Manager 2012, but as it is unsupported it is not a good idea. But in Orchestrator we have a generic activity named “Run .NET Script”. This activity can be used to run PowerShell script, for example Operations Manager 2012 command shell. To do this we need to first install Operations Manager 2012 console and command shell on each runbook server, then we need to change execution-policy on them, to allow our script. Note that Orchestrator is running PowerShell in 32-bit mode. More info about that here.

When you have done this you might need to reboot your runbook server to load Operations Manager dlls. If you get a event in the application log like the event below, solve it with a runbook server reboot.

Time to build the runbook. It is a pretty simple example but I think it gives you a idea what you could do. The first example will return alert name and alert id for all alerts with resolution state equals new (0). As you can see we first load the Operations Manager module, we then create a connection to the Operations Manager management group. In this example it is running on a server named FIELD-OM8381. We then get name and if for each alert with resolution state 0. On the second image you can see settings to output information from the PowerShell script to the Orchestrator data bus. In this example we publish alert name and alert id to the data bus.

If we want to update each alert to a new resolution state we could add another Run .NET with a script like the example below. As you can see we use the output from the first activity (alert id) as input to this activity. We configure each alert with resolution state 1, 1 is equal to Orchestrator in my Operations Manager sandbox.

Active Directory Integration with Orchestrator

Posted: 15th November 2011 by Anders Bengtsson in Orchestrator
0

I guess many of you have start playing with System Center 2012 Orchestrator already, if not, download it here. A common question around Orchestrator is integration with Active Directory and multiple Active Directory forests. Earlier versions of integration packs for Active Directory required the runbook server to be member of the target Active Directory forest, which could be a challenge when working with multiple Active Directory forests. Tonight I did a test with the System Center 2012 Orchestrator Beta Integration Pack for Active Directory (download here) and it seems to work fine between different Active Directory forests, even without a trust. I have Orchestrator RC installed in the field.local AD forest and creating Active Directory objects over in the Contoso.local Active Directory forest. There is no trust between the two forests. Great!

Older Entries