Monitoring desktop client hardware, operating system and application faults can be of great value in terms of reducing total cost of ownership (TCO) through identification of widespread faults in the monitored environment. In Operations Manager Agentless Exception Monitoring (AEM) can help you with that. AEM enables you to monitor operating systems and applications for error. Client monitoring configures clients to send error reports to an Operations Manager Management Server. With the reporting function client monitoring mode can then generate reports based on these errors. AEM clients don’t need a agent and all client settings are controlled by a group policy.

This week I tested to run AEM between two un-trusted Active Directory forests and it seems to work fine. The biggest challenge is to get name resolution to work, to find a common share where to write error logs and also the design of Operations Manager. Each computer that reports a application error with AEM will be seen as a Windows Computer, with unknown state. Therefor you might want to use a dedicated management group for AEM and client monitoring in general. In the picture below you see a AEM view from the OM12 console, including errors from two different forests.

More info around AEM here.

This week when I was working with Linux monitoring from a resource pool in Operations Manager 2012 I got this alert

SSL Certificate Error
Description: The SSL Certificate used by the Agent has a configuration error.

The server certificate on the destination computer (SERVERNAME:1270) has the following errors: The SSL certificate could not be checked for revocation. The server used to check for revocation might be unreachable. The SSL certificate is signed by an unknown certificate authority.

During discovery of Linux/UNIX machines Operations Manager when the discovery wizard deploys an agent, it retrieves the certificate from the agent, signs the certificate, deploys the certificate back to the agent, and then restarts the agent. Operations Manager uses certificates to authenticate access to the agent computer. During this process the certificate is signed by the management server that is doing the discovery. So even if you configure monitoring to be executed by a resource pool, one of those management servers will do the discovery. After discovery any of management server in the resource pool can do the job, for example run a task against the machine.

You get this alert when another management server in the resource pool is trying to communicate with the Linux/UNIX machine. The root cause is that the another management server is not trusting the first management server as certificate authority, the one that sign the certificate. Therefor when the Linux/UNIX machine is trying to communicate the management server is saying it is using a untrusted certificate. The resolution is to copy the certificate from the management server that sign the agent certificate (Linux/UNIX machine in this case) to all other management server in the resource pool.

More info how to copy the certificate between management servers is here.

Tonight I created a example on how to use a Linux/UNIX command to return data as performance data in OM12. In this example we build a rule that count files in a folder and returns it as performance data.

1. In the Operations Manager console, navigate to Author/Management Pack Objects/Rules
2. Right-click rules and create a new Rule, select Collection Rules/Probe Based/UNIX/Linux Shell Command (Performance) rule
3. Select a management pack
4. General, input a name and a rule target. For example “Field – X plat – Number of files” as name and “SUSE Linux Enterprise Computer” as monitor target
5. Schedule, select how often you want the rule to run the command, for example every 15 minute
6. Shell Command Details, input the script you want to use. The command can be a path to a binary or script file or a single-line shell command. In this example we can the following command to count files in the /tmp folder
   find /tmp | wc -l
7. Filter Expression, use default settings and click Next
8. Performance Mapper, input name of object, counter and instance, for example
   Object: Folder Management
   Counter: Number of files
   Instance: /tmp
9. Click Create and your rule is ready!

You can now navigate to a performance view and show the data collected by the rule (might need to wait some minutes depending on your rule settings)

Earlier this week I did some tests around Ubuntu Server and Operations Manager 2012. I did the same in Operations Manager 2007 R2 and the way to get the monitoring to work is almost the same in both products. All challenges that I meet was the same in both products. Before we continue I would like to remind you that Ubuntu is not support by Microsoft in Operations Manager 2007 or 2012. The management pack and the agent I am using is community projects and is not supported either.

I installed a X86 Ubuntu Server version 10.04.3. I configured it with a static IP-number (sudo vi / etc/network/interfaces (you might need to remove DHCP client to get that setting static sudo apt-get remove dhcp-client)), DNS settings (sudo vi / etc/resolv.conf) and restarted networking (sudo / etc/init.d/networking restart). Note there is a space in front of etc, due to some security setting in the blog platform:) )

If you are in a sandbox and don’t care about the firewall you can disable it by running sudo ufw disable. I would not recommend that for production servers but I would not recommend using a un-supported agent either 🙂

The first discovery result in this error

Second try, after updated the forward and reverse DNS zones, result in this error

As I didn’t had a management pack for Ubuntu or a Ubuntu agent I thought that could be a good next step. There is a Ubuntu agent and a Ubuntu management pack at Codeplex that you can download and extract. You will notice there is two GetOSVersion.sh files, according to instruction at Codeplex you should use these files and replace the default file on your management server (C:\Program Files\System Center Operations Manager 2012\Server\AgentManagement\UnixAgents). Operations Manager copies this file over to the Linux/UNIX machine (/ tmp/scx-username) during discovery and executes the script. The script will get what kind of Linux/UNIX it is and report back to Operations Manager, that decides if it has a management pack or not for the version. The challenge is that we had two files, to decide which one to use you can copy them over to your Ubuntu machine and manually run the, You will then see that only the GetOSVersion.sh that came with the management pack returns valid XML. In other word copy the GetOSVersion.sh from the agent folder to your UnixAgents folder on the management server.

After that I still had some problem with the discovery, so I installed the agent manually on the Ubuntu machine (sudo dpkg -i scx-1.0.4-265.Ubuntu.10.x86.deb) and restarted the server (sudo reboot). After reboot I verified that Microsoft SCX CIM server was running (ps –ef|grep scx).

Then I ran the discovery again and a new error showed up. As you can see in the picture below there seems to be a problem with the certificate that the Ubuntu machine is trying to use. Normally the Linux machine will get a certificate signed by the management server, during the discovery. But in this example we installed the agent manually so the certificate is self-signed by the Ubuntu machine. If you copy the certificate file (/ etc/opt/microsoft/scx/ssl/scx-host-ubuntu02.pem) to a Windows machine and rename it to .cer you can open it and look at it. To solve this certificate issue, copy the certificate from your Linux box to your management server, run scxcertconfig -sign scx-host-<hostname>.pem scx_new.pem. Then rename scx_new.pem to the name of your Linux generated certificate and replace it on your Linux box. Restart the SCX service (sudo scxadmin -restart).

After that the discovery worked fine and the Ubuntu machine showed up healthy in the console. Don’t forget to configure accounts and profiles for your Ubuntu machine

 Please note that this is un-supported by Microsoft and provided “as is” with no warranties at all.

On the Operations Manager 2012 Release Candidate download page there is a file named Microsoft.Unix.ShellCommand.Library.exe. The description of this package is

UNIX/Linux Shell Command Template Management Pack
This Management Pack implements authoring templates that allow the creation of rules, tasks, and monitors based on execution of shell commands on UNIX/Linux agents

If you download this file, extract it and import the management pack you will see some new features in the console

Of course this is a fantastic tool for us Operations Manager administrators. In March 2009 I wrote a blog post how to monitor a logfile on Linux, Logfile Check on Linux. The scenario was to monitor a file size. The challenge was that we first needed to use a bash script to check the file size, if the file size was above the threshold we wrote a warning to a logfile. The script was triggered by a cron to run every X minute. Then we used default features in OM 2007 R2 to monitor the logfile. That was the solution in 2007 R2. Cron > bash script > logfile > Operations Manager 2007 R2.

Now, with this new feature in Operations Manager 2012 we can run the script direct from a monitor, send back the result and control the status of the monitor. To do the same as we did in the blogpost from March 2009, but with Operations Manager 2012, follow these steps

1. In the Operations Manager console, navigate to Author/Management Pack Objects/Monitors
2. Right-click monitors and create a new Unit Monitor, select Scripting/Generic and UNIX/Linux Shell Command Two State Monitor
3. Select a management pack
4. General, input a name and a monitor target. For example “Field – X plat – File Size” as name and “SUSE Linux Enterprise Computer” as monitor target
5. Schedule, select how often you need to monitor to check the file size, for example very 15 minute
6. Shell Command Details, input the script you want to use. The command can be a path to a binary or script file or a single-line shell command. In this example we are looking at the script.sh file, if it larger than 4
   s=$( stat -c %s /root/script.sh); if [ $s -gt 4 ]; then echo $s Error; else echo “Ok”; fi;
7. Error Expression, use the default parameter name and configure “Operator Contains Error”
8. Healthy Expression, use the default parameter name and configure “Operator Contains Ok”
9. Configure Health, use default settings
10. Configure Alerts, check “Generate alerts for this monitor”. Input a alert description. In this example I have insert WS-Man data as a variable
    
11. Click OK and your monitor is ready!

When the file size is about the threshold, in this script example that is 4, you will get an alert looking like this

As I friend of mine would say, “That is pretty much Ninja!” 🙂

In some scenarios you want to forward an alert to a engineer direct from the Operations Manager console. In this post I will show you a example how that can be done with a task. The task will run a power shell script that picks up properties of the alert and forward it by e-mail. Start by copy the script, mailforward.ps1, to C:\scripts on the machine running the console. The create the console task,

1. In the Operations Manager console, navigate to Authoring > Tasks and create a new task
2. Create a Console Tasks/Alert command line. Select a destination management pack and click Next
3. On the General Properties page, input a task name, for example “Forward by e-mail”
4. On the Command Line page, input
• Application: %SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe
• Parameters: C:\scripts\mailforward.ps1 ‘$Name$’ ‘$Description$’ ‘$Managed Object Name$’
• Working directory: C:\
• Check “Display output when this task is run”
5. Click OK to save the task

Now, when you select an alert you can see your new task in the actions pane, and if you click it you can input a note and a recipient, in the task output your will see the complete e-mail that is sent.

Note that in this example it is only the e-mail alias that you need to input, not the complete e-mail address. If you need to input a complete e-mail address you will need to update the powershell script. You also need to update the script with your e-mail domain, from address and mail server. You can download the script here, mailforward. Place it in C:\scripts on each machine that is running the console, or on a shared disk. Make sure that your task is using the correct path to the script. Do not forget to allow your console workstation to send e-mail. with your mail server

Please note that this is provided “as is” with no warranties at all.