Here are some notes from what I did when I installed a new agent in a workgroup/untrusted environment with Operations Manager 2007 R2. In this example my CA server is named DC01 and my agent is named DMZ-a01. These steps presuppose that the other side, a gateway server or a management server, is already configure correct. In this scenario I had a gateway server between the management server and the agent.
Remember to configure your management group to allow manually installed agent (Administration/Settings/Security).
1. From the agent, browse to http://dc01/certsrv
2. Add http://dc01 to your trusted sites in IE
3. If you get a error saying that the CA must be configured to use HTTPS authentication or ActiveX can’t be loaded, change the security settings for trusted sites zone, enable Initialize and script ActiveX controls not marked as safe for scripting. Then reload the page
3. Download a CA certificate, certificate chain, or CRL
4. Download CA Certificate chain
5. Once the cert is downloaded, open a MMC with the Certificates (Local Computer) snap-in and import the certificate under Trusted Root Certification Authorities
The next step is to request and install the proper certificate from the root CA server.
1. From the agent browse to http://dc01/certsrv
2. Request a certificate
3. Advanced certificate request
4. Create and submit a request to this CA
5. If you get a error saying that the CA must be configured to use HTTPS authentication, change the security settings for trusted sites zone, enable Initialize and script ActiveX controls not marked as safe for scripting. Then reload the page
6. On the Advanced Certificate Request page input
Name, needs to be FQDN of the machine, for example dmz-a01
Type of certificate needed: Other
CSP: Microsoft Enhanced Cryptographic Provider v1.0
Check Mark keys as exportable
Name: needs to be FQDN of the machine, for example dmz-a01
7. Submit the request
8. On your root CA (DC01), open the Certification Authority console, issue the certificate under pending requests
9. On the machine that request a certificate (DMZ-a01), browse to http://dc01/certsrv
10. View the status of a pending certificate request
11. Install the certificate by clicking on it
12. Open a MMC with the Certificates snap-in, for “my user account”. Under Personal certificates, export the certificate including the private key.
13. Open a MMC with the Certificates snap-in, for “local computer”. Import the certificate under personal certificates.
Next step is to install the agent software. You need to copy the MOMCertImport.exe and the suitable agent folder (for example AMD64) from the installation source to your agent machine. Run the agent setup, input the FQDN of the management server and the name of the management group. Select a action account, if you unsure, select local system and click next. When the installation of the agent is complete, run MOMCertImport.exe and import the certificate. Then restart the System Center Management service (HealthService).
Your agent should now show up under pending management in the Operations Manager console. Approve it and you are done!
There is a tool, certificate generation wizard, that you can download here, it can make untrusted agent scenarios easier.