ITM6 Data Collection via SSH – A simple check box?

Introduction

ITM Agent Builder v6.2.2.2 enhanced the “Output from a script” data source to enable commands to be run on remote systems using “ssh”. This has proved to be very useful for the collection of monitoring data from unsupported versions of UNIX, Linux and Windows.

Enabling an agent for the remote collection of data is easy enough, check the box “Enable data collection using SSH”, but there is little IBM documentation to assist with the set-up of “Public Key” authentication for such connections. The Agent Builder guide just states “The generation and distribution of the public keys is an administrative task that must be done outside of the agent and Agent Builder.” Not very useful for first timers. The following steps explain by example that set-up.

Background

The Agent Builder enables the systems administrator to build ITM6 agents to collect custom monitoring data from resources, monitoring data that may not be available from standard IBM agents. A “Data Sources” is configured through the Agent Builder to identify how monitoring data is collected and how that data is interpreted. One of the most useful data sources is “Output from a Script”, enabling any script to be run to collect monitoring data.

Historically, the script for such a data source was always executed locally on the server where the ITM6 custom agent had been installed. As of ITM6 Agent Builder v6.2.2.2 the check-box “Enable data collection using  SSH” appeared, see figure 1. Selecting that check-box automatically adds the section “Configuration for Secure Shell (SSH)” to the “Runtime Configuration” options and is usually enabled as part of an agent supporting “multiple instances” or “sub-nodes”, thus enabling a single server to collect data from multiple distinct systems.

"Enable data collection using SSH" check-box

Figure 1

 

In such agents the SSH “Runtime Configuration” parameters must be configured post-installation either for each instance of an agent or for each sub-node. These parameters identify the specific resources to be monitored by specifying the server on which to execute the script.

The SSH connection configuration consists of two pages. The first, “SSH Remote Connection” page requires parameters that are mostly self-explanatory, see figure 2. Those parameters identify the target host, the listen SSH port number and the SSH authentication type. Additionally, options are available to define whether ITM6 disconnects from the target host between each data collection and whether it uploads the monitoring script each interval.

Figure 2

Figure 2

The parameter “Authentication Type” can take the values “Password or “Public Key”. Most people are familiar with “Password” authentication as this mimics the usual manual “SSH” connection method, used by most administrators when connecting via SSH. Hence, “Password” authentication is simple to set-up. However, this method of authentication is generally discouraged for such automations.

The more secure option is “Public Key” authentication (or PKI – “Public Key Infrastructure”). This requires a pass phrase protected private and public key on the server hosting the agent with the public key also on the target server. Although PKI set-up is relatively well documented for common technologies such as OpenSSH it is less well documented for the ITM6 set-up as in this scenario.

The following steps explain by example that set-up. It is based on a scenario where an agent called “Remote Command” is installed on a server with the hostname “vm60”, termed the “agent host” in the notes. This agent is being configured to collect data via SSH from a server with the hostname “vm50”, termed the “target server” in the notes. In the scenario both servers are running CentOS v6. To enable the SSH collection of the data using the “Public Key” authentication option requires:

  1. Public-Private key pair to be created and stored on the agent host (vm60)
  2. Configuration of the “Remote Command” agent on the agent host (vm60)
  3. Installation of the Public Key on the target server (vm50)

 

Public-Private Keys

SSH on Linux includes the tool “ssh-keygen” to create Public-Private Key pairs. In this scenario the key pair will be used exclusively by ITM6 and hence the key pair will be created in a sub-directory of $CANDLEHOME. Access to that sub-directory will be restricted for security reasons.

vm60> mkdir /opt/IBM/ITM/ssh-keysvm60> chmod 700 /opt/IBM/ITM/ssh-keys

 

When creating the key pair the administrator is prompted for the fully qualified file name for the private key and a pass phrase. The public key is created with the extension “.pub” added to the private key filename. The filenames and pass phrase will be required when configuring the custom ITM6 agent.

vm60> ssh-keygen -t rsaGenerating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa): /opt/IBM/ITM/ssh-keys/remote_agent_key

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /opt/IBM/ITM/ssh-keys/remote_agent_key.

Your public key has been saved in /opt/IBM/ITM/ssh-keys/remote_agent_key.pub.

The key fingerprint is:

57:42:3a:24:fa:31:0f:ba:b7:ca:eb:24:fd:2e:f2:be root@vm60.orbdata.test

 

This command creates the key pair in the specified directory. You may wish to change the access permissions to the public key.

vm60> ls -ltr /opt/IBM/ITM/ssh-keystotal 8

-rw-r–r– 1 root root  404 Apr 23 11:23 remote_agent_key.pub

-rw——- 1 root root 1743 Apr 23 11:23 remote_agent_key

vm60> chmod 700 /opt/IBM/ITM/ssh-keys/remote_agent_key.pub

 

Configuration of the Custom ITM6 Agent

Configuration of the ITM6 custom agent on the agent host, either from the MTEMS or invoked from “itmcmd config -A”,  will prompt for the “Remote SSH Connection” details and subsequently present a “Public Key” configuration page (assuming that authentication type is selected).

The “Public Key” configuration page requires the values for the fully-qualified public and private key filenames and the pass phrase as used during the creation of the key pair. It also requires a valid user name for the target system, in this scenario “netcool is used.

Figure 3

Figure 3

The remote data collection will be executed in the context of that user, hence the user must have the relevant permissions to collect the required monitoring data. The same user is used for ALL SSH connections where there are multiple “Output from script” data sources configured.

Installation of the Public Key on the Target Server

An SSH daemon must be running on the target server for the ITM6 custom agent to connect. Such tools are common on UNIX and Linux platforms. The only custom configuration required is to import the Public Key. This is commonly a simple update to the file “authorized_keys” located in the sub-directory “.ssh” of the user’s home directory on the target server.

For this scenario the “netcool” user on the target server was configured for the data collection. In this case there was no existing sub-directory “.ssh”, hence the both the sub-directory and file were created. For security reasons the directory and file access permissions are restricted. Note that such access permission restrictions are required for OpenSSH to accept the public key, it will ignore the public key if those permissions are not set.

The Public Key can be copied to the target server either as a file or via a cut-and-paste of the file content. The former was used in this instance.

vm50> mkdir .sshvm50> cat /tmp/remote_command_key.pub  >> .ssh/authorized_keys

vm50> rm /tmp/remote_command_key.pub

vm50> chmod -R 700 .ssh

vm50> ls -ltr .ssh

total 4

-rwx——. 1 netcool netcool 404 Apr 23 11:58 authorized_keys

Data Collection Issues

The custom agent can be started once the above steps have been completed. The monitoring data collected should be visible from the default workspace for the Navigator Item representing the relevant data source. If no data is displayed navigate to the “Performance Object Status” workspace and check the report for an “Authentication Failed” in the “Error Code” column.

Figure 4

Figure 4

Further details may be available in the agent log file located in the standard path “$CANDLEHOME/logs”, for example:

vm60> tail /opt/IBM/ITM/logs/vm60_99_vm50_k99agent_53579db7-01.log
(53579EA3.0000-9:userauth.c,1110,"userauth_publickey_fromfile") -18 - Username/PublicKey combination invalid
(53579EA3.0001-9:sshconnector.cpp,853,"SshConnector::connect") (vm50) Authentication by public key failed!

Conclusions

Simple? Yes, for those familiar with ITM6 Agent Builder, creating a custom agent to collect data from a remote system using the “output from script” data source is a simple step. For those unfamiliar with the use of PKI and SSH, hopefully the above steps will ensure the runtime configuration is also a simple step.