ITM Centralized Configuration and the Log File Agent

Some time ago I wrote about the Centralized Configuration feature of IBM Tivoli Monitoring.  Introduced in version 6.2.2 Fixpack 2, this feature provides ITM agents with a pull-based mechanism that enables them to dynamically obtain configuration files, such as private situations and situation thresholds from a central source.  The central source can be another ITM agent or a web server.

One agent that has really embraced and exploited this feature is the ITM Log File Agent.  As the name suggests, this agent provides a log monitoring capability and supports the monitoring of text based log files, the Windows Event Log and errpt on AIX.  The Log File Agent inherits an operating model from the TEC Adapter, so the use of a format file and configuration file to define the required monitoring will be immediately familiar to anybody with a TEC and Tivoli Framework background.  The TEC Adapter style format files, which use a C style printf pattern matching notation, are still supported, however a new regular expression based syntax is available for the development of new formats.

Version 6.2.3 Fixpack 2 of the Log File Agent introduced the “autodiscovery” functionality that really opened the door for use with Centralized Configuration.  Up until this point, a separate agent instance was required for each format and configuration file pairing.  Using the autodiscovery option, an agent instance can now be created and configured with the details of a directory that will contain the required format and configuration files.  The agent “discovers” these file pairings and automatically generates the sub-node representations as seen in the Tivoli Enterprise Portal.  This discovery process is dynamic, so additional configuration and format files can be placed in the directory and they will be detected and loaded by the agent on the fly, no restart is required.  This functionality means that a single agent instance can be used to perform the monitoring defined in multiple configuration and format file pairings and, more importantly, that the centralized configuration feature can be used to automate the distribution of those files.

In the remainder of this article I’ll describe how Centralized Configuration and the Log File Agent can be configured to work together.

The Central Source

The first stage is to configure a central source that will act as the distribution point for the format and configuration files.  Although an ITM agent can be used for this purpose, it’s more practical to use a dedicated web server if possible. For this article I’m using a Nginx web server running on my TEMS.  To simplify the management of the files distributed from the server I’ve adopted a separation of content based on the destination agent type.  So for my Log File Agents I’ve created an lo directory under a parent cc directory which is served by the web server. Within the lo directory I’ve created two further directories, a hosts directory which contains the master load configuration file for each server and a files directory, which contains all of the format and configuration files to be distributed out.  As a result I have a directory structure which looks like:

  • cc
    • lo
      • hosts
      • files

The Agent

To activate the Centralized Configuration for an agent a configuration load list needs to be created.  This load list is used to define the server that the agent should connect to and the files that it should request. The default location for the Log File Agent’s load list (assuming the default installation path on Linux/UNIX is used) is /opt/IBM/ITM/localconfig/lo/lo_cnfglist.xml, although it can be changed using the IRA_SERVICE_INTERFACE_CONFIG_LOADLIST environment variable.

As described in the ITM Administrator’s guide, a neat way to approach the initial load list configuration is to use a generic bootstrap.  By utilising the load list key word substitution feature a generic configuration can be constructed that will result in the agent downloading its specific configuration file hosted from the central web server.   The following XML implements the bootstrap:

<ConfigurationArtifact>
    <ConfigServer Name="CENTRAL-CONFIG-SERVER" URL="http://nitrogen/cc/" />
    <ConfigFile
        Server="CENTRAL-CONFIG-SERVER"
        Name="@HOSTNAME@_cnfglist.xml"
        Disp="CNFGLIST"
        Path="@PRODUCT@/hosts"
        Activate="YES" />
</ConfigurationArtifact>

The ConfigServer element defines the central server and most importantly the base URL the agent should connect to.  In this example the web server is running on the same nitrogen host as the Log File Agent.

The ConfigFile element is used to define each file that the agent should request, which in this example would result in the agent requesting the following URL http://nitrogen/cc/lo/hosts/nitrogen_cnfglist.xml when the @@ keywords are substituted with the relevant hostname and product values. The Disp attribute is used to denote the Disposition of the file being requested and it gives the agent an indication as to where it should place the received file and how to process it.  In this case we specify a Disp type of CNFGLIST, which indicates that the file received is another configuration load list.  The Activate attribute informs the agent that it should merge and process the new configuration load list.

The Centrally Hosted Load List

With the initial generic bootstrap load list in place, attention turns to the load list we want the agent to ultimately end up with.  As detailed in the bootstrap configuration, this file needs to be located in the cc/lo/hosts directory of the web server and needs to be named nitrogen_cnfglist.xml.  It’s in this file that we will define the Log File Agent specific configuration and format files that the agent should automatically pull down.

Here is an example:

<ConfigurationArtifact>
    <ConfigServer Name="CENTRAL-CONFIG-SERVER" URL="http://nitrogen/cc/" />
    <ConfigFile
        Server="CENTRAL-CONFIG-SERVER"
        Name="@HOSTNAME@_cnfglist.xml"
        Disp="CNFGLIST"
        Path="@PRODUCT@/hosts"
        Activate="YES" />
    <ConfigFile
        Server="CENTRAL-CONFIG-SERVER"
        Name="syslog.fmt"
        Disp="CUSTOM"
        Path="@PRODUCT@/files"
        LocalPath="@ITMHOME@/config/lo"
        LocalName="syslog.fmt" />
    <ConfigFile
        Server="CENTRAL-CONFIG-SERVER"
        Name="syslog.conf"
        Disp="CUSTOM"
        Path="@PRODUCT@/files"
        LocalPath="@ITMHOME@/config/lo"
        LocalName="syslog.conf" />
</ConfigurationArtifact>

The first thing to notice here is that the load list is effectively self-referencing i.e. the first ConfigFile entry refers to itself.  This is fundamental to the operation of centralised configuration and ensures that any changes to the load list held centrally, such as the addition of new files for example, will be detected and processed by the agent.  The subsequent ConfigFile entries refer to the configuration file and format file that will be loaded into the Log File Agent.  These files are located in the cc/lo/files directory on the web server. The Disp attribute is set to CUSTOM, which means the LocalPath and LocalName attributes are required to inform the agent where the received files should be stored.  The @ITMHOME@ key word is used to reference the installation directory, so in the case of this agent the syslog.fmt and syslog.conf files will end up in the /opt/IBM/ITM/config/lo directory.  This is significant, as we will see shortly.

The Log File Agent

The last stage is to configure a Log File Agent instance to use the autodiscovery feature.  This is simply done by creating a new instance and ensuring that a valid directory is specified for the LOG_FILE_ADAPTER_GLOBAL_SETTINGS.KLO_FILE_DISCOVERY_DIR configuration parameter.  By default this is set to ${CANDLE_HOME}/config/lo, which just happens to be the LocalPath defined in the load list for the format and configuration files.

LO_config

When creating the instance I tend to use a name of “autodiscovery”, so it is clear what the instance relates to.  Note also that values for the configuration and format file do not have to be supplied.  If you are only running an autodiscovery instance you will probably want to modify or disable the KLO_Log_Agent_Config_Error situation though, as this will erroneously trigger on the INACTIVE status of the LogfileEvents/LOG FILE node.

The Process Flow

With the relevant configuration files in place, the resulting processing performed by the Log File Agent on startup is as follows:

  1. The agent finds the initial bootstrap load list and connects to the centralized configuration server in order to acquire the referenced file
  2. The file is pulled down, overwriting the initial bootstrap version, and is then processed
  3. The new load list now contains references to the syslog.fmt and syslog.conf files which are duly downloaded and placed in the /opt/IBM/ITM/config/lo directory
  4. The Log File Agent discovers the presence of these files, automatically creates a new sub-node and commences monitoring of the relevant files

To deploy additional configuration/format file pairings to the agent, the central load list file just needs to be updated to include the relevant ConfigFile entries.  The Log File Agent will periodically check the timestamp of this file and will download and process it if a modification is detected.  The frequency that this check is performed is controlled by the IRA_SERVICE_INTERFACE_CONFIG_INTERVAL environment variable, which defaults to 60 minutes.  This process also applies to the configuration and format files, so if any changes are made, such as the inclusion of a new format, the Log File Agent will automatically update itself with the new version.

As I’ve hopefully demonstrated, Centralized Configuration and the Log File Agent work together like strawberries and cream*, simplifying the ongoing management and administration and for those customers looking to migrate away from TEC and the Tivoli Framework, it’s one less hurdle to overcome.

* It’s that Wimbledon time of year again, I had to get a reference in somewhere!

Visits: 422

Author:

Ant Mico