Exception Handling in Netcool/IMPACT

An unhandled exception in any Netcool/IMPACT automation solution can result in the break-down of operational workflow and potentially result in critical error conditions going unnoticed.

How do we address this problem?
The creation of effective and efficient exception handling.

Netcool/IMPACT includes a basic level of error handling. Impact treats all errors as ‘unhandled’ exceptions and logs exception messages to the PolicyLogger.log.
The default behaviour is not sufficient to effectively monitor your Netcool/IMPACT environment and additional exception handlers must be created to trap and report errors. Exception handlers can be broken down into two forms:

  1. Policy level exceptions
  2. General/Unhandled exceptions

A key difference in these types of exception handling is that Policy Level exceptions are handled, meaning that the exception will not automatically cause the policy execution to terminate. Impact assumes that the developer understands this error condition and has created to code to work around it.
Impact treats unhandled exceptions as unexpected, fatal errors from which the policy execution cannot recover. An unhandled exception results in the immediate termination of the policy execution.

As exception handlers are essentially sections of policy code, all resources available to your policies are also available to your exception handlers. This feature enables exception handlers to use multiple channels to notify support personnel.


POLICY LEVEL EXCEPTIONS

As the name suggests this form of exception handling is performed at the policy level. Through the use of action functions Handle() and Raise() exception handling code is added to your policies in order to trap specific error conditions and handle them accordingly. There are two specific methods of exception handling at a policy level:

    1. Logical exception handling
    2. Java class based exception handling 

LOGICAL EXCEPTION HANDLING

An exception handler created to handle a specific logical error in the policy code. The Handle() function is used to declare an exception handler. The exception handler is given a logical name and contains a section of code designed to handle a specific error condition. The Raise() function calls an exception handler by name, executing the code and handling the error condition. 

Conditions:

The exception handler must be inserted at the top of your policy, preferably before the main policy code.
Raise() is then called from any point in the policy code as many times as is required to the handle associated error conditions.
An argument can be passed from the main policy to the exception handler via the Raise() function.

Environment:

The environment in which the exception handler runs contains the following objects:

  • The policy environment
    The policy environment including EventContainer and all temporary variables
  • ErrorMessage
    A variable who’s value equals the concatenation of the generic policy exception information and the value of the variable passed as an argument

Implementation Example:

The following code exert shows a single logical exception handler created to handle Data Source lookup errors.

//exception handler
handle lookupFailure {

@EnrichFlag = 2;  #marked as failed
@AutomationResult = 0;  #marked as failed
@AutomationResultDetail = ErrorMessage;
orbdata_utils.logmsg("LOGICAL EXCEPTION",_serial,ErrorMessage);
ReturnEvent(EventContainer);
exit();

}

//extract from main policy code showing the call to exception handler
_inventoryArray = GetByFilter(‘INVENTORY’, “Node='” + @Node + “‘”,false);
if (Num <> 1) {
  msg = “Inventory lookup failed for Node=” + @Node;

raise lookupFailure(msg);

}


JAVA CLASS BASED EXCEPTION HANDLING

Similar to the Logical Exception Handler but in this case no Raise() function is required to call the handler. This exception handler is created using the Handle() function exactly as seen in the Logical Exception Handler, but the exception handler is not assigned a logical name chosen by the writer, instead it is assigned the java exception class name. The exception handler is called automatically when an exception defined by the associated java exception class is thrown.

The following exert from the PolicyLogger.log shows an unhandled exception caused by a failure to connect to a Data Source, in this case because the credentials are invalid..

Caused by: com.micromuse.common.parser.PolicyException: Unhandled Exception: [jcc][t4][2013][11249][3.57.82] Connection authorization failure occurred. Reason: User ID or Password invalid. ERRORCODE=-4214, SQLSTATE=28000 in policy: enrichment at line: 73.
Caused by: com.micromuse.common.datasource.SQLDataSourceException: com.micromuse.common.datasource.SQLDataSourceException: JDBC Connection Pool could not connect to datasource as: db2inst1 at jdbc:db2://impact61:50000/CUSTOMER

This exception message is used to create an exception handler designed to warning support staff of the error.

Conditions:

The exception handler must be inserted at the top of your policy before the main policy code.
The Java class specified must contain the entire classpath as shown below:

tick   

handle com.micromuse.common.datasource.SQLDataSourceException {…}

cross   

handle SQLDataSourceException {…}

Environment:

The environment in which the exception handler runs contains the following objects:

  • The policy environment
    The entire policy environment from which the exception occurred including EventContainer and all temporary variables
  • ExceptionMessage
    A variable holding the error message reported by the java exception class
  • ErrorMessage
    A variable holding the policy error including line number, etc concatenated with the ExceptionMessage

Example environment:

  1. ExceptionMessage  =  com.micromuse.common.datasource.SQLDataSourceException: com.micromuse.common.datasource.SQLDataSourceException: JDBC Connection Pool could not connect to datasource as: db2inst1 at jdbc:db2://impact61-test:50000/CUSTOMER
    ErrorMessage  =  Exception in policy: enrichment at line: 74. com.micromuse.common.datasource.SQLDataSourceException: JDBC Connection Pool could not connect to datasource as: db2inst1 at jdbc:db2://impact61-test:50000/CUSTOMER  

Implementation Example:

The following code exert shows an exception handler designed to trap and handle generic Data Source connection errors such as the one shown in the exception messages above. The java exception class used in this example will trap all generic Data Source connection errors such as Data Source unavailable, host resolution and invalid credentials.
The exception handler is designed to notify of Data Source errors in three ways:

    1. Update the alert in context, warning of an enrichment failure
    2. Create a new alert and send it to the ObjectServer to be handled as normal by operations
    3. Send an email to the support team
    4. Log the exception to the policy log for auditing purposes

handle com.micromuse.common.datasource.SQLDataSourceException {

//update the alert
@EnrichFlag = 2;  #marked as failed
@AutomationResult = 0; #marked as failed
@AutomationResultDetail = ErrorMessage;
ReturnEvent(EventContainer);
//create event to warn of failure
_DSErrorEvent = NewEvent("OrbDataEventReader");
_DSErrorEvent.EventReaderName = "OrbDataEventReader";
_DSErrorEvent.Identifer = "DSAConnectionError:enrichment:" + GetServerName()+ ;
_DSErrorEvent.Node = "Impact61:" + GetServerName();
_DSErrorEvent.AlertKey = "DSA Error:enrichment";
_DSErrorEvent.AlertGroup = GetServerName() + ":Impact DSA Error";
_DSErrorEvent.Severity = 5;
_DSErrorEvent.Summary = ExceptionMessage;
_DSErrorEvent.LastOccurrence = GetDate();
ReturnEvent(_DSErrorEvent);
//send email to support
sendemail("", sysman@demo.com,"Impact DSA connection error",ExceptionMessage, impact@demo.com,true);
//log error to policy log
orbdata_utils.logmsg("DSA EXCEPTION",_serial,ErrorMessage);
exit();

}

 


 

GENERAL/UNHANDLED EXCEPTIONS

Any exception not handled by a Policy Level exception handler is deemed ‘unhandled’. All unhandled exceptions are processed by the PolicyLogger service.
The PolicyLogger service logs all exception errors to the PolicyLogger.log in RED, clearly identifying exceptions and enabling developers to analyse exception conditions and create suitable exception handlers.

Logging of exception messages is fine but our goal is to actively capture and report error conditions..

No problem, the PolicyLogger service has a configuration option named “Error Handling Policy” allowing us to specify a policy to be executed each time an unhandled exception occurs.
As briefly described at the start of this article an unhandled exception will result in the termination of the current policy run. It is not possible to logically handle an error condition via an error handling policy and then continue processing of the original policy.

PolicyLogger-full

By default the policy configured is “DefaultExceptionHandler”, an extremely simple policy containing a single statment which logs the value of EventContainer to the PolicyLogger.log.

By creating your own policy designed to process known but ‘unhandled‘ exceptions, simple but effective management of unhandled exceptions can be accomplished.

The downside to handling errors in this way is that Impact still deems these errors ‘unhandled’ and therefore continues its default behaviour of logging RED exception messages to the PolicyLogger.log.

Error Handling Policy Environment:

As this exception handler is not specific to a single policy additional environment variables are made available to identify the source of incoming exceptions..

  • The policy environment
    The entire environment of the policy which threw the error including the EventContainer and all temporary variables
  • PolicyError
    A context holding data specific to the thrown exception. An example is shown below:

PolicyError = (

NumEventErrors=value,    – Number of exception errors which occured during policy execution

PolicyName=value,    – Name of Policy which threw the exception

PolicyErrorMessage=value    – Exception message

)

 

Implementation Example:

The following code sample shows a generic Error Handling Policy. This code exert shows 3 sections of code, each designed to handle generic exceptions..

  1. The first section simply logs details of all ‘unhandled’ exceptions to the PolicyLogger.log
  2. The second section is triggered if the incoming exception was caused by an exception in the enrichment policy. The exception handler will update the alert which was being processed by the enrichment policy at the point of the exception, updating alerts fields to reflect the error
  3. The third section is triggered by a Data Source connection failure due to incorrect credentials. If this exception is encountered a new alert is created and returned to the ObjectServer towarn of the failure
    Handling Data Source exceptions in this way negates the need for individual exception handlers to be added to each and every policy

 

//Log details of all exceptions to PolicyLogger.log
orbdata_UTILS.logmsg("Unhandled exception caught::" +
PolicyError.PolicyName + "::" + _serial + "::" +
PolicyError.PolicyErrorMessage);

 

//If exception source is enrichment policy then update alert being enriched with failure
if (PolicyError.PolicyName like 'enrichment') {

@EnrichFlag = 2;
@AutomationResult = 0;  #marked as failed
@AutomationResultDetail = PolicyError.PolicyErrorMessage;
ReturnEvent(EventContainer);

}

 

//Is this a DataSource error? If so send a warning event to ops!
if (PolicyError.PolicyErrorMessage like 'Connection authorization failure') {

//Create event to warn of failure
_DSErrorEvent = NewEvent("OrbDataEventReader");
_DSErrorEvent.EventReaderName = "OrbDataEventReader";
_DSErrorEvent.Identifer = "DSEventGeneralException" + GetServerName()+ PolicyError.PolicyName;
_DSErrorEvent.Node = "Impact61";
_DSErrorEvent.AlertKey = "DataSource Error:" + PolicyError.PolicyName;
_DSErrorEvent.AlertGroup = GetServerName() + ":Impact DSA Error";
_DSErrorEvent.Severity = 5;
_DSErrorEvent.Summary = PolicyError.PolicyErrorMessage;
_DSErrorEvent.LastOccurrence = GetDate();
ReturnEvent(_DSErrorEvent);

}

Visits: 968