Configuring transport module properties files

Transport properties files define how the gateway sends events using the transport module.

The following sections describe the properties files that the gateway can use to send events using the transport module:

The jmsTransport.properties file
The fileTransport.properties file
The mqttTransport.properties file
The httpTransport.properties file
The multiChannelHttpTransport.properties file
The pulsarTransport.properties file
The socketTransport.properties file
The scalaTransport.properties file

Note: You can configure the gateway to operate in one of two modes of operation: standard mode or IBM Operations Analytics - Log Analysis (LA) mode. The transport module properties files associated with gateways running in standard mode are jmsTransport.properties, fileTransport.properties, mqttTransport.properties, httpTransport.properties and socketTransport.properties. The transport module properties file associated with gateways running in LA mode is scalaTransport.properties.

Note: On Windows platforms, the delimiter in all the paths defined in the transport module properties files should be escaped, for example, if you are setting the path for the key store, the path should be defined as follows:

keyStore=C:\\IBM\\Tivoli\\Netcool\\omnibus\\java\\security\\client.jks

The jmsTransport.properties file

Table 1 describes the properties defined in the jmsTransport.properties file. which is the transport module properties file associated with the Java Message Service (JMS) transport module protocol.

Note: You configure the JMS-related properties if you set the Gate.XMLGateway.TransportType and Gate.XMLGateway.TransportFile properties to JMS and jmsTransport.properties in the G_XML.props properties file.

Table 1. JMS transport module properties
Property name	Description
initialContextFactory	Use this property to specify the context factory class name of the JMS provider. Note: The initialContextFactory is the initial context factory for the Java™ Naming Directory Interface (JNDI) provider being used. The default is the Websphere JNDI provider.
jmsFilter	Use this property to specify the expression for JMS servers for sending the intended messages or events to the probe.
jmsHeaderMap	Use this property to assign a value to the JMS header. Multiple instances of this property is supported. Configuration format: `jmsHeaderMap=<jms_header_name>:<value>` Supported JMS headers for sender update: `JMSCorrelationID` `JMSType` The assigned value can be static or dynamic. A static value is a string enclosed with single quotes, for example: `jmsHeaderMap=<jms_header_name>:'This is static value'` A dynamic value can be obtained from raw Netcool event data by configuring a data source in the property. The data source can be any mapped field name specified in the gateway’s map field, for example: `jmsHeaderMap=<jms_header_name>:MyNode` For details about using dynamic values, see Dynamic value assignment for jmsHeaderMap and jmsPropertyMap.
jmsPropertyMap	Use this property to assign a value to the JMS property. Multiple instances of this property is supported. Configuration format: `jmsHeaderMap=<jms_property_name>:<value>` JMS property names are customizable. The assigned value can be static or dynamic. A static value is a string enclosed with single quotes, for example: `jmsHeaderMap=MyJmsProperty:'This is static value'` A dynamic value can be obtained from raw Netcool event data by configuring a data source in the property. The data source can be any mapped field name specified in the gateway’s map field, for example: `jmsHeaderMap=jmsPropertyMap:MyNode` For details about using dynamic values, see Dynamic value assignment for jmsHeaderMap and jmsPropertyMap.
providerURL	Use this property to specify the URL of the JMS provider. You can specify the URL of the data store for the JNDI provider, or you can specify the path to a binding file. Note: If you specify a binding file, always create that file using the IBM® WebSphere® MQ JMS Administration tool. In addition, copy the WebSphere jar files to the $OMNIHOME/java directory. For information about creating binding files and JMS Administration tool jar files, see the IBM WebSphere MQ information center.
queueConnectionFactory	Use this property to specify the identifier of the queue connection factory.
topicConnectionFactory	Use this property to specify the identifier of the topic connection factory.
username	Use this property to specify the user name for the JMS connection.
password	Use this property to specify the password for the JMS connection.

Note: You must ensure that all property values in the transport properties file are unquoted; that is, contain no quotation marks.

The fileTransport.properties file

Table 2 describes the properties defined in the fileTransport.properties file. which is the transport module properties file associated with the Data File transport module protocol.

Note: You configure the data file-related properties if you set the Gate.XMLGateway.TransportType and Gate.XMLGateway.TransportFile properties to File and fileTransport.properties in the G_XML.props properties file.

Table 2. Data file transport module properties
Property name	Description
jsonFilename	Use this property to specify the full path to a JSON file to run through the gateway. The default is ${OMNIHOME}/var/jsonAlarm.txt.
streamFilename	Use this property to specify the full path to a stream capture file to run through the gateway. The default is ${OMNIHOME}/var/xmlAlarm.txt. Note: You can specify either the streamfilename property or the xmlfilename property but not both.
xmlFilename	Use this property to specify the full path to a standard stream of XML to run through the gateway. Note: You can specify either the streamfilename property or the xmlfilename property but not both.
sleepInterval	Use this property to specify the interval (in milliseconds) that the gateway waits between iterations when tailing the source file. The default is 1000 milliseconds.

Note: You must ensure that all property values in the transport properties file are unquoted; that is, contain no quotation marks.

The mqttTransport.properties file

Table 3 describes the properties defined in the mqttTransport.properties file, which is the transport module properties file associated with the MQTT transport module protocol.

Note: You configure the MQTT-related properties if you set the Gate.XMLGateway.TransportType and Gate.XMLGateway.TransportFile properties to MQTT and mqttTransport.properties in the G_XML.props properties file.

Table 3. MQTT transport module properties
Property name	Description
connectionURL	Use this property to specify the URL of the MQTT bus provider. The default is tcp://localhost:1883.
clientId	Use this property to specify an ID for each MQTT client connection to the MQTT provider. The clientId of each MQTT client connection must be unique. The default is MQTTClient.
topicName	Use this property to specify the name of a topic to subscribe to for messages. To subscribe to multiple topics, you can define multiple topic names. The default is "". Note: The MQTT wildcard character `#` can be used to define a wildcard set of topics to subscribe to. For example, a value of `#` would register interest in all topics known to the MQTT provider.
cleanStart	Use this property to specify whether the connection should perform a clean start. The default is false. If you specify a value of false, any message stored by the MQTT provider for a connection client ID will be returned on startup.
keepAlive	Use this property to specify the frequency (in seconds) with which the MQTT provider connection is polled during periods of inactivity. Polling keeps the connection alive and active. The default is 30.

Note: You must ensure that all property values in the transport properties file are unquoted; that is, contain no quotation marks.

The httpTransport.properties file

Table 4 describes the properties defined in the httpTransport.properties file, which is the transport module properties file associated with the HTTP and HTTPS transport module protocol.

Note: You configure the HTTP- and HTTPS-related properties if you set the Gate.XMLGateway.TransportType and Gate.XMLGateway.TransportFile properties to HTTP and httpTransport.properties in the G_XML.props properties file.

Table 4. HTTP/HTTPS transport module properties
Property name	Description
batchHeader	Use this property to specify the header for batch messages. A batch message consists of a group of Netcool events of which every event is originally an XML file, but each has been stripped of its XML declaration to ensure XML parsing at the endpoint. To complete the batched events as an XML file, batchHeader must contains the following: The XML declaration: `<?xml version="1.0" encoding="UTF-8">` An additional XML opening tag for enclosing the batched events. Note: batchFooter must have an XML closing tag corresponding to the opening tag specified within this property. The default is "".
batchFooter	Use this property to specify the footer for batch message. Note: batchFooter must have an XML closing tag corresponding to the opening tag specified within the batchHeader property. The default is "".
batchSeparator	Use this property to set the separator used between events in a batch. This property is usually set to a comma (`,`) for the batched JSON events use case, that is: when Gate.XMLGateway.TransformerInputType is `json` and batchSize is greater than `1`. The default is "".
bufferFlushTime	Use this property to specify the interval (in seconds) at which the gateway flushes the buffer. When the difference between the current time and the time at which the last event was put into the buffer is equal to or greater than the bufferFlushTime interval, all buffered events are batched up for HTTP posting. The default is 20.
bufferSize	Use this property to specify the maximum size that the gateway allows for the buffer. When the buffer size is reached, all events in the buffer are batched up for HTTP posting. The default is 10.
clientURL	Use this property to specify a URL definition for a HTTP/HTTPS destination. The default is "".
concurrentRequestThreads	Use this property to set the thread pool size for HTTP requests. The default is 1.
httpTimeout	Use this property to specify the time (in seconds) that the gateway allows for the HTTP connection. The default is 10.
retryLimit	Use this property to specify the maximum number of HTTP operation retries that the gateway can make. After a successful HTTP operation, the gateway makes no more retries. The default is 2.
retryWait	Use this property to specify the time (in seconds) that the gateway waits before retrying an HTTP operation . The default is 5.
threadPoolSize	Use this property to specify the number of threads that should be used to handle client connections to the internal HTTP/HTTPS server used by the transport module. The default is 10.
useNullHostnameVerifier	Use this property to disable hostname verification in the HTTP client against the web server’s SSL certificate. This unblocks the authentication exception caused by the hostname in a valid SSL certificate being different from the web server’s current hostname. The default is false.
username	Use this property to specify the username to use for basic authentication. Leave this property blank for no authentication. The username and password in the HTTP client has the following constraints: username and password cannot override the credentials embedded in the endpoint URL passed from the Transformer module. For any endpoint without prior configured credentials, the username and password will be used in `POST` operations, if these properties are configured. The default is "".
password	Use this property to specify the password to use in basic authentication. This property is ignored if username is blank. The password property can be set to the value generated from the encryption by ConfigCryptoAlg. For more information see Encrypting gateway and transport module properties The default is "".

Note: You must ensure that all property values in the transport properties file are unquoted; that is, contain no quotation marks.

HTTP operation retry

If the gateway fails to send out a message using the HTTP operation, it retries for the number of times specified by the retryLimit property. When the target returns a positive response, the gateway stops retrying.

The gateway waits for the number of seconds specified by the retryWait property between each HTTP operation retry.

When the gateway is in retry mode, if the condition for the next message sending is ripe (see the description for bufferSize and bufferFlushTime) the new message is put on hold until the retry of the previous message is completed. This prevents concurrent attempts of message sending which might disturb event ordering at the receiving end.

Event transformation and batching

The netcool event produced from the empty Transformer is in the following format:

<?xml version="1.0" encoding="UTF-8"?>
	<tns:netcoolEvent type="update" 
      xmlns:tns="http://item.tivoli.ibm.com/omnibus/netcool">
		<tns:netcoolField name="Serial" type="integer">1</tns:netcoolField>
		<tns:netcoolField name="Node" type="string">1.1.1.1</tns:netcoolField>
		<tns:netcoolField name="Acknowledged" type="integer">0</tns:netcoolField>
		<tns:netcoolField name="ServerName" type="string">NCOMS_P</tns:netcoolField>
		<tns:netcoolField name="ServerSerial" type="integer">1</tns:netcoolField>
	</tns:netcoolEvent>

To batch multiple Netcool events into XML compliant format, you must make some configuration changes in the Transformer Module and Transport Module.

Transformer Module

transformer.xsl

<tns:transformer name="netcoolEvents" type="northbound"  
     endpoint=“http://<ip>:<port>” 
     className="com.ibm.tivoli.netcool.integrations.transformer.XSLTTransformer">
<tns:property name="xsltFilename" type="java.lang.String" 
     value="${OMNIHOME}/java/conf/netcoolstripxmlheader.xsl" 
     description="XSLT file for converting Netcool events to NC events"/>
</tns:transformer>

This XLST definition strips the XML declaration from every Netcool event produced from the Transformer Module. The header-stripped event is then placed in the HTTP Transport’s buffer.

When the condition is ripe (refer to the bufferSize and bufferFlushTime properties), events in the buffer are batched up and enclosed by the string values specified by batchHeader and batchFooter.

<?xml version="1.0" encoding="UTF-8"?>
<tns:netcoolEventList  xmlns:tns="http://item.tivoli.ibm.com/omnibus/netcool">
	<tns:netcoolEvent type="update" 
       xmlns:tns="http://item.tivoli.ibm.com/omnibus/netcool">
		<tns:netcoolField name="Serial" type="integer">1</tns:netcoolField>
		<tns:netcoolField name="Node" type="string">1.1.1.1</tns:netcoolField>
		<tns:netcoolField name="Acknowledged" type="integer">0</tns:netcoolField>
		<tns:netcoolField name="ServerName" type="string">NCOMS_P</tns:netcoolField>
		<tns:netcoolField name="ServerSerial" type="integer">1</tns:netcoolField>
	</tns:netcoolEvent>
	<tns:netcoolEvent type="update" 
       xmlns:tns="http://item.tivoli.ibm.com/omnibus/netcool">
		<tns:netcoolField name="Serial" type="integer">2</tns:netcoolField>
		<tns:netcoolField name="Node" type="string">2.2.2.2</tns:netcoolField>
		<tns:netcoolField name="Acknowledged" type="integer">0</tns:netcoolField>
		<tns:netcoolField name="ServerName" type="string">NCOMS_P</tns:netcoolField>
		<tns:netcoolField name="ServerSerial" type="integer">2</tns:netcoolField>
	</tns:netcoolEvent>
</tns:netcoolEventList>

HTTP Transport configurations for event-format-oriented use cases

Table 5. Use Case 1: Gateway send batched events
Property name	Description
bufferSize	2 or above.
batchHeader	`<?xml version="1.0" encoding="UTF-8"?><tns:netcoolEventList xmlns:tns=http://item.tivoli.ibm.com/omnibus/netcool">`
batchFooter	`</tns:netcoolEventList>`

Use Case 2: Gateway send single event

Table 6. Use Case 2: Gateway send single event
Property name	Description
bufferSize	1
batchHeader	Leave empty.
batchFooter	Leave empty.

The multiChannelHttpTransport.properties file

Table 7 describes the properties defined in the multiChannelHttpTransport.properties file, which is the transport module properties file associated with the Multi-Channel HTTP transport module protocol.

Note: You configure the Multi-Channel HTTP-related properties if you set the Gate.XMLGateway.TransportType and Gate.XMLGateway.TransportFile properties to MultiChannelHttp and multiChannelHttpTransport.properties in the G_XML.props properties file.

Table 7. Multi-Channel HTTP transport module properties
Property name	Description
httpConnectionPropertiesFile	Use this property to specify the path to the transport properties file. The default is multiChannelHttpTransport.json.
ConfigCryptoAlg	Use this property to specify the encryption method to be used. The default is AES_FIPS.
ConfigKeyFile	Use this property to specify the path to the encryption key file. The default is aesfips.key.
MaxMsgSize	Use this property to specify the maximum size of the message payload in bytes. This is intended to limit the size of batched message events to comply with the requirements of the service endpoint. The default is 0 (which specifies no limit to the message body size).

For details about configuring the gateway to integrate with Microsoft Azure Event Hubs, see Integrating the gateway with Microsoft Azure Event Hubs.

Although this integration was developed for Azure Event Hubs, the functionality may also be applied to other similar integrations that uses the REST API.

Below is an example of the multiChannelHttpTransport.json configured for JSON batch events:

   "SEND":
   {       "NEW_EVENT":
      {
          "uri":"https://host:port/endpoint",
          "method":"POST",
          "headers":"Authorization=++Authorization++,Content-Type=application/json”,
          "content":"",
         "headerFile":"/opt/IBM/tivoli/netcool/omnibus/java/conf/newHeader.props",
         "headerFileRefresh":"60",
          "requireSSL":"true",
         "bufferSize":"1000",
         "bufferFlushTime":"45",
         "batchHeader":"[",
         "batchSeparator":",",
         "batchFooter":"]"
      }
}

The contents of newHeader.props:

Authorization=Bearer thisisthesecrettoken

If your service endpoint requires a certificate for authentication, you can import the certificate into the $NCHOME/platform/<platform>/jre64_<version>/jre/lib/security/cacerts.

The pulsarTransport.properties file

Table 8 describes the properties defined in the pulsarTransport.properties file, which is the transport module properties file associated with the Pulsar transport module protocol.

Note: You configure the Pulsar-related properties if you set the Gate.XMLGateway.TransportType and Gate.XMLGateway.TransportFile properties to PULSAR and pulsarTransport.properties in the G_XML.props properties file.

Table 8. Pulsar transport module properties
Property name	Description
ConnectionPropertiesFile	Use this property to specify the JSON file holding the Apache Pulsar connection properties.
PulsarClientMode	Use this property to specify whether the Pulsar client is running in `CONSUMER` or `PRODUCER` mode. Note: By default PulsarClientMode is commented out, so you must uncomment it to use it.

Note: You must ensure that all property values in the transport properties file are unquoted; that is, contain no quotation marks.

The socketTransport.properties file

Table 9 describes the properties defined in the socketTransport.properties file, which is the transport module properties file associated with the socket transport module protocol.

Note: You configure the socket-related properties if you set the Gate.XMLGateway.TransportType and Gate.XMLGateway.TransportFile properties to Socket and socketTransport.properties in the G_XML.props properties file.

Table 9. Socket transport module properties
Property name	Description
serverPort	Use this property to specify the port to which the events will be sent.
clientAddress	Use this property to specify the address to which the gateway connects. The format of the address is `host:port`.
subscribeMessages	This property is not used by the gateway.
subscribeResponses	Use this property to specify a list of response processors for responses to the subscribe messages.
disconnectMessages	Use this property to specify messages sent when disconnecting an asynchronous client.
disconnectResponses	Use this property to specify response processors for the disconnect messages.
activeAlarmsMessages	Use this property to specify messages sent when requesting active alarms as an asynchronous client.
messageTerminator	Use this property to specify a regex that should match the end of each alert from the target system.
readResponseAttempts	Use this property to specify how many times the probe should attempt to read a response to a message.
readerResponseDelay	Use this property to specify how long, in seconds, the probe should wait between attempts to read a response.
threadPoolSize	Use this property to specify the maximum number of threads to use to process connections made to the probe when it is running as a server.

Note: You must ensure that all property values in the transport properties file are unquoted; that is, contain no quotation marks.

The scalaTransport.properties file

Table 10 describes the properties defined in the scalaTransport.properties file, which is the transport module properties file associated with the SCALA transport module protocol.

Note: You configure the SCALA-related properties if you set the Gate.XMLGateway.TransportType and Gate.XMLGateway.TransportFile properties to SCALA and scalaTransport.properties in the G_SCALA.props properties file.

The gateway uses the Java Secure Socket Extension (JSSE) framework to implement secure transport over Secure Socket Layer (SSL) and Transport Layer Security (TLS). The JSSE framework hides the complexities of the underlying protocols associated with SSL and TLS. Thus, the following transport properties defined in the scalaTransport.properties file map directly to the JSSE properties: keyStore, keyStorePassword, trustStore, and trustStorePassword.

Table 10. Properties in the scalaTransport.properties file
Property name	Description
scalaURL	Use this property to specify the URL of the target data collector application to which the gateway connects. The default is http(s)://`some`.`host`.com:`port`/Unity/`DataCollector`.
scalaRetryMax	Use this property to specify the maximum number of attempts that the gateway makes to connect to the target data collector application before dropping the message. If you set this property to 0 (zero), the gateway tries to connect to the target data collector application indefinitely. The default is 0. Note: While the IBM Operations Analytics - Log Analysis data collector is not reachable, events are buffered in the gateway. This allows the gateway to handle temporary or intermittent connectivity problems to the IBM Operations Analytics - Log Analysis data collector. A prolonged period of connectivity loss potentially results in event loss due to the gateway's inability to buffer large numbers of events if the event rate is extremely high. In situations where the event rate is high, it is unlikely that the gateway can send all of the data and catch up once the IBM Operations Analytics - Log Analysis data collector is available.
scalaRetryPeriod	Use this property to specify the amount of time (in seconds) between each reconnection attempt to the target data collector application. The default is 30 seconds.
keyStore	Use this property to specify the location of the Java keystore file that contains the private keys for any HTTPS ports. See the JSSE for Java security property javax.net.ssl.keyStore. The default is $OMNIHOME/java/security/client.jks.
keyStorePassword	Use this property to specify the password to access the private keys for any HTTPS ports from the keystore file specified in the keyStore property. See the JSSE for Java security property javax.net.ssl.keyStorePassword. The default is password.
trustStore	Use this property to specify the location of the Java trust store file that contains the server's public key for any HTTPS clients. See the JSSE for Java security property javax.net.ssl.trustStore). The default is $OMNIHOME/java/security/cacerts.jks.
trustStorePassword	Use this property to specify the password to access the server's public key for any HTTPS clients from the trust store file specified in the trustStore property. See the JSSE for Java security property javax.net.ssl.trustStorePassword. The default is password.
threadPoolSize	Use this property to specify the number of threads that the HTTP servers share to process incoming requests. The default is 16 threads.
username	Use this property to specify the username to use for authentication with the target data collector application. The default is no value (that is, the property is blank). Leave this property blank for no authentication.
password	Use this property to specify the password to use for authentication with the target data collector application. The password property is ignored if the username property is left blank. The default is no value (that is, the property is blank).
eventBufferSize	Use this property to specify the maximum number of events to contain in each batch of log record data that the transport module sends to the target data collector application. The default is 200 events.
eventBufferFlushTime	Use this property to specify the amount of time (in seconds) to wait for new events before flushing the buffer. The flush timer is reset on each event added to the batch. The default is 30 seconds.
enableTrace	Use this property to enable diagnostic tracing of communications between the transport module and the target data collector application. The default is false (that is, disable diagnostic tracing).
jsonMsgHostname	Use this property to specify the hostname that corresponds to the data source for received data. This will be the name of the host where the gateway is running. The default is <local hostname>.
jsonMsgPath	Use this property to specify the path that corresponds to the data source for received data. The default is NCOMS.
readTimeout	Use this property to specify the length of time (in seconds) that the gateway allows for reading on a socket. The default is 30 seconds.

Note: You must ensure that all property values in the transport properties file are unquoted; that is, contain no quotation marks.