Resolving problems when using Kafka nodes

Use the advice given here to help you to avoid or resolve problems that might arise when you are using KafkaConsumer or KafkaProducer nodes.

Connecting to a Kafka cluster

Procedure

When you set the Kafka user ID and password by using the mqsisetdbparms command, ensure that you have used the form kafka::KAFKA::integrationServerName.
For more information, see Configuring security credentials for connecting to Kafka.
If you are connecting to IBM® Event Streams in IBM Cloud, ensure that the Security protocol property on the Kafka node is set to SASL_SSL. For more information about configuring the security credentials for connecting to Event Streams, see Using Kafka nodes with IBM Event Streams.

Using log4j to investigate connection and configuration problems

About this task

If you have a problem connecting to Kafka, and the problem persists after you have validated that the configuration of the nodes is correct, you might find more information by enabling tracing from the Kafka client.

The Kafka client uses the log4j component to capture trace information. The lib sub-directory of the IBM Integration Bus installation directory contains a log4j.properties file, which contains an example of how tracing can be enabled for the Kafka client:

 ############################################################################
  ## Kafka logger, un-comment the following section to write Kafka log messages
  ## to a file.
  ## The logging level, currently 'DEBUG' is the most detailed level of 
  ## logging available. The log4j.appender.kafkaAppender.Threshold value may
  ## be changed to one of 
  ##     DEBUG, TRACE, INFO, WARN, ERROR, FATAL 
  ## to reduce the information written to the log file. 
  ############################################################################
  log4j.logger.org.apache.kafka=DEBUG, kafkaAppender
  log4j.appender.kafkaAppender=org.apache.log4j.FileAppender
  log4j.appender.kafkaAppender.File=/tmp/kafkadebug.log
  log4j.appender.kafkaAppender.Threshold=DEBUG
  log4j.appender.kafkaAppender.Append=true
  log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
  log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

Procedure

Complete the following steps to enable tracing from the Kafka client:

Copy the log4j.properties file from the lib sub-directory of the IBM Integration Bus installation directory to the workpath of the integration server:
```
# cp install dir/server/lib/log4j.properties workpath/components/broker name/server uuid/log4j.properties
```
Alternatively, you can copy the log4j.properties file to the workpath for the integration node, to modify the level of tracing for all integration servers of the integration node.
In the file that you have copied, uncomment the kafkaAppender configuration, and modify the level of logging and the destination for the captured trace, as required.
Restart the integration server to start capturing logging information from the Kafka client.
Note: The default configuration places no limit on the size of the trace file captured, so ensure that you have sufficient capacity in the logging destination for the trace.