IBM Support

Troubleshooting Infosphere Information Server events with Apache Kafka

Troubleshooting


Problem

Events cannot be sent to or received from Apache Kafka topics, or the Infosphere Information Analyzer "Run analysis" does not work.

Symptom

1. Infosphere Information Server events cannot be sent to or received from Apache Kafka topics.

2. Infosphere Information Analyzer "Run analysis" option does not work when selected on a workspace data set..

3. Infosphere Information Analyzer cannot find assets.

4. Infosphere Information Server console.log takes a lot of space on system hard disk.

Cause

1. The Apache Zookeeper server is not referenced correctly.

2. The Apache Kafka server is stopped, or the Apache Zookeeper server is stopped.

3. A topic in the Apache Kafka server has been deleted or is temporarily not available.

Environment

Governance Rollup patch 1 for Information Server Version 11.5 as of February 2016, or

Information Server version 11.5.0.1 or later.

Beginning with Information Server version 11.7.0.1, this document does not apply if Enterprise Search is activated.

Diagnosing The Problem

Look the log file for one of the following errors:

      Kafka event could not be delivered: Failed to send messages after 3 tries.


      kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
      at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
      at kafka.producer.Producer.send(Producer.scala:77)
      at kafka.javaapi.producer.Producer.send(Producer.scala:33)
      ...
or
      Kafka event could not be delivered: replication factor: 1 larger than available brokers: 0
      kafka.admin.AdminOperationException: replication factor: 1 larger than available brokers: 0
      at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:70)
      at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:171)
      at kafka.admin.AdminUtils.createTopic(AdminUtils.scala)
      ...
or
      SLF4J: Failed toString() invocation on an object of type [org.apache.kafka.common.Cluster]
      [...] 000000bb SystemErr R java.lang.NullPointerException
      [...] 000000bb SystemErr R at org.apache.kafka.common.PartitionInfo.toString(PartitionInfo. java:72)
or
      java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

The log file is located here:

Information Server using WebSphere full profile:


    AIX or Linux:
    /opt/IBM/WebSphere/AppServer/profiles/InfoSphere/logs/server1/SystemOut.log
    Windows:
    C:\IBM\WebSphere\AppServer\profiles\InfoSphere\logs\server1\SystemOut.log

Information Server using WebSphere Liberty profile:

    AIX or Linux:
    /opt/IBM/InformationServer/wlp/usr/servers/iis/logs/messages.log
    Windows:
    C:\IBM\InformationServer\wlp\usr\servers\iis\logs\messages.log

On Windows, in the console log file, look for the following error:

      org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'coordinator': Error reading field 'host': Error reading string of length 25455, only 21 bytes available
          at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)

The console log file is located here:

Information Server using WebSphere full profile:

Windows:
C:\IBM\InformationServer\wlp\usr\servers\mis\logs\console.log

Information Server using WebSphere Liberty profile:

Windows:
C:\IBM\InformationServer\wlp\usr\servers\iis\logs\console.log

and

C:\IBM\InformationServer\wlp\usr\servers\mis\logs\console.log

Resolving The Problem

    1. Check that the Apache Zookeeper server is referenced correctly in Infosphere Information Server. Get the server hostname and port number as follows, provided that you installed Information Server under the default path.
      AIX or Linux:
      /opt/IBM/InformationServer/ASBServer/bin/iisAdmin.sh -d -k com.ibm.iis.sdp.zookeeper.connect
      Windows:
      C:\IBM\InformationServer\ASBServer\bin\iisAdmin.bat -d -k com.ibm.iis.sdp.zookeeper.connect
      Check that the hostname and the port number are correct and correspond to your installed Shared Open Source patch.
    2. Restart all servers of the Shared Open Source patch.
      1. If you have version 11.5.0.1 with Governance Rollup Patch 2 or later:
        AIX:
        /opt/IBM/InformationServer/shared-open-source/bin/stop-aix-services.sh
        /opt/IBM/InformationServer/shared-open-source/bin/start-aix-services.sh
        Linux:
        /opt/IBM/InformationServer/shared-open-source/bin/stop-linux-services.sh
        /opt/IBM/InformationServer/shared-open-source/bin/start-linux-services.sh
        Windows:
        C:\IBM\InformationServer\shared-open-source\bin\stop-windows-services.bat
        C:\IBM\InformationServer\shared-open-source\bin\start-windows-services.bat
      2. If you have version 11.5 with Governance Rollup Patch 1 or version 11.5.0.1 without Governance Rollup Patch 2:
        AIX or Linux:
        /opt/IBM/InformationServer/shared-open-source/bin/stop_unix_services.sh
        /opt/IBM/InformationServer/shared-open-source/bin/start_unix_services.sh
        Windows:
        C:\IBM\InformationServer\shared-open-source\bin\stop_windows_services.bat
        C:\IBM\InformationServer\shared-open-source\bin\start_windows_services.bat

        Check that they are up and running with the following commands.
        AIX:
        lssrc -g InfoSrv
        Linux:
        service InfoSrvZookeeper status
        service InfoSrvKafka status
        service InfoSrvSolrCloud status
        Windows:
        wmic service where ( name like "InfoSrv%" ) get name, state, processId
    3. If the issue still occurs, delete the Kafka topics. They will be recreated automatically. Follow these steps:
      1. Start the zookeeper client command line application with the <hostname> and <port number> got above in Section 1:
        AIX or Linux:
        /opt/IBM/InformationServer/shared-open-source/zookeeper/install/bin/zkCli.sh -server <hostname>:<port number>
        Windows:
        C:\IBM\InformationServer\shared-open-source\zookeeper\install\bin\zkCli.cmd -server <hostname>:<port number>

        On the command prompt, enter following commands:
        rmr /brokers/topics
        rmr /config/topics
        quit
      2. Stop all servers of the Shared Open Source patch as explained above in Section 2.
      3. Delete the topics directories created by kafka.
        1. If you have version 11.5.0.1 with Governance Rollup Patch 2 or later:
          AIX or Linux:
          rm -rf /opt/IBM/InformationServer/shared-open-source/kafka/data/1/*
          Windows:
          rmdir /S /Q C:\IBM\InformationServer\shared-open-source\kafka\data\1
          mkdir C:\IBM\InformationServer\shared-open-source\kafka\data\1
        2. If you have version 11.5 with Governance Rollup Patch 1 or version 11.5.0.1 without Governance Rollup Patch 2:
          AIX or Linux:
          rm -rf /opt/IBM/InformationServer/shared-open-source/kafka/logs/*
          Windows:
          rmdir /S /Q C:\IBM\InformationServer\shared-open-source\kafka\logs
          mkdir C:\IBM\InformationServer\shared-open-source\kafka\logs
      4. Start again all servers of the Shared Open Source patch as explained above in Section 2.

[{"Product":{"code":"SSZJPZ","label":"IBM InfoSphere Information Server"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"IBM Stewardship Center","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"11.5.0.2;11.5.0.1;11.5;11.7.0.0;11.7;11.7.0.1","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}},{"Product":{"code":"SSZJLG","label":"InfoSphere Information Analyzer"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"","label":""}},{"Product":{"code":"SSZJPZ","label":"IBM InfoSphere Information Server"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":" ","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
16 June 2018

UID

swg21977649

Page Feedback