Resolving problems when starting an integration node

Use the advice given here to help you to resolve common problems that can arise when you start an integration node.

About this task

When you start an integration node by using the mqsistart command, the mqsicvp command is run automatically to check that the environment is set up correctly (for example, the installed level of Java™ is supported). On Linux® and UNIX, the mqsicvp command also verifies that the ODBC environment (if specified) is configured correctly. For more information, see mqsicvp command.

For advice about specific problems that can occur when you start an integration node, see the following sections.

The integration node fails to start because there is not enough space in the Java TMPDIR directory or access permissions for the Java TMPDIR directory are inadequate

Procedure

  • Scenario: The integration node fails to start and either an error message indicates that insufficient space is available or a BIP4512 exception indicates a java.lang.NoClassDefFoundError in the stack trace.
  • Explanation: This error has two possible causes:
    • The integration node uses Java JAR files. When the integration node starts, the Java runtime environment extracts the JAR files into a temporary directory, Java TMPDIR. On Linux, UNIX, and z/OS® computers, the TMPDIR directory is typically /tmp; on Windows computers, it is c:\temp. If this directory is not large enough to hold the JAR files, the integration node does not start.
    • If the program is packaged as a PAR file, the user must have access to the system temporary directory and adequate space must be available in the temporary directory.
  • Solution: Use one of the following methods to specify the location of this temporary JAR directory:
    • Use the environment variable TMPDIR.
    • Set the system property java.io.tmpdir.
    Allow at least 50 MB of space per integration server in this directory for IBM® Integration Bus components. You might need more space if you deploy large user-defined nodes or other JARs to the integration node. You should ensure that all the dependencies of the compute node class are deployed to the integration node.

Diagnostic message ICH408I is issued on z/OS when your integration node fails to start

About this task

Two scenarios are described here. Choose the appropriate one.

Procedure

  • Scenario 1: The following diagnostic message is written to the SDSF SYSLOG on z/OS when your integration node fails to start: 
     ICH408I USER(MA10USR ) GROUP(TSOUSER ) NAME(OTHER, A N (ANO) 484     
   /argo/MA10BRK/ENVFILE                                          
 -                                         --TIMINGS (MINS.)--             
  ----PAGING COUNTS---                                                     
 -JOBNAME  STEPNAME PROCSTEP    RC   EXCP    CPU    SRB  CLOCK   SERV  PG  
    PAGE   SWAP    VIO SWAPS                                               
   CL(DIRSRCH ) FID(01D7D3E2E3F1F9002D08000000000003)                      
   INSUFFICIENT AUTHORITY TO LOOKUP                                        
   ACCESS INTENT(--X)  ACCESS ALLOWED(OTHER ---)
  • Explanation: The started task ID, under which the integration node runs, must be in a RACF® or z/OS UNIX System Services (USS) group that has rwx permissions on the integration node directory. For example, consider an integration node that is created under directory /argo/MA00BRK. It runs under started task ID MA00USR. Issuing the ls -al command from the root directory / to find the permission bit settings on /argo returns: 
    drwxrwx---  5 BPXROOT  ARGOUSR      8192 Jul 30  13:57  argo
    If you issue the id MA00USR command to find the group membership of started task ID MA00USR you see: 
    uid=14938(MA00USR) gid=5(TSOUSER) groups=229(ARGOUSR)
    These results show that the started task ID MA00USR potentially has rwx permissions on subdirectories to /argo, because these permissions are set for both the user and the group that is associated with MA00USR. If the permissions are not set correctly, you see the type of diagnostic message shown in the scenario.
  • Solution: Make sure that the started task ID, under which the integration node runs, is in a RACF or USS group that has rwx permissions on the integration node directory.
  • Scenario 2: The following diagnostic message is written to the SDSF SYSLOG on z/OS when your integration node fails to start: 
           ICH408I USER(MA10USR ) GROUP(WMQIBRKS  ) NAME(MA10USR                 
         CSFRNG CL(CSFSERV )                                               
         INSUFFICIENT ACCESS AUTHORITY                                     
         FROM ** (G)                                                       
         ACCESS INTENT(READ   )  ACCESS ALLOWED(NONE   )
  • Explanation: During the integration node start up process, Java accesses a secure random number generator. Consequently, the integration node started task ID needs access to the CSFRNG resource in the CSFSERV class.
  • Solution: Make sure that the started task ID that the integration node runs under has access to the CSFRNG resource in the CSFSERV class.

Abend code 047 is issued with a diagnostic message

Procedure

  • Scenario: On z/OS, your integration node generates abend code 047 when you try to start it, and the following diagnostic message is written to the SDSF SYSLOG: 
    IEA995I SYMPTOM DUMP OUTPUT 463                                    
SYSTEM COMPLETION CODE=047                                         
 TIME=10.53.47  SEQ=00419  CPU=0000  ASID=008E                     
 PSW AT TIME OF ERROR  078D0000   98D09E52  ILC 2  INTC 6B         
   ACTIVE LOAD MODULE           ADDRESS=18D08828  OFFSET=0000162A  
   NAME=SPECIALNAME                                                
        61819987 968995A2 A3618499 89A58599 */argoinst/driver*     
        F1F46D82 96858261 A4A29961 93979761 *14_boeb/usr/lpp/*     
        A6949889 61828995 61828997 89948189 *wmqi/bin/bipimai*     
        95                                  *n               *     
   DATA AT PSW  18D09E4C - 58109948  0A6B5820  B8E95020            
   GPR  0-3  00000000  0000003C  00000000  00000000                
   GPR  4-7  18D10300  18D115F0  00000013  00000004                
   GPR  8-11 18D111CF  18D101D0  18D0BBBE  18D0ABBF                
   GPR 12-15 98D09BC0  18D101D0  98D09E22  00000000                
 END OF SYMPTOM DUMP
  • Explanation: System completion code 047 means that an unauthorized program issued a restricted Supervisor Call (SVC) instruction. The diagnostic message also indicates that the program in error was bipimain.
  • Solution: When you install IBM Integration Bus, issue the command extattr +a bipimain from the bin directory of the installation path to give APF authorization to program bipimain.

Error message BIP2228 is issued when you try to start a second integration node on Linux or UNIX

Procedure

  • Scenario: Error message BIP2228, which mentions semctl in the syslog, is displayed when you try to start a second integration node on Linux or UNIX.
  • Explanation: This error typically indicates a permissions problem with a semaphore used by IBM Integration Bus. A semaphore is created when the first integration node starts after a reboot (or after an initial installation), and only members of the primary group of the semaphore creator can access this semaphore. This problem is a consequence of the UNIX System V IPC primitives that are used by IBM Integration Bus.

    The BIP2228 message is logged by any integration node that is started by a user who is not a member of the primary group of the semaphore creator. The integration node tries to access the semaphore, but fails with a permissions-related error. The integration node then terminates with the BIP2228 message.

  • Solution: Avoid this problem by ensuring that all user IDs used to start IBM Integration Bus have the same primary group. If this action is impractical, ensure that all user IDs are members of primary groups of all other user IDs. Contact your IBM Support Center for further assistance.

MQIsdp client connection is refused by the integration node

Procedure

  • Scenario: When a new MQIsdp client tries to connect to the integration node, its connection is refused.
  • Explanation: MQIsdp Client ID fields must be unique. If a client sends a CONN packet that contains the same Client ID as a currently connected client, the behavior is undefined.
  • Solution: Ensure that Client IDs are unique.

Error messages BIP2604 and BIP2624 are issued when you start an integration node or a new message flow

Procedure

  • Scenario: The following messages are written to the USS syslog on z/OS when your integration server, or a newly deployed or started message flow, fails to start: 
    (PMQ1BRK.default)[8]BIP2624E: Unable to connect to queue manager 'PMQ5':
        MQCC=2; MQRC=2025; message flow node 'ComIbmMQConnectionManager' 
(PMQ1BRK.default)[8]BIP2604E: Node failed to open Websphere MQ queue
       'INPUT1' owned by queue manager 'PMQ5': completion code 2; reason code 2025
  • Explanation: The WebSphere® MQ return code of 2025 indicates that the maximum number of concurrent connections has been exceeded. On z/OS, a typical cause of this problem is the setting for IDBACK in the WebSphere MQ CSQ6SYSP macro.
  • Solution: See the z/OS System Setup Guide section of the WebSphere MQ Version 7.5 product documentation online for information about setting the IDBACK variable.

When you start the integration node through the DataFlowEngine, it cycles continually

Procedure

  • Scenario: When you start the integration node through the DataFlowEngine, it continually cycles, starts and stops, and errors BIP2801 and BIP2110 appear in the log: 
    Unable to load implementation file '/opt/IBM/DistHu b/v2/lib/libdhbNBIO.so',  rc=The file access permissions do not allow the specified action.
IBMibm Integration Bus internal program error.
  • Explanation: The permissions on /opt/IBM have a value of 700, meaning that the integration node service user ID cannot read the disthub files.
  • Solution: Set the permissions on /opt/IBM to 755, which is rwxr-xr-x.

You have changed your logon password and cannot start your integration node on Windows

Procedure

  • Scenario: You have changed your logon password on Windows, and when you start the integration node, you see the following error message: 
    BIP8026E: It was not possible to start the component.
The component could not be started using the service user ID that was supplied when 
the component was created. Ensure that the service user ID and password are still 
valid. Ensure that the service user ID has permission to access all of the products 
directories, specifically the 'bin' and 'log' directories.  Check for system 
messages (on Windows this would be the application event log).
  • Solution: Change properties of your integration node by completing the following steps:
    1. Change your integration node by using the command: 
       mqsichangebroker integrationNodeName -i ServiceUserID -a ServicePassword
      For example, to change your logon password to user1pwd for user ID user1 on the integration node called WBRK_BROKER, use the following command: 
      mqsichangebroker WBRK_BROKER -i user1 -a user1pwd
    2. Restart your integration node.

The Java installation is at an incorrect level

Procedure

  • Scenario: You issue the command to start an integration node , but the integration node does not start and the system log includes message BIP8892, for example: 
    Verification failed. The installed Java level 1.3.2 does not 
   meet the required Java level 1.5.
  • Explanation: The command performs a check on the level of the Java product installed on the computer to ensure that the Java product is at the required level.
    The check has failed, therefore the integration node is not started.
  • Solution:
    • On distributed systems, you must use the JVM that is supplied with IBM Integration Bus; no other Java product is supported. Check that you have run mqsiprofile, or (on Windows only) that you issued the mqsistart command from the correct command console.

      If you ran the profile command, check the settings of the environment variables in mqsiprofile; MQSI_JREPATH, PATH, and the appropriate library path environment variables for your operating system. Change these settings to point to the integrated JVM, and ensure that no other Java installation is in the path.

    • On z/OS, install the correct level of Java that is reported in this message, update the integration node profile BIPBPROF and submit BIPGEN to refresh the component ENVFILE.

Authorization errors are reported on z/OS

Procedure

  • Scenario: You issue the command to start an integration node, but the component does not start and the JOBLOG includes message BIP8903, for example: 
    Verification failed. The APF Authorization check failed 
   for file '/usr/lpp/mqsi/bin/mqsireadlog'. 
WebSphere Message Broker requires that only bipimain is APF Authorized 
   for successful operation. File '/usr/lpp/mqsi/bin/mqsireadlog' 
   fails that requirement. 
If the file indicated in the message is bipimain, use the USS command 
   extattr to ensure that it is APF Authorized. 
If the file indicated in the message is not bipimain, use the USS 
   command extattr to ensure that it is not APF Authorized. 
For more information, search the product documentation for "APF attributes".
  • Explanation: When you start an integration node, a series of checks is made to ensure that the environment is set up correctly.
    One or more of the checks for the integration node identified in the message has failed, therefore the integration node is not started. The errors shown here indicate that the authorizations for some files are incorrect and incompatible with integration node operation.

    The integration node requires that a single file, bipimain, is APF authorized, but the checks indicate that the authorization for file mqsireadlog is incorrect.

  • Solution: The file bipimain must be APF authorized, and all other files must not have that authorization.
    Make the required changes to authorization for the files that are identified in the error messages and try the operation again.

Error message BIP8875 is issued when you start an integration node

Procedure

  • Scenario: You issue the mqsistart command to start an integration node, but the integration node does not start and the system log shows message BIP8875, for example: 
    The component verification for IBNODE has finished, 
but one or more checks failed.
  • Explanation: The command performs a series of checks to ensure that the integration node environment, WebSphere MQ queues, and Java are correct and accessible.
    One or more of the checks for the integration node identified in the message has failed, therefore the integration node is not started.
  • Solution: Look in the system log, or in the Application log in the Event Viewer on Windows.
    Additional messages have been written before this message to indicate which checks have failed. All the checks are performed every time you issue mqsistart, therefore all errors are included in the log. Some messages are also returned when you run the command from the command line.

    Investigate the one or more errors that have been reported and check return codes and additional details. Look at the complete message content to check for typical causes of the error, and follow the advice given for the messages that you see in the log. View the complete message text in the Diagnostic Messages reference topics.

    For example, you might see one or more of the following messages:

    • BIP8875W: The component verification for 'component_name' has finished, but one or more checks failed.
    • BIP8877W: The environment verification for component 'component_name' has finished, but one or more checks failed.
    • BIP8883W: The WebSphere MQ verification for component 'component_name' has finished, but one or more checks failed.
    • BIP8885E: Verification failed. Failed to connect to queue manager 'queue_manager_name'. MQRC: return_code MQCC: completion_code
    • BIP8887E: Verification failed for queue 'queue_name' on queue manager 'queue_manager_name' while issuing 'operation'. MQRC: return_code MQCC: completion_code
    • BIP8888E: Verification failed. Failed to disconnect from queue manager 'queue_manager_name'. MQRC: return_code MQCC: completion_code
    • BIP8892E: Verification failed. The installed Java level 'level_installed' does not meet the required Java level 'level_supported'.
    • BIP8893E: Verification failed for environment variable 'variable_name'. Unable to access file 'file_name' with user ID 'user_ID'. Additional information for IBM support: data1 data2.
    • BIP8895E: Verification failed. Environment variable 'variable_name' is incorrect or missing.
    • BIP8896E: Verification failed. Unable to access the registry with user ID 'user_ID'. Additional information for IBM support: data1 data2
    • BIP8897E: Verification failed. Environment variable 'variable_name' does not match the component name 'component_name'.
    • BIP8903E: Verification failed. The APF Authorization check failed for file 'file_name'.
    • BIP8904E: Verification failed. Failed to start file 'file_name' with return code 'return_code' and errno 'error_number'.

    If you cannot resolve the problems that are reported, and you receive a message such as BIP8893 that includes additional information, include these items in the information that you provide when you contact IBM Service.

Warning messages BIP8288-BIP8297 are shown in the syslog when you start an integration node

Procedure

  • Scenario: Warning messages BIP8288-BIP8297 are shown in the syslog when you start an integration node on Linux and UNIX systems.
  • Explanation: One or more problems were detected with the ODBC environment on Linux and UNIX systems.
    When you start an integration node by using the mqsistart command, the mqsicvp command is run automatically to check that the integration node environment is set up correctly. On Linux and UNIX systems, this command also verifies that the ODBC environment is configured correctly. Warning messages are written to the syslog in the following situations:
    • The file that is referenced by the ODBCINI environment variable does not exist, or the integration node does not have access to read or write to the file.
    • The ODBCSYSINI environment variable is not set.
    • The directory that is referenced by the ODBCSYSINI environment variable does not contain a file called odbcinst.ini, or the integration node does not have access to read or write to the file.
    • The IE02_PATH is not set.
  • Solution: Examine the warning messages in the syslog. To view more information, run the mqsicvp command from the command line.

Integration node startup on z/OS is very slow

Procedure

  • Scenario: The integration node startup on z/OS takes many minutes, with an extended time taken in loading the imbjplug2 .lil file.
  • Explanation: When IBM Integration Bus is run in a shared file system sysplex environment, the LPAR that the integration node started in does not necessarily own the file system mount points that the integration node uses. In this scenario all file accesses have to pass through the coupling facility, which adversely affects performance. During startup the integration node accesses and reads many files, and loads many Java class files. All these file operations are slowed, causing longer startup times.
  • Solution: For optimal startup performance, mount the directories that the integration node accesses in the LPAR in which the integration node is started. In particular, mount the IBM Integration Bus installation directories locally. Mounting file systems lists the directories that IBM Integration Bus on z/OS needs on the file system at run time; mount them locally for optimal performance.

Error messages AMQ7626 and BIP8048 are displayed when you try to start an integration node

Procedure

  • Explanation: You see these messages when using an integration node on a queue manager that is configured for global coordination with Oracle.
  • Solution: Start the queue manager manually with the -si flag before starting the integration node.

Error messages BIP8048 and BIP8059 are displayed when you try to start an integration node

Procedure

  • Explanation: You see these messages when starting an integration node that does not have an associated queue manager, or the queue manager is not started.
  • Solution: If you do not have a queue manager associated with the integration node, create it. Start the queue manager by using the strmqm command, and then start the integration node.