IBM Support

Troubleshooting problems with creating InfoSphere DataStage projects

Troubleshooting


Problem

This document describes common problems and useful troubleshooting techniques when the creation of a new InfoSphere DataStage project fails.

Symptom

Creating a project fails during the installation of InfoSphere Information Server, when using the Administrator client, or when using the dsadmin command.

Cause

There can be various causes of these types of problems, such as environmental issues, firewall configuration and repository database configuration.

Diagnosing The Problem

When project creation fails, the first step is to locate the error message. The error message location is based on how the project creation was started:

  • Installation - search in the installation logs (for example, /opt/IBM/InformationServer/logs/) for "RUN BP DSR_QUICKADD.B". The error message is a few lines following this text.
  • Administrator client - the error message is displayed in a message box.
  • dsadmin command - the error message is displayed on the console.

After you locate the error message, there might be more detailed error information in the following log file on the DataStage Server system:

/home/_Credential_Mapped_Username_/ds_logs/dstage_wrapper_trace_N.log
  • where N is a number from 1 to 20 for the log files for the last 20 DataStage sessions
  • where _Credential_Mapped_Username_ is the DataStage Server user name that credentials are mapped to in Web Administration console

Note: The logs go to ${user.home}/ds_logs where ${user.home} is the home directory of the credential mapped user or the user running the installation for projects created at installation time. On Windows computers, the user home directory is usually C:\Documents and Settings\_Credential_Mapped_UserName_

Tracing DataStage Server processes (IBM support and advanced users only)
Sometimes tracing the dsrpcd and child processes can produce useful information from which IBM support can narrow down the causes of a project creation problem. Operating systems differ in how to trace processes but a few examples are shown here:

AIX: "truss -o /tmp/dsrpcd_truss.out -fp <dsrpcdPID>"
RedHat: "strace -fp <dsrpcdPID> > /tmp/dsrpcd_strace.out"

The above commands attach to the running dsrpcd process and will record all of the system calls made by that process and its children during subsequent client-server sessions; for example creating a project from the Administrator client or the dsadmin command line.

Enabling JVM startup tracing (IBM support and advanced users only)
To produce extra diagnostic information for the JVM initialization (after all of its libraries have been successfully loaded), you can add the following lines to /opt/IBM/InformationServer/Server/DSEngine/dsenv:

XMOG_TRACE_LEVEL=TraceVerbose
XMOG_TRACE_FILE=/tmp/xmogtrace.txt
export XMOG_TRACE_LEVEL
export XMOG_TRACE_FILE

On Windows these tracing options can be set as System Environment Variables by using the System Control panel.

Remember to restart the DataStage Server engine processes after adding these variables, and to remove these environment variables after they are no longer needed.

Enabling repository database tracing (IBM support and advanced users only)
To enable tracing of the code that populates the repository database follow these steps:
  1. Create a file on the DataStage Server system in /opt/IBM/InformationServer/ASBNode/conf/ called NewRepos.debug.properties (the file name is case sensitive)
  2. In the file add the following three lines:
    log4j.logger.com.ascential.dstage=DEBUG
    log4j.logger.com.ibm.datastage=DEBUG
    NewRepos.spy.trace=true

The dstage_wrapper_trace_N.log will then contain extra tracing information the next time a project creation is attempted. Ensure that you delete the NewRepos.debug.properties file when finished. In addition, spy trace files, such as dstage_wrapper_spy_N.log, are produced in the same directory as the log files. These files contain a detailed record of low level method calls and can grow quite large.

Running project creation manually (IBM support and advanced users only)
The project creation code runs in the context of a dsapi_slave process which does not have any console output. Locate the full "RUN BP DSR_QUICKADD.B" command line from the domain installation log files on /opt/IBM/InformationServer/logs/.

Use the following commands to run the project creation code so that you can view the console output:
    Linux and UNIX
    1. cd /opt/IBM/InformationServer/Server/DSEngine
    2. . ./dsenv
    3. bin/uvsh
    4. RUN BP DSR_QUICKADD.B <arguments from log file> <newProjectName> /opt/IBM/InformationServer/Server/Projects/<newProjectName> CREATE
    5. QUIT

    Windows
    1. cd C:\IBM\InformationServer\Server\DSEngine
    2. bin\uvsh
    3. RUN BP DSR_QUICKADD.B <arguments from log file> <newProjectName>  C:\IBM\InformationServer\Server\Projects\<newProjectName> CREATE
    4. QUIT
Note that running the project creation manually may leave the repository contents relating to <newProjectName> inconsistent between DSEngine and the metadata repository, if the project creation failed. A manual cleanup of these two environments may be therefore be necessary, in which case IBM support should be contacted.

Resolving The Problem

The dstage_wrapper_trace_N.log usually gives an indication of where the problem lies. These log files might require IBM customer support, but some errors can be interpreted to attempt further diagnostics.

In some cases the error message explicitly says what the problem is, for example: "There is a mis-match between the code on the client and the code on the server. Please ensure that patch JRXXXXX (or later) is installed on the client machine."

The following is a summary of the known issues that can cause project creation to fail.

1) Incorrectly configured repository database.


    8.0.x message: "Error creating DR elements, Error was -1", "Invalid node name: %1"

    8.1 and later message: "DSR.ADMIN: Error creating DR elements, Error was Unique constraint violation."

    These types of errors usually occur because the repository database returns an error when attempting to make an update. The dstage_wrapper_trace_N.log file may contain more specific details about the exact database error.

    There may also be a database log, depending on what type of database the repository is running in, which contains more information. For example, DB2 has the db2diag tool which can be run to find out the exact reason why an update failed. Typical failures are: out of disk space, memory configuration problems etc.

    For repository database errors it is important to confirm that the database has been created using the scripts supplied on the installation media. These scripts configure important database parameters which if missed may cause project creation problems.

    It is also important that the database has been created using the correct character set, as per the database creation script documentation on the installation media (typically UTF16/32). If a different character set was used some of the metadata stored can become corrupted or may cause unexpected primary key violations. If the wrong character set has been used the product needs to be reinstalled.

    For errors at this level the WebSphere Application Server logs may contain additional information. The files SystemOut.log and SystemErr.log can be found in:

    ...WebSphere/AppServer/profiles/_profile_name_/logs/server1/


2) Leftover metadata in repository database from a previously failed project creation

    8.0.x message: "Error creating DR elements, Error was -1"

    This problem only occurs on 8.0.x systems and can be identified by looking in the dstage_wrapper_trace_N.log file for a "unique constraint violation" error. This can occur when a project creation failed and did not remove all of its metadata from the repository. Even though the project cannot be seen in DataStage, attempting to create a new project of the same name will result in this error.

    To work around this problem, you can simply create a project with a different name. Alternatively, IBM support can provide a tool and instructions for how to remove the leftover data from the repository.


3) Unable to create log file on the DataStage Server


    8.0.x message: "Error creating DR elements, Error was -1"

    8.1 and later message: "DSR.ADMIN: Error creating DR elements, Error was log4j:ERROR setFile(null,true) call failed."

    Just before the metadata repository is populated with the default project contents, a log file is created on the DataStage Server system in /home/_Credential_Mapped_Username_/ds_logs/. If this log file cannot be created, the project creation will fail. On Windows computers, the user home directory is usually C:\Documents and Settings\_Credential_Mapped_UserName_

    The usual reasons why this log could not be created are either because the user has no home directory at all or they do not have appropriate permissions on it.


4) Incorrectly configured locale on the DataStage Server


    8.0.x message: "Error creating DR elements, Error was -1"

    8.1 and later message: "DSR.ADMIN: Error creating DR elements, Error was Unmatched quotation marks"

    This problem is ultimately caused by bad locale configuration on the DataStage Server system. This manifests itself because the "hostname" command is run during project creation, and instead of returning the correct host name it returns a string such as "couldn't set locale correctly".


5) Failed to load JVM into the DataStage Server process (dsapi_slave)


    8.0.x message: "(The connection is broken (81002))"

    8.1 and later message: "CDSRUserAccess.EnterCS: Internal error - Failed to open file : UV.ACCOUNT (81002)"

    The JVM (Java Virtual Machine) can fail to load for several reasons. If it does fail to load, the dsapi_slave process will be terminated, resulting in connection broken errors (81002) being produced on the client.

    A core file may be produced which can be used to determine what caused the process to be terminated.

    Possible causes of this problem are:

    • The LIBPATH (or equivalent) is too long and caused a buffer overflow. This can be confirmed by using the Administrator client to execute the "env" command using the Command button. If the contents of LIBPATH appear to have been duplicated then it is probable that dsenv has been sourced twice (dsenv does not need to be sourced when starting the DataStage Server engine processes via uv -admin -start).
    • Incompatible or missed patches on the Client, Server and Domain systems. By looking in the version.xml file of each system you can confirm what patches have been installed. Ensure that patches have been installed on all appropriate systems.
    • Environment variables such as LDR_CNTRL may have been added or modified in the IBM/InformationServer/Server/DSEngine/dsenv file. Generally speaking LDR_CNTRL settings in dsenv should not be modified unless otherwise directed by IBM.
    • Incompatible operating system kernel parameters.

6) Firewall configuration


    8.0.x message: "Error creating DR elements, Error was -1"

    8.1 and later message: "DSR.ADMIN: Error creating DR elements, Error was com.ascential.xmeta.exception.ServiceException"


    The DataStage Server system needs to communicate with the domain system, which means that certain ports need to be open between these systems if they are located on separate machines.

    This sort of problem can be confirmed by looking in the dstage_wrapper_N.log file for errors of the kind "Connection refused:host=<hostname>,port=2809". Ensure the firewall is correctly configured and use telnet <hostname> <port> from the DataStage Server machine to confirm the port is accessible.

    The necessary firewall configuration can be found in the installation guide.


7) Trusted authentication between DataStage Server system and the Domain system failed


    8.0.x message: "Error creating DR elements, Error was -1"

    8.1 and later message: "DSR.ADMIN: Error creating DR elements, Error was Mapping failed to copy attributes: MetaTable -> DSTableDefinition (EObject: null, MetaTable)"

    The DataStage Server system authenticates with the Domain system by a process called trusted authentication. This uses a secure certificate exchange rather than explicit user name and password authentication. If this fails, the project creation will not succeed. Trusted authentication failure is usually identified by multiple exceptions in the DataStage Server ds_logs that says "Null session".

    This can fail for a number of reasons:

    • If the DataStage Server is installed onto a Windows system (say C:\IBM\InformationServer), installing the clients into a different directory (say C:\IBM\InformationServer2) will cause the certificate exchange to fail, ultimately causing the project creation to fail. See APAR JR34441 for more information.
    • The number of trusted sessions reaches a maximum limit, so a new session cannot be started. This is usually identified by an entry in the WebSphere logs that says the limit has been reached. If so, restarting WepSphere Application Server will clear everything so that new sessions can be created and project creation can succeed.


8) DataStage was not installed on the Domain system

    8.0.x message: "Error creating DR elements, Error was -1"

    8.1 and later message: "DSR.ADMIN: Error creating DR elements, Error was Mapping failed to copy attributes: MetaTable -> DSTableDefinition (EObject: null, MetaTable)"

    When installing the Domain and DataStage Server onto different physical systems, the installation of DataStage Server will fail to create projects specified in the installer if DataStage has not been installed onto the Domain. These errors can be found in the installation logs.

    Furthermore, attempting to create projects using the Administrator client or command line will also fail. In both these cases, the exceptions will say that "The package with URI "http:///1.1/DataStageX.ecore" is not registered".

    DataStage can be added to the Domain system by re-running the installer, selecting DataStage and deselecting the other components.


9) Locale (regional settings) customized on the Client system

    8.0.x message: "Error creating DR elements, Error was -1"

    8.1 and later message: "Invalid Node Name %1"



    If the regional language settings have been modified to use a customized short date format (for example "ddd dd/MM/yyyy") it can cause the DataStage Administrator client to send the wrong date information to the DataStage Server, causing project creation to fail. A patch for this issue has been created, under APAR JR34770.


10) Disk / partition full or user quota reached on DataStage Server system

    8.0.x message: "Error creating DR elements, Error was -1"

    8.1 and later message: "DSR.ADMIN: Error creating DR elements, error was log4j: ERROR failed to flush writer."

    The project creation operation will create a log file on the DataStage Server system, called dstage_wrapper_trace_N.log, in the path indicated at the beginning of this document. This will fail if the disk /partition is full or the user to which credentials have been mapped to has reached their disk quota. Free up space as necessary and retry the operation.


11) Project creation fails at "Initializing demo files..." within the Administrator client.

    8.5 message: "Errors were detected during project creation that may render project <name> unstable.

    Caused by: DSR.ADMIN: Error creating DR elements, Error was <date timestamp> java.utils.prefs.FileSystemPreferences$2 run.

    This error states that there was a problem with being able to write Java preference data.

    There are two things that can cause this error to occur.

    i) SE (Security Enhanced) Linux is enabled.

    If SELinux is enabled, please disable it.


      To determine if SELinux is installed and in enforcing mode, you can do one of the following:
      • check the /etc/sysconfig/selinux file
      • run the sestatus command
      • check the /var/log/messages file for SELinux notices (Notice format might differ between RHEL 4 and RHEL 5.)
      To disable SELinux, you can do one of the following:
      • set it in permissive mode and run the setenforce 0 command as a superuser
      • modify /etc/sysconfig/selinux and reboot the machine.

    ii) The user ID that is trying to create the project does not have a local home directory to write to.

    If there is no home directory for the user ID, create a local home directory with write permissions

    (766) and have the group as part of the local dstage group.


12) Stack Execution Disable (SED) is enabled (AIX only)

    8.0.x message: "Error creating DR elements, Error was -1"

    8.1 and later message: "Unable to confirm the JVM can be loaded into the DataStage server process 'DSR_CREATE.PROJECT.B TestJVM' failed"

    If Stack Execution Disable (SED) is enabled in AIX, the JIT compiler fails when trying to execute code it generated in the process data area. This occurs with all of the DataStage executables that have embedded JVMs. Solutions is to turn off the SED at the system level and reboot the machine. To turn off the SED use the command:

    sedmgr -m off

    (The above is also covered in APAR JR40821)



13) Unable to increase the tablespace for the metadata repository (XMETA)

    8.0.x message: "Error creating DR elements, Error was -1"

    8.1 and later message: "DSR.ADMIN: Error creating DR elements, Error was unable to save"

    The SystemOut.log may show an error like:

    Oracle¨ORA-01653: unable to extend table XMETAUSER.LOGGING_LOGGINGEVENT1466CB5F by 128 in tablespace XMETA



    If DB2 is used for the metadata repository (XMETA) look in the <db2instance_home>/sqllib/db2dump/db2diag.log for errors.

    To resolve this problem, increase the tablespace, and retry the operation. Note that it may be necessary to manually delete any partially created project, which can be done by following the material here: http://www-01.ibm.com/support/docview.wss?uid=swg27021312.

14) Error updating secondary indices.
    Error message: "DSR.ADMIN: Error updating secondary indices. Status code = -135 DSJE_ADDPROJECTFAILED"

    A known cause for the error "error updating secondary indices" is one or more missing I_* directories in the /opt/IBM/InformationServer/Server/Template directory. If the there is another DataStage engine installation (of the same version and patch level) available it is possible copy the Template directory from the working engine and use it to replace the Template directory on the broken engine. However be careful the backup the existing Template directory first. Note that if a Template directory is taken from a working engine of a different patch level, some of the patches on the broken engine may be rendered ineffective.
15) Error creating new projects.

    Error message: "DSR.ADMIN: Error creating DR elements. Error was failed domain trusted
    login (failure reason unknown)"

    This error occurs when the DataStage engine tier is unable to obtain trusted connection to service tier during project create. To resolve this issue, you need to run the UpdateSignerCerts.sh (.bat) from the /opt/IBM/InformationServer/ASBNode/bin directory (C:\IBM\InformationServer\ASBNode\bin on Windows). For more information on this command, please refer to the following documentation:
    Running the UpdateSignerCerts command

    If UpdateSignerCerts does not resolve the problem, and the DataStage engine tier is installed on Windows, please also check if the DataStage client is installed on the same machine as the DataStage Engine, but in a different location. For example, if DataStage Engine is at location:
    C:\IBM\InformationServer
    but the DataStage client is installed at other locations such as:
    E:\IBM\Information Server, or C:\IBM\Clients\InformationServer, then that can cause this same error.

    On a Windows server, if the DataStage Engine and Client are installed on the same machine, they must be installed at same location to prevent project create error. To resolve, uninstall only the DataStage client, then reboot the machine and confirm Project create now works. If a need still exists for the DataStage client on same machine, it can then be re-installed, but at same Information Server location as the DataStage engine.

16) Project create fails with following error after upgrading server tiers to 11.5.0.2 + Service pack 2:

    Caught RemoteException: enum constant ASSET_IMPORT_NOTIFICATION does not exist in class com.ibm.xmeta.util.config.FeatureToggleConstants; nested exception is:
    java.io.InvalidObjectException: enum constant

    Service pack 2 (11.5.0.2 + SP2) introduced a new feature toggle "ASSET IMPORT NOTIFICATION" in jar file xmeta_common.jar which exists on 3 tiers (engine, services, and client). If service pack 2 is not installed to all 3 tiers, then the xmeta_common.jar updates on one or more tiers are backlevel and do recognize the new feature. The solution is to install service pack 2 to all tiers. This error specifically occurs during project create if SP2 has been installed to engine tier but not client tier.

[{"Product":{"code":"SSVSEF","label":"IBM InfoSphere DataStage"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"--","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"9.1;8.7;8.5;8.1;8.0;11.5;11.3;11.7","Edition":"All Editions","Line of Business":{"code":"LOB10","label":"Data and AI"}},{"Product":{"code":"SSZJPZ","label":"IBM InfoSphere Information Server"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":" ","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"9.1;8.7;8.5;8.1;8.0;11.5;11.3;11.7","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
16 June 2018

UID

swg21396106