Bulk load program problems

General problems

Errors occur when processing an XML file

Problem

Parsing errors occur when processing an XML file.

Solution

Ensure that the XML markup in the Identity Markup Language (IdML) file is valid, and that it conforms to the IdML specification:

• You can validate the XML markup using the W3C Markup Validation Service at http://validator.w3.org/.
• You can validate the conformance of the XML to the IdML specification using the idmlcert.jar tool. To use this tool, go to the $COLLATION_HOME/sdk/dla/validator/v2 directory and run the following command:

  java -jar idmlcert.jar idml_book_xml

  where idml_book_xml is the path and file name of an IdML book XML file.

Problem

The following message occurs when processing an XML file:

The file was already processed and will not be processed again.
The logs may contain more information. Return code is: 14

Solution

The bulk load program has already processed the file. To overwrite previous data, run the appropriate bulk load command for your operating system with the -o option.

The bulk load process takes too long to complete

Problem

The bulk load process takes too long to complete, and you think that you should stop it before it completes.

Solution

Use the -g option on the command line of the loadidml.sh or loadidml.bat script. The -g option causes data to be passed into the TADDM server in large chunks, which improves the performance for XML files with certain types of data. However, the -g option cannot be used if any objects in the XML file contain errors.

Typically, pressing Ctrl-C to stop the bulk load process is not a good solution. While the bulk load client is running, review the time that the following logs were last modified:

• $COLLATION_HOME/log/bulkload.log
• $COLLATION_HOME/log/services/TopologyManager.log

If these logs are updated regularly, especially the TopologyManager.log file, the TADDM server stores the data and the bulk load program waits for the TADDM server to complete the storage operation.

For better performance, complete the following steps:

• Increase the bulk cache size in the /etc/bulkload.properties file. For instructions, see the Bulk load parameters tuning section of this documentation.
• Make sure that a Java™ virtual machine is not running out of memory. You can do that by collecting thread dumps of TADDM processes (http://www.ibm.com/support/docview.wss?uid=swg21598190) and reviewing them. If necessary, increase the memory size to 2 GB for proxy (domain server deployment) or 6 GB for storage service (streaming server deployment) by updating the -DTaddm.xmx64=4g parameter in ./deploy-tomcat/ROOT/WEB-INF/storage-server-context.xml file if you use TADDM 7.3.0, or in the ./apps/ROOT/WEB-INF/storage-server-context.xml file if you use TADDM 7.3.0.1, and later .

  The following example illustrates an updated parameter for streaming server deployment:

  value-Xms768M|-Xmx1512M|-DTaddm.xmx64=6g|

• You can also increase the memory size in the com.ibm.cdb.bulk.allocpoolsize property in the bulkload.properties file. This property specifies the maximum amount of memory that can be allocated to the Bulk Loader process. It is an Xmx value that is passed to the main Java class of the Bulk Loader. Specify the value in megabytes.
• When you load a large amount of data, run the stats utility on your database with intervals between 30 minutes and 1 hour. For information about RUNSTATS utility for DB2, see DB2 database maintenance, and for information about DBMS_STATS for ORACLE, see Oracle database maintenance.

Tip: Tests that were run on the ITNMIP book indicate that the performance is optimal when you set the bulk load process properties and parameters to the following values:

com.ibm.cdb.bulk.cachesize=4000
com.ibm.cdb.bulk.allocpoolsize=4096   
value-Xms768M|-Xmx1512M|-DTaddm.xmx64=6g| 

It is also important that you run the RUNSTATS command frequently during the bulk loading process.

Unable to load CICS Transaction Server IdML books

Problem

The discovery library adapter (DLA) for CICS® Transaction Server 4.1 creates IdML books but these books cannot be loaded into TADDM.

Solution

Apply PTF R60M PSY UK71451 for CICS Transaction Server 4.1 and recreate the IdML books.

Unable to load IdML books

Problem

The bulk load process cannot be completed. The error.log file contains following type of error:

java.lang.UnsupportedOperationException: [BaseJdo.E.37] Duplicated objects of the class1 
and class2 classes with the same GUID guid have been detected. 
The in-memory object merge is not supported.

Solution

TADDM does not support IdML books with defined different objects that use the same naming rules and attributes. Normally, they are located after the following comment in an IdML book:

 <!--Create LPAR class for compatibility...--> 

Recreate IdML books without duplicates.

Cannot load z/OS or z/VM book because of duplicated objects

Problem

The discovery library adapter (DLA) for z/OS or z/VM creates IdML books but these books cannot be loaded into TADDM. Messages suggesting that the books contain duplicated objects are displayed.

A message similar to the following error might be displayed:

Duplicated objects of the com.collation.platform.model.topology.app.ConfigFile and com.collation.platform.model.topology.core.LogicalContent classes with the same GUID 196319ED7590336299DA8FE41C817A31 have been detected. The in-memory object merge is not supported.

Solution

For z/OS DLA apply PTF UA68244 and recreate the IdML books.

For z/VM DLA apply IF0001 and recreate the IdML books.

Note: z/VM DLA is not supported anymore.