Troubleshooting migration

• Job fails in the Verify step.

  This indicates that a configuration error was detected before beginning the migration process. This can be due to either incorrect data entered when you created the migration jobs or a configuration problem. Review the log output for the error detected, then correct and rerun. The logs are located in temporary_directory_location/nnnnn, where temporary_directory_location is the value that you specified when you created the migration jobs (where the default is /tmp/migrate) and nnnnn is a unique number that is generated and displayed during the creation of your migration jobs as well as displayed in the JESOUT DDNAME of the WROUT and WRERR steps of your batch job stream.

• Job fails after the Verify step.

  In the event of failure in the migration job after the Verify step, you can rerun the migration job; but first, you must delete the WebSphere Application Server for z/OS® configuration home directory that is created in the CRHOME step. This corresponds to the home directory that you entered when you created the migration jobs, and it can also be found in the migration Job Control Language (JCL) environment variable V6_HomeDir. Because the migration procedure creates a new configuration file system for each node being migrated, it is a simple process to delete the configuration and start from scratch.

• Problems occur with a federated node migration.

  A federated node is the most complex node to migrate because it is essentially two migrations that are rolled into one. A federated node requires a migration of the node configuration information that is contained in the deployment manager's primary repository as well as the configuration information contained in the federated node. Federated node migration requires an active connection to the deployment manager. If you havesecurity that is enabled , it is essential that you follow the instructions that were generated when you created the migration jobs. The migration job must be submitted with a WebSphere Administrator's user ID that has been properly configured for obtaining secure connections.

• Job fails during the application-installation phase of migration.

  If you select the option for the migration process to install the enterprise applications that exist in the Version 7.0 or later configuration into the new Version 9.0 configuration, you might encounter error messages during the application-installation phase of migration.

  The applications that exist in the Version 7.0 or later configuration might have incorrect deployment information—typically, invalid XML documents that were not validated sufficiently in previous WebSphere Application Server runtimes. The runtime now has an improved application-installation validation process and will fail to install these malformed EAR files. This results in a failure during the application-installation phase of WASPostUpgrade and produces an E error message. This is an unrecoverable migration error.

  If migration fails in this way during application installation, you can do one of the following:

  • Fix the problems in the Version 7.0 or later applications, and then remigrate.
  • Proceed with the migration and ignore these errors.
    1. Restart the migration job in the FINISHUP step to allow the remaining migration functions to be performed.

       Do this by adding the RESTART=FINISHUP parameter to the job card and resubmitting the job.

    2. Later, fix the problems in the applications and then manually install them in the new Version 9.0 configuration by using the administrative console or an install script.
• Out-of-space errors occur.

  The migration logs are located in temporary_directory_location/nnnnn, where temporary_directory_location is the value that you specified when you created the migration jobs (where the default is /tmp/migrate) and nnnnn is a unique number that was generated during the creation of your migration jobs. Normally, the space requirements for the migration logs are small. However, if you enable tracing the log files can be large. The best practice is to enable tracing only after problems have been found. If tracing is required, try to only enable tracing related to the step in the process that is being debugged. This helps to reduce the space requirements.

  You can enable tracing when you create the migration jobs by using the z/OS Migration Management Tool or the zmmt command. To enable tracing with the zmmt command, set the following properties to a possible value in the response file:

  • zmbEnablePreUpgradeTrace
  • zmbEnablePostUpgradeTrace
  • zmbEnableProfileTrace
  • zmbEnableScriptingTrace

  Set zmbEnablePreUpgradeTrace and zmbEnablePostUpgradeTrace to a value between 0 for no tracing to 4 for all tracing. Set zmbEnableProfileTrace and zmbEnableScriptingTrace to either 0 for no tracing or 1 to enable tracing.

  You can also change the variables in the migration Job Control Language (JCL) to appropriate values in the JCL member BBOWMxEV in the DATA data set that is created when you use the z/OS Migration Management Tool or the zmmt command to create a migration definition. When modifying the JCL, set TraceState and profileTrace to disabled or enabled, and set preUpGradeTrace and postUpGradeTrace to a value between 0 for no tracing to 4 for all tracing.

  Note: For a deployment manager, the member name is BBOWMDEV; for a federated node, the member name is BBOWMMEV.

  During migration, a backup copy of your Version 7.0 or later configuration is made. This backup becomes the source of the information being migrated. The default backup location is /tmp/migrate/nnnnn. This location can be changed when you create the migration jobs. Depending on the size of the node being migrated, this backup can be large. If your temporary space is inadequate, then you need to relocate this backup.

• Out-of-memory errors occur.
  To improve memory usage, perform the following actions:

  1. Edit your job file to prevent sharing of application space, and increase the JVM heap minimum and maximum.
     The following is an example of how to edit the BBOWM3D job for a deployment manager migration so that it uses up to 768 MB of heap, which is more than the default of 256.

     BPXBATCH SH + export _BPX_SHAREAS=NO; +
     export IBM_JAVA_OPTIONS="-Xms256M -Xmx768M"; + 
     /wit/bigtmp/bbomigrt2.sh WASPreUpgrade + 
     /wit/bigtmp/24173105/_ + 
     1>> /wit/bigtmp/24173105/BBOWMG3D.out +
     2>> /wit/bigtmp/24173105/BBOWMG3D.err; 

  2. Edit the appropriate migration script.

     If you are migrating from a system where you have access to the read-only driver file system, edit the WASPreUpgrade.sh and WASPostUpgrade.sh scripts in the bin directory.

     If you cannot edit the read-only driver system, use the three migration jobs instead of the single migration job so that the migration pauses after profile creation and you can edit the profile scripts. Running the BBOWMG3* migration job is equivalent to running the following three jobs in the order listed:

     • BBOWMPRO
     • BBOWMPRE
     • BBOWMPOS

     The BBOWMG3* job and the three individual jobs are mutually exclusive—either run the single BBOWMG3* job or run the three individual jobs, but not both.

  The *PRO job performs product installation and profile creation. After this job completes successfully, you have access to the Version 9.0 profile that will be used for the migration. Go into the Version 9.0 installation directory and go to the equivalent ${WAS_HOME}/bin location. You will find the WASPreUpgrade.sh and WASPostUpgrade.sh scripts at this location. If they are symlinks back to the read-only file system, remove the symlinks and copy the original files from the read-only file system to this location. After you have actual migration script files in ${WAS_HOME}/bin, edit those files and change the lines for Performance Java™ Options to change the heap settings to a value appropriate for your system. For example:

  set PERFJAVAOPTION=-Xms256M -Xmx768M

  You can now continue your migration. If you decided to run the three individual jobs, launch the BBOWMPRE job and after that is successful (RC=0) run the BBOWMPOS job. If you edited the read-only file-system copy of the migration script files, you can run the appropriate BBOWMG3* job.

• Batch job time is exceeded.

  Each z/OS installation is different with respect to job classes and time limitations. Make sure that you have specified appropriate job classes and timeout values on your job card.

• Migration of a deployment manager or stand-alone application server completes but no applications are installed.
  The log file might show messages similar to the following:

  MIGR0339I: Application WISO_wisoadmin_war.ear is deploying using the wsadmin command.
  MIGR0241I: Output of wsadmin.
  Error: unable to allocate 268435456 bytes for GC in j9vmem_reserve_memory.
  JVMJ9VM015W Initialization error for library j9gc23(2): Failed to instantiate heap. 256M requested
  Could not create the Java virtual machine.

  The problem is that the WASPostUpgrade script launched from bbomigrt2.sh does not have enough remaining address space to initialize the Java Virtual Machine (JVM). Typically, this indicates that the spawned process is running in the same address space as the WASPostUpgrade JVM.

  You can use the environment variable _BPX_SHAREAS to tell the underlying process whether or not spawned processes should share the same address space as the parent process. The default value (null) is NO, but administrators can change this to YES or MUST to get a performance benefit because the address space does not need to be copied during fork or spawn actions.

  If your system is experiencing the problem described here, run the migration job with the environment variable _BPX_SHAREAS explicitly set to NO. You can set this either in the system profile (/etc/profile) or in the user profile for the user running the migration job. Use the following entry to set this variable to NO:

  export _BPX_SHAREAS = NO 

  After the migration job completes, you can update the profile to reset _BPX_SHAREAS to its original value.

• Failures occur during server startup after migration.

  Review the instructions that were generated when you created the migration jobs. Verify that the JCL procedures have been copied over correctly to your PROCLIB, the RACF® definitions have been created, and the Version 9.0 libraries have been authorized. Make sure that the daemon process associated with your cell is at the appropriate level. The daemon process must be at the highest WebSphere Application Server for z/OS version level of all servers that it manages within the cell.

  After migrating to a Version 9.0 cell that contains or interoperates with Version 7.0 or later nodes that are not at Version 6.0.2.11 or later, the cluster function might fail. When starting these Version 7.0 or later application servers, you might see the following problems:

  • You might see a first failure data capture (FFDC) log that shows a ClassNotFoundException error message. This exception is thrown from the RuleEtiquette.runRules method and looks something like the following example:

    Exception = java.lang.ClassNotFoundException
    Source = com.ibm.ws.cluster.selection.SelectionAdvisor.<init>
    probeid = 133
    Stack Dump = java.lang.ClassNotFoundException: rule.local.server
    at java.net.URLClassLoader.findClass(URLClassLoader.java(Compiled Code))
    at com.ibm.ws.bootstrap.ExtClassLoader.findClass(ExtClassLoader.java:106)
    at java.lang.ClassLoader.loadClass(ClassLoader.java(Compiled Code))
    at java.lang.ClassLoader.loadClass(ClassLoader.java(Compiled Code))
    at java.lang.Class.forName1(Native Method)
    at java.lang.Class.forName(Class.java(Compiled Code))
    at com.ibm.ws.cluster.selection.rule.RuleEtiquette.runRules(RuleEtiquette.java:154)
    at com.ibm.ws.cluster.selection.SelectionAdvisor.handleNotification(SelectionAdvisor.java:153)
    at com.ibm.websphere.cluster.topography.DescriptionFactory$Notifier.run(DescriptionFactory.java:257)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1462)

  • You might see a java.io.IOException that looks something like the following example:

    Exception = java.io.IOException
    Source = com.ibm.ws.cluster.topography.DescriptionManagerA. update probeid = 362
    Stack Dump = java.io.IOException
    at com.ibm.ws.cluster.topography.ClusterDescriptionImpl.importFromStream(ClusterDescriptionImpl.java:916)
    at com.ibm.ws.cluster.topography.DescriptionManagerA.update(DescriptionManagerA.java:360)
    Caused by: java.io.EOFException
    at java.io.DataInputStream.readFully(DataInputStream.java(Compiled Code))
    at java.io.DataInputStream.readUTF(DataInputStream.java(Compiled Code))
    at com.ibm.ws.cluster.topography.KeyRepositoryImpl.importFromStream(KeyRepositoryImpl.java:193)

  • Avoid trouble: After migrating to WebSphere Application Server Version 9.0, the Session Initiation Protocol (SIP) proxy JSR116 traffic might fail with retransmission timeouts and error messages. When this error occurs, the following error message might be displayed:

    TCP Channel initialization failed.  The socket bind failed for host and port 5060.

    To resolve this problem, you can delete the transport chain, UDP_SIP_PROXY_CHAIN, in the serverindex.xml file at the node level of the server where the error occurred.

After migration, carefully review the job output and log files for errors.

If you migrate a node to Version 9.0 then discover that you need to revert to Version 7.0 or later, read Rolling back environments.

For current information available from IBM® Support on known problems and their resolution, read the IBM Support page. IBM Support also has documents that can save you time gathering information needed to resolve this problem. Before opening a PMR, read the IBM Support page.

New ports that are registered on a migrated Version 9.0 node agent include: WC_defaulthost, WC_defaulthost_secure, WC_adminhost, WC_adminhost_secure SIB_ENDPOINT_ADDRESS, SIB_ENDPOINT_SECURE_ADDRESS ,SIB_MQ_ENDPOINT_ADDRESS, SIB_MQ_ENDPOINT_SECURE_ADDRESS. These ports are not needed by the node agent, and can be safely deleted.