IBM Support

Troubleshooting: Clustering WebSphere Process Server V6.2

Technote (troubleshooting)


Problem(Abstract)

This document describes how to troubleshoot clustering problems for WebSphere® Process Server. This information can save you time by helping you address common clustering problems before you contact IBM support.

Resolving the problem

Tab navigation


This technote describes ways to avoid problems in a clustered environment, and it helps you solve common problems:




Avoiding problems in a clustered environment

This section includes hints about maintenance and planning a deployment environment, and improvements for a deployment environment setup, which can help you avoid common problems:
  1. Set up separate WebSphere Process Server environments.


    It is common to have separate systems with the same configuration but that are used for different purposes. Do not perform tasks such as development or testing on a production system. Have at least one system (development or staging) that is intended for testing the planned maintenance for the production system. For example, on a staging system that reflects the production environment, you can perform application test runs as well as complex processes, such as upgrading the environment to a new patch level. Although the staging or development system does not have to have the same cluster configuration as the production environment, if it does, development and implementation tasks are easier.

  2. Plan and test product maintenance.


    When you update the product to a newer version using a fix pack, single fix, or refresh pack, first read the installation instructions carefully. In particular, the fix pack instructions include sections for additional required tasks for clustered environments. Have a staging system available that reflects the production system and a corresponding test platform so that you can fix problems before applying an update to a production environment.


    Keep the product updated to the latest major version number so that you fix all of the problems that are addressed in the fix pack. Also stay informed of mandatory critical fixes that are announced on the WebSphere Process Server support site.

  3. Analyze the impact on deployed applications after upgrading the product.


    Installing a fix pack or refresh pack can affect the functionality of deployed applications or performance. In particular, a refresh pack can enhance the set of product features. Test applications on a staging or test environment that reflects the production system, focusing on functionality and performance.

  4. Implement a backup strategy.


    It is important to back up your system configuration regularly and especially before applying a new fix pack or refresh pack. If the update fails, you can reapply the original configuration.


    Two backup tools are provided with WebSphere Application Server: backupConfig and restoreConfig. The tools allow you to back up the entire node configuration. You must run a backup tool on each profile. For related documentation, see:

    Some fix packs update core product files in addition to updating the profile configuration. Read the installation instructions concerning these updates. Backing up your profiles allows you to restore them when an upgrade fails and you must roll back the fix pack. Uninstalling a fix pack does not remove updates to the profiles. This is expected behavior because profiles might be created after the upgrade.


    In addition, certain fix packs that include a profile upgrade might modify the WebSphere Process Server databases. Implement a database backup strategy on a preproduction test environment so you can restore a failed product upgrade. Refer to the installation instructions for fix packs and refresh packs to familiarize yourself with the modifications. Considerations you must be aware of in a product migration scenario are documented as well:



  5. Choose the right database.


    The database back end you use for WebSphere Process Server clustered environments is important for backup strategies and performance. Do not use databases that are unsuitable. For example, you can use a Derby database for testing purposes on a single profile in a development environment, but do not use a Derby database in a production system.

  6. Tune product and application performance.


    Tuning is an important task. Without an appropriate analysis of application and product configuration, you might experience problems in a production environment. Run a performance test on a deployment environment that reflects the production system. Tuning WebSphere Process Server includes tuning the components of WebSphere Application Server (Java™ Virtual Machine, databases, connection pools, and messaging infrastructure). The product documentation guides you through performance tuning:
  7. Use advanced recovery mechanisms in a clustered environment.


    In a clustered environment, when one member of the cluster fails, the remaining members (server instances) assume the workload after a failover is initiated. Recovery of processes and the messaging engine cluster depends on the location and configuration of the transaction log. Configure peer recovery mechanisms for the clusters so that all members share the same transaction log (place the log file on a high-availability file system) and recovery can be performed by the remaining servers. Otherwise, the failed server must be restarted (should be restarted first in recovery and then normal mode) to recover outstanding transactions and processes. See the following topic for information on peer-recovery capabilities:


Resolving deployment environment setup problems

This section describes basic cluster setup for WebSphere Process Server deployment environments. Starting with WebSphere Process Server V6.2.

Overview of available topology patterns
Before you create a deployment environment, you must choose an appropriate topology (IBM topology pattern). Consider the following guidelines before you troubleshoot:

  • In a Single Cluster topology, all functional components (user applications, messaging infrastructure, Common Event Infrastructure (CEI), and support applications) run on the same cluster. Because of scalability limitations, avoid this approach if you use asynchronous interactions or long-running processes.

  • In a Remote Messaging topology, the messaging infrastructure is hosted on a separate cluster. All other functional components remain on the application cluster, which hosts user and support applications and CEI. This topology is appropriate for clusters with limited use of CEI capabilities.

  • Use Remote Messaging and Remote Support topology for production environments because of its flexibility and scalability. In this topology, the application cluster hosts user applications, the messaging cluster hosts the messaging infrastructure, and a third cluster hosts support applications and the CEI service.

Approaches for creating a deployment environment


You can create a deployment environment at one of three times:

  • During product installation

  • Using the Profile Management Tool

  • After you create the deployment manager and federate the custom nodes to the cell. Then you create the deployment environment using the administrative console (using the administrative console gives you full control of the definition).

    As you create a deployment environment, keep in mind the following points:

  • A deployment environment is a description of the basic topology. It does not refer to specific resources (servers, product-related support applications, data sources, or service integration buses).

  • After a deployment environment is generated, WebSphere Process Server does not have a relationship to the configuration description. Therefore, you cannot modify the deployment environment or regenerate the topology if a resource is modified; you must create the deployment environment again.

  • After a deployment environment is configured for a specific database vendor or back-end system, you cannot modify the settings.

  • It is a good practice to back up the system configuration before and after a deployment environment is generated.

Considerations for creating a deployment environment


This section describes prerequisites and important hints for creating a deployment environment and configuring a database:

  • Troubleshooting problems before creating the deployment environment

    • Depending on the approach you select to create a deployment environment, you might need to install the product first. If the installation fails. you create the deployment environment until the product is installed correctly. Refer to the following troubleshooting information:
    • Profile creation is a required part of creating a deployment environment. If problems occur during profile creation or augmentation or to verify that profile creation succeeded, you must review the appropriate log files. Refer to the following information on log files:
  • Database back end for WebSphere Process Server

    • In production environments, you usually use a remote high-end database back end to implement a high-availability and backup strategy. WebSphere Process Server product installations need access to the back-end database. Before creating a deployment environment on a system, make sure that the database is available over the network.

    • Do not use a Derby database for a production environment or as a remote database. This database is not supported and does not work.

    • Independent of the approach you use to create a deployment environment, you need administrative permission (DBA) on the database to create the necessary databases and schemas.

  • Using the product installer to create a deployment environment

    • Because the deployment environment is created during the product installation, you cannot install a fix pack at the same time. Keep in mind that the resulting deployment environment is not based on the latest available and recommended fixes.

    • Applying a fix pack or refresh pack to a deployment environment that is based on the version of the installation medium requires additional time and configuration steps (depending on the fix or refresh pack).

    • Selecting the option deployment environment as installation type, you start with the machine that hosts the deployment manager. Subsequent product installations that will be federated to the deployment manager cell depend on the selected topology pattern, which you select during deployment manager profile creation.

    • After the deployment manager profile is created, make sure that it is started and that it can be reached from all locations where additional product installations and profiles are created. This is a prerequisite for having all nodes federated to the cell and participating in the topology pattern that was selected during deployment manager installation.

  • Using the Profile Management Tool to create a deployment environment.

    • To use the Profile Management Tool to create a deployment environment, WebSphere Process Server must already be installed. Make sure that you have applied the latest available and recommended fixes before creating a deployment environment.

    • Use the same path (advanced or typical path) in the Profile Management Tool to create the deployment manager and custom profiles. Using different paths can cause problems when finishing the configuration and starting the deployment environment:

    • One step in creating a deployment environment is selecting the topology pattern when the deployment manager profile is configured. When the profile is created, make sure that the deployment manager is started before you create and federate custom nodes.

  • Using the administrative console to create a deployment environment.

    • To use the administrative console to create a deployment environment, WebSphere Process Server must already be installed. Make sure that you have applied the latest available and recommended fixes before creating a deployment environment.

    • This is the most flexible approach for creating deployment environments. It provides maximum control during deployment environment configuration.

    • First, create the deployment manager profile and ensure that the deployment manager is startedbefore federating custom nodes.

    • Second, create the custom profiles (which depend on the topology pattern, available machines, and architecture) and federate them to the deployment manager cell.

    • You cannot create a deployment environment until the deployment manager is started In addition, all node agents of custom profiles to be federated to the cell must be started

    • After you create the deployment environment using the administrative console, review and complete the deferred configurations before starting the deployment environment.


Verifying a deployment environment configuration

This section guides administrators through the process of verifying the functionality of a deployment environment:

  1. Was the product installed successfully?
    • Yes, I verified the product installation as described in the product documentation. Continue to the next question.

    • No, a problem is logged in the related log files, as described in the troubleshooting documentation for WebSphere Process Server installations. Review the topics for information on how to handle installation problems.

  2. Have the profiles been created and augmented successfully?

    • Yes, the profile related log files do not include a problem or error message:
      <productInstallRoot>/logs/wasprofile
      /wasprofile_create_<profileName>.log
      (WebSphere Application Server profile)
      /wasprofile_augment_<profileName>.log
      (WebSphere Process Server augmentation)


      Continue to the next question.

    • No, the profile creation or augmentation is marked as failed or as partially successful. Review the appropriate troubleshooting information After the problem is identified and solved, you can try to recover from augmentation files.

  3. Did the deployment manager and the node agents start successfully?

    • Yes, the deployment manager and node agent-related log files indicate that the server is "ready for e-business." No error messages are logged. Continue to the next question.

    • No, the following log files contain error messages preventing the deployment manager or a node agent to be started:


      <profileName>/logs/<deploymentManagerName>|nodeagent
      /startServer.log
      /SystemOut.log
      /SystemErr.log


      Common problems are caused by port conflicts between profiles that are hosted on the same system. Check the log files for port conflicts and adjust the settings if necessary.
      This assumes that default security is enabled on a deployment environment (if it has not been adjusted during the configuration process) and that it is used by the deployment manager and by federated profiles during the startup of certain components. Starting the deployment manager may fail if the security settings are incorrect or cause problems when authenticating WebSphere Process Server components. If you have lost your credentials (you may not be able to log in on the administrative console), you can reset the security settings manually (locate the configuration file below and set the enable attribute to false) to log in and correct the problems:


      <deploymentManagerName>/config/cells/<cellName>/security.xml

      <security:Security [...] enabled="true|false" [...]>
      [...]
      </security:Security>
  4. Can you access the administrative console of the deployment manager.

    • Yes, I logged in with the user name and password that was specified during the deployment manager profile creation and on the default port (9060) using a Web browser:
      http://hostname:port/ibm/console


      Continue to the next question.

    • No, I receive a HTTP 404 error, or the message that the Web browser cannot connect.


      Make sure that the deployment manager started successfully. The server log file includes the following message (if the deployment manager was started using a shell command, the message displays there, too):


      Server <serverName> open for e-business; process id is <id>

      In addition, check that the port that is used in the URL of the Web browser reflects the correct virtual host mapping. Review the server log file to determine the port number that the administrative console (Integrated Solutions Console) is bound to:


      SRVE0250I: Web Module Integrated Solutions Console has been bound to admin_host[*:9060,*:9043].


      If you cannot determine the port number in the log files, review the deployment manager configuration settings for the host (WC_adminhost):

      <deploymentManagerName>\config\cells\<cell>\nodes\<deploymentManagerNode>\serverindex.xml

      <specialEndpoints endPointName="WC_adminhost">
      [...]
      <endPoint host="*" port="9060" />
      [...]
      </specialEndpoints>


      If you still cannot see the administrative console, make sure that the same port number has been defined as a hostAlias for the admin_host virtual host. Host aliases are defined in the following file:

      <deploymentManagerName>\config\cells\<cellName>\virtualhosts.xml

      <host:VirtualHost xmi:id="VirtualHost_2" name="admin_host">
      [...]
      <aliases xmi:id="HostAlias_4" hostname="*" port="9060" />
      [...]
      </host:VistualHost>

  5. Can you connect to the database back-end after starting the database server?

    • Yes, the database server is reachable from all systems where WebSphere Process Server is installed. In addition, an account with administrative permission (DBA) is available and was used during the deployment environment setup. I have started the deployment manager and all node agents, logged in to the administrative console, and tested the data source connections:


      Administrative console Resources JDBC Data sources select a JDBC data source Click Test connection


      All data sources that belong to my deployment environment passed the connection test:





      Continue to the next question.

    • No, an error message is reported during the connection test using the administrative console as described above, redirecting me to the appropriate log file:





      Review the log files of the server and node where the error is reported to get the error message and reason for the failure. After the problem is corrected, test the connection again.


      If you are using a Derby database, make sure that it is not used as the remote or production system database. In addition, in a simple local clustered environment where Derby Network Server is used, make sure that the server is started and available:
  6. Did the deployment environment start successfully?

    • Yes, I have started the deployment environment in the administrative console:


      Servers Deployment Environments <deploymentEnvironmentName> Click Start


      After some time, the deployment environment status is flagged as started and all clusters that belong to the deployment environment are also started:


      Servers Clusters <clusterName>

    • No, I have started the deployment environment and the related clusters are marked as started, but the deployment environment status is still in a stopped state:


      Check if the clusters and application servers are started (you can expect a partial start of a cluster if not all of the servers in a particular cluster are started already):


      Servers Clusters <clusterName>
      Servers Application servers <serverName>


      If clusters and servers are marked as started but not the deployment environment, you might have encountered the following known limitation in deployment environments:


      Deployment environment status incorrectly displayed in WebSphere Process Server V6.1 administrative console


      In any case, make sure that no error messages or warnings are displayed after you start the deployment environment. An unstarted node agent is a potential root cause. Otherwise, identify the failed servers first and review their log files to identify logged error messages for the further investigation.

  7. Did the installed applications start successfully?

    • Yes, all applications are marked as started or partially started (if not all of the cluster-related servers are expected to start):


      Applications Enterprise Applications <applicationName>


      In addition, I can access the Integration Applications using the administrative console:





      The following applications are available in the Web browser using their default URL mapping:


      Business Process Choreographer
      http://:/bpc
      Business Process Choreographer Observer

      http://:/bpcobserver
      Business Rules Manager

      http://:/br

    • No, some of the applications did not start.


      Identify the applications that failed to start. Review the server log files where the application is mapped to for error messages that indicate the problem.


      If you cannot access the Web front end of a service application (Business Process Choreographer, Business Process Choreographer Observer, or Business Rules Manager), check the following configurations:

      • Make sure that the host name and port number are correct.

      • Make sure that you use the correct context root for the Web application:


        Applications Enterprise Applications <applicationName> Context Root for Web Modules
At this point, you have reviewed the most important aspects of your deployment environment configuration. For other problems, review the next sections of this guided series, which address problem determination and resolution through collecting and analyzing data.

Related information

Troubleshooting installation and configuration
Troubleshooting and support
IBM Support Assistant
MustGather for WebSphere Process Server V6
Recommended Fixes for WebSphere Process Server
Verifying your deployment environment
Troubleshooting Guide for WebSphere Process Server

Document information

More support for: WebSphere Process Server
Clustering

Software version: 6.2

Operating system(s): AIX, HP-UX, IBM i, Linux, Solaris, Windows, z/OS

Software edition: NA

Reference #: 1313679

Modified date: 09 September 2008