Collect troubleshooting data for performance problems in IBM Business Process Manager (BPM)

Collect the troubleshooting information for performance problems in IBM Business Process Manager. Gathering this information before calling IBM Support helps in understanding the problem and saves time when analyzing the data.

Getting an overall picture of the performance issue

Performance issues involve many aspects: Business Process Manager server, application, database, outside connectors, network, and other variables. After the scope of the issue is narrowed, tuning can start and then performance is tested again to find the next bottleneck.

The most important first step is to accurately identify the specific action that is slow and to reproduce the condition. Intermittent performance problems are more difficult to track down.

Note: When the environment is a cluster, isolate the activity to one server. To achieve this situation, one or more of the following conditions is required:

• Turn off all nodes but one
• Access a node directly and bypass the load balancer
• Minimumize the number of users (hopefully one) to reproduce the issue. In some cases, performance degrades as more users are added to the system. A systematic control of adding users to the system might be necessary in some cases.

Provide the following information:

• Where specifically are you seeing performance problems? (for instance, the Authoring Environment, User Experience (launching a business process definition (BPD), progressing to the next coach in a flow, and so on), Custom solution code, Systemic (impacting all areas), and so on). Review the Improving application performance in IBM Business Process Manager V7.5 by using DynaCache document on developerWorks.
• Can you quantify the performance issue? How long does a specific action take? Do you have logging that shows where the time is being spent? Do you have performance tools to help analyze or collect data? 
• Is this a new problem or have you always had it in this environment? Do you see the same problem in other environments on the network?
• Is the performance problem consistent or does it change? Is the performance worse at certain times of day? Is performance bad under load? Does the performance improve after a restart of the application server?
• What are the hardware specifications of the machines hosting the Business Process Manager servers (number of CPUs, RAM, and so on)? Verify that you meet all of the system requirements.
• What are the hardware specifications of the database server?
• Do your IBM Business Process Manager and database servers run in a virtual machine? If yes, provide the software that you are using as well as its version and type. Also review the specific tuning guide.
• Which topology are you using? Are your servers clustered or stand-alone? If you have a clustered environment, provide more details (different physical machines, hostnames, load balancer, and so on). Make sure that all the prefix URLs, like <portal-prefix>, are pointing to the load balancer if you have one in place.
• Is your database a shared resource?
• Version information of your IBM Business Process Manager and other involved software
• Version information of your operating system and your database server

Collecting diagnostic information

Important: All of this data must be collected at the same time.

1. Enable the SQL tracing:
   
   In IBM Business Process Manager 7.5.x and 8.x:
   *=info:WAS.clientinfopluslogging=all
   
   In WebSphere Lombardi Edition 7.1 and 7.2:
   com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement=all
   
   
2. Reproduce the performance issue and collect the complete directories from all profiles and servers:
   • install​​_​​root/​​profiles/profile​​_​​name​​/​​logs​​/server​​_​​name 
   • install_​​root/​​profiles/profile​​_​​name/​logs​​/ffdc
     
     
3. Collect the Java™ virtual machine (JVM) thread dumps - Java cores separated by 30 seconds from the main Java virtual machine.
   • IBM Business Process Manager 7.5.x and 8.x - server1 for stand-alone, AppTarget for clustered environment
   • WebSphere Lombardi Edition - twprocsvr
     
     Methods of collecting:
   • Windows -
     heapdumps (Collect this data only if you are experiencing out-of-memory problems)
     Java thread dumps
   • Linux - kill -3 <java_pid>
     
     
4. Collect Instrumentation logging - collected for no more than five (5) minutes.
   
   
5. Complete the database diagnostics:
   • Diagnostic performance report from the database.
     • Oracle - Automated Workload Report (AWR)
     • Microsoft SQL Server - Dashboard report (v2005, v2008)
     • IBM DB2 - Run the following commands against the Process Server database:
       • db2 get db cfg
       • db2support . -d <dbname> -c -g -s -o db2support_primary.zip
       • db2support . -d <dbname> -g -s -o db2support_standby.zip
       • db2evmon -path /tmp > sqltrace.txt (http://www.ibm.com/developerworks/data/library/techarticle/0303kolluru/0303kolluru.html)
   • Database table queries
     • select count(*) as lsw_usr_xref from lsw_usr_xref;
     • select count(*) as lsw_usr_grp_xref from lsw_usr_grp_xref;
     • select count(*) as lsw_usr_grp_mem_xref from lsw_usr_grp_mem_xref;
     • select count(*) as lsw_grp_grp_mem_xref from lsw_grp_grp_mem_xref;
     • select count(*) as lsw_grp_grp_mem_exploded_xref from lsw_grp_grp_mem_exploded_xref;
     • select count(*) as lsw_task from lsw_task;
     • select count(*) as lsw_bpd_instance from lsw_bpd_instance;
     • select count(*) as lsw_bpd from lsw_bpd;
     • select count(*) as lsw_snapshot from lsw_snapshot;
     • select count(*) as lsw_project from lsw_project;
     • select count(*) as lsw_favorite from lsw_favorite;
     • select count(*) as lsw_poversions from lsw_po_versions;
     • select code.NAME, COUNT(t.STATUS) from lsw_task t right join LSW_TASK_STATUS_CODES code on code.STATUS_VALUE = t.STATUS group by code.NAME order by code.NAME use twprocdb;
     • select code.NAME, COUNT(bpd.EXECUTION_STATUS) from LSW_BPD_INSTANCE bpd right join lsw_bpd_status_codes code on code.STATUS_ID = bpd.execution_status group by code.NAME order by code.NAME;
       
       
6. Review problems with the slow loading of a specific page in a browser.
   Examples: loading a coach, navigating the Process Inspector page, or login to the IBM Business Process Manager Portal
   • Use Mozilla FireFox and the Firebug plugin. After installing Firebug, add the NetExport plugin to allow exports of the network traffic traces.
   • If you are unable to use FireFox, then an alternative HTTP trace is needed that shows all the files that have been downloaded and how long it takes. The Information Technology staff at your company might have tools to monitor speeds of web pages.
     
     
7. Use the following servlet to display the current number of the sessions in memory for your applications. You need to call it for each server in your AppTarget cluster:
   http://server_name:port_number/servlet/com.ibm.ws.webcontainer.httpsession.IBMTrackerDebug
   • The server_name variable is your AppTarget server that hosts the DefaultApplication.ear application.
   • The port_number variable is your WC_defaulthost port for this server.

Your server must be started. Provide the output of the servlet.

Note: If your browser displays errors, check if the DefaultApplication.ear is installed and started on your AppTarget cluster. If it is not, you can install it from install_root /installableApps.

General diagnostic information

Collect the general MustGather information as described in Collect troubleshooting data for the IBM Business Process Manager products.

What to do next

1. Review the logs and traces at the time of the problem to try to determine the source of the problem.
   
   
2. Use IBM Support Assistant to search for known problems in the information center, forums, and technotes.
   
   
3. If you cannot find related problems or cannot solve the problem, send the information you have collected to IBM by following the instructions in Exchanging Information with IBM Technical Support.