IBM Support

MustGather: Read first For IBM FileNet Content Manager 5.2 - Process Engine Component

Question & Answer


Question

What information should be collected when I experience a problem with the Process Engine component of the IBM FileNet Content Manager 5.2?

Answer


Must Gather for the Process Engine component of the IBM FileNet Content Manager 5.2 software

Collecting data before calling IBM support will help you understand the problem and save time analyzing the data.

Collecting data early, before opening the PMR, helps IBM® Support quickly determine if:

  • Symptoms match known problems (rediscovery).
  • There is a non-defect problem that can be identified and resolved.
  • There is a defect that identifies a workaround to reduce severity.

Locating the root cause of a problem can speed development of a code fix.

Describing the problem
Gathering Operating System Versions, Settings and Logs
Gathering vwtool output
Gathering DB version info
Gathering Content Engine status
Gathering Process Engine status
Gathering Logs and Traces
Gathering javacores
Gathering problem-specific data

Note that there are many significant differences between the new Process Engine component of the IBM FileNet Content Manager 5.2 software and the older Process Engine 4.x and Process Engine 5.0 software.

The Must Gather for Process Engine 5.0 is here:
http://www.ibm.com/support/docview.wss?uid=swg21446755

The Must Gather for Process Engine 4.5.1 is here:
http://www.ibm.com/support/docview.wss?uid=swg21327304.

This technote describes the information that must be gathered for the Process Engine component of the IBM FileNet Content Manager 5.2. You can collect this information "by hand," using the information described here. Or, you can use the automated Data Collector for the IBM FileNet Content Manager 5.2.

The use of the automated IBM FileNet Content Manager 5.2 data collector is highly encouraged, and is significantly easier than gathering the information "by hand." If you use the data collector, you would only have to be concerned about the steps described under the "Describing the Problem" and "Collecting Information Specific to the Problem" headings. For example, if you can reproduce the problem, you might enable the appropriate traces and replicate the problem and then run the automated data collector. The data collector would then gather those traces along with all of the other basic information described below.

The technote describing how to run the IBM FileNet Content Manager 5.2 data collector to collect data for the Process Engine component is here:
http://www.ibm.com/support/docview.wss?uid=swg21627110



Describing the Problem

This background information is needed for opening a PMR, but is not gathered by the data collector:

  • How often does this problem occur?
  • Can the problem be reproduced? And if so, how is it reproduced?
  • How many users are affected by this problem?
  • Is the problem in newly exercised functionality? Or was this previously working and now it's stopped working?
  • What, if anything, has been changed?
  • What is the business impact of this problem?
  • Is there a workaround?

Gathering Operating System Versions, Settings and Logs

These commands collect:
  • OS Version
  • OS Environment Variables
  • OS Kernel parameters
  • OS running processes
  • OS networking information
  • OS locale information
  • OS log files

Run the following commands on the Content Manager server and collect the output:

AIX Servers
Command​ Collects​
/usr/sbin/instfix -ivq ​ OS Patches or Maintenance level​
env ​ OS Environment Variables​
errpt -a​ OS Configuration & Error Reporting​
/usr/sbin/lsattr -E -l sys0 ​ OS Kernel parameters​
ps -ef​ currently running processes​
netstat -an​ socket information​
locale​
locale -a​
locale information​
/var/adm/syslog/syslog.log​ include system log files​


HP Servers
Command​ Collects​
/usr/sbin/swlist -l product ​ OS Patches or Maintenance level​
env ​ OS Environment Variables​
/usr/sbin/dmesg ​ OS Configuration & Error Reporting​
/usr/sbin/sysdef ​ OS Kernel parameters​
ps -ef​ currently running processes​
netstat -an​ socket information​
locale​
locale -a​
locale information​
/var/adm/syslog/syslog.log​ include system log files​


Solaris Servers
Command​ Collects​
/usr/bin/showrev -a ​ OS Patches or Maintenance level​
env ​ OS Environment Variables​
/usr/sbin/prtdiag -v ​ OS Configuration & Error Reporting​
/usr/sbin/sysdef -D ​
/usr/bin/getconf -a​
OS Kernel parameters​
ps -ef​ currently running processes​
netstat -an​ socket information​
locale​
locale -a​
locale information​
/var/adm/messages & /var/log/syslog​ include system log files​


Windows Servers
Command​ Collects​
WMIC /OUTPUT:C:\ProcessList.txt PROCESS get Caption,Processid,Usermodetime,Commandline​ current running processes including the command line​
set​ OS Environment Variables​
Windows 2003: ​
cscript C:\windows\system32\eventquery.vbs ​
/L application /R 1000 /V ​
cscript C:\windows\system32\eventquery.vbs ​
/L system /R 1000 /V ​

Windows 2008 and newer:​
wevtutil epl System Systemlog.evtx​
wevtutil epl Application Applicationlog.evtx​
OS Event Logs​
netstat -an​ socket information​
chcp​ display code page​


Linux Servers
Command​ Collects​
uname -a​ OS Version​
/usr/bin/showrev -a ​ OS Patches or Maintenance level​
env ​ OS Environment Variables​
/usr/sbin/prtdiag -v ​ OS Configuration & Error Reporting​
/usr/sbin/sysdef -D ​
/usr/bin/getconf -a ​
OS Kernel parameters​
/usr/bin/getconf -a​ configuration info​
/bin/dmesg​ messages​
ps -ef​ currently running processes​
netstat -an​ socket information​
locale​
locale -a​
locale information​
/var/log/messages.*​ include system log files​



IBM FileNet Content Manager 5.2 Process Engine Version and Build

java -classpath \Program Files\IBM\FileNet\ContentEngine\lib\pe.jar filenet.vw.toolkit.utils.VWBuildStamp

You may need to fully specify the path to java and you may need to fully specify the path to pe.jar.

Run vwtool and collect the output from these vwtool commands:

Run vwtool <connection point> in a command prompt window. Then run these vwtool commands, collecting the resulting output:

version
asyncrpc list PEVerify
asyncrpc list CmdLine_Transfer
pedbrpt
usralias l
stepprocessors i
configdetails
schemastatus
sysconfig b
appspace
dump f c s
config
queueconfig *
rosterconfig *
logconfig *
dump s
regionConfig
dbviews
views
loadstatus
count *



Process Engine Database information

Database Type and Version... For example, Oracle 11g 11.1.0.6 or SQL Server 2008 or DB2 LUW 9.5 FP2. Note that vwtool's configdetails command (collected above) will include this information in its output.

Is the database on the same server as the Content Manager Process Engine or on a remote database server?

Include the OS type and version information for database server if the database is remote.


Content Engine Configuration and Status

In a web browser, go to your Content Engine's basic info pages:

http://cpehost:9080/wsi/FNCEWS40MTOM

http://cpehost:9080/FileNet/Engine

In the above URLs, replace cpehost with the actual hostname of your Content Manager server, and, if necessary, replace 9080 with the actual port for your CE Server.

Include the returned HTML and XML pages.



Process Engine Configuration and Status

Include the returned HTML page for these URLs as seen in a browser:

http://cpehost:9080/peengine/IOR/admin/help

http://cpehost:9080/peengine/IOR/ping?cp=myCP&farm=true&dbinfo=true
http://cpehost:9080/peengine/IOR/ping?systeminfo=true
http://cpehost:9080/peengine/IOR/ping?cp=myCP&java_dump=true
http://cpehost:9080/peengine/admin/async
http://cpehost:9080/peengine/admin/cm/audit
http://cpehost:9080/peengine/admin/cm/stats
http://cpehost:9080/peengine/admin/cm/task?name=*&action=info
http://cpehost:9080/peengine/admin/api

 

In the above URLs, replace cpehost with the actual hostname of your FileNet Content Manager server, and, if necessary, replace 9080 with the actual port for your server.

In the response to the java_dump=true request, in the "Other message" area, you'll see the location of the java dump that was created. Look in that location and capture the *javacore.*.txt file.

You must substitute a valid connection point name for myCP in the above URLs.


Logs and Traces

Include the Content Manager Process Engine logs and traces from the last 7 days. The logs are in the "Log File Location" folder which is listed in the response to the IOR/ping page. This might be, for example, the C:\Program Files\IBM\WebSphere\AppServer\profiles\AppSrv01\FileNet\server1 folder.

Also, collect the VWUpgrade_system.log and VWUpgrade_trace.log files if they exist. They'll be in the same "Log File location" as the other logs.
 


Content Manager Javacores

Include any javacore.* files that may have been created during the last seven days. These might be in C:\Program Files\IBM\WebSphere\AppServer\profiles\AppSrv01 for example.



Collecting Information Specific to the Problem

The following sections describe extra information that can be collected for particular types of problems.


Farm Issues - Load Balancer Configuration

Include the following information:

  • Brand and model of load balancer
  • Load Balancer settings and configuration information

The data collector does not gather this load balancer configuration and settings automatically.


Tracing In General

The data collector does not automatically enable or create any tracing. But it will collect any traces that have already been created at the time the data collector is run.

If you will be collecting traces, it is easiest to generate the traces before running the data collector so that the traces are automatically included as part of the data collection.

If you have collected any traces, be sure to state the date and time when the problem has occurred.
 


Screenshots

The data collector does not automatically take or gather any screenshots. It may be appropriate to include screenshots of errors or other symptoms that will help us understand the problem.
 


Content Management Server Fault

If the problem is reproducible, it can be invaluable to enable the appropriate Process Engine traces and re-created the problem.  For example, in vwtool, you can use the trace command to enable server tracing. The exact traces you enable would vary, depending upon the type of server problem. And after enabling the traces, reproduce the problem.

Include the trace output file from a period where the problem has occurred. Be sure to state the date and time when the problem has occurred.

Include any dumps or cores that may be produced.



Reproducible Problem

If the problem is reproducible, it may be helpful to turn on "ORB RPC", "ORB RPC INPUT", and "Database Access" tracing. Then, reproduce the problem and include trace output. Be sure to state the date and time when the problem has occurred.



Content Manager Process Engine Hung or Performance Problem

If the server is hung or there is a performance issue, get three java dumps, pausing for approximately 30 seconds between each java dump. To generate a java dump, use this URL in a browser:

http://cpehost:9080/peengine/IOR/ping?cp=myCP&java_dump=true
 

In the above URLs, replace cpehost with the actual hostname of your Content Manager server, and, if necessary, replace 9080 with the actual port for your CE Server.

Include the three javacore files.

Also note the CPU load when the server was hung. Was the load at zero (indicating everything was blocked)? Or was it at some other number, indicating the server was possibly looping?

If the server is hung and the CPU load is not at zero, turn on "ORB RPC", "ORB RPC Input", and "Database access" tracing, and include the trace output.

Be sure to state the date and time when the problem has occurred.

You may need to use the WebSphere data collector as well, to gather more information specific to the hang.



Security / Connection to CE / Logon / Group Membership issues

Enable the appropriate Content Manager Process Engine traces using vwtool's trace command. For security issues, it is generally good to enable the tracing for "Security calls" and "Environment cache".

Then, reproduce the problem and include the trace output. Be sure to state the date and time when the problem has occurred.




PE API Issues

Enable the appropriate java trace options in the fnlog4j.properties file.

Put that fnlog4j.properties file in the client's JVM’s LIB folder. Note that the appropriate JVM may be within an application server.

See the ...\ProcessEngine\data\fnlog4j.properties.sample file for an example fnlog4j.properties file.

Replicate the problem and submit the resulting java trace file. Be sure to state the date and time when the problem has occurred.

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSTHRT","label":"IBM Case Foundation"},"Component":"Process Engine","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"5.5.x, 5.2.x","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
08 August 2019

UID

swg21589938