Troubleshooting
Problem
This is a problem determination document to assist in the analysis of synchronization problems with the IBM WebSphere Application Server.
Resolving The Problem
Tab navigation
- Learn more
- Troubleshoot
- Collect data
- Analyze data- selected tab,
Trace/Exception analysis:
To investigate the synchronization process, the traces from the MustGather are needed.
Review the following errors and links for solutions:
Review this technote SocketTimeoutException Read timed out seen during SyncNode request if you find the following error within the syncNode.log file:
[6/20/06 8:36:03:489 GMT+01:00] 405b06ac SOAPConnector < reconnect
[SOAPException: faultCode=SOAP-ENV:Client; msg=Read timed out;
targetException=java.net.SocketTimeoutException: Read timed out]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(Unknown Source
Use annotated trace comparison for analysis. Reviewing the following trace, and then comparing against the trace gathered, greatly assists in pinpointing the problem area.
The following trace illustrates a synchronization operation from start to finish. The following trace level is set for both dmgr and node agent:
Admin=all=enabled:Sync=all=enabled
All traces are taken from the node agent trace file, except the ones starting with "(dmgr trace)".
Step 1. Initialization sync:
[8/22/04 21:27:04:686 EDT] 2e4c94c6 NodeSyncTask > initiate: false
[8/22/04 21:27:04:686 EDT] 2e4c94c6 SyncResult > initialize
[8/22/04 21:27:04:686 EDT] 2e4c94c6 SyncResult < initialize
[8/22/04 21:27:04:686 EDT] 2e4c94c6 NodeSyncTask d setState: 1 <-- sync state: IDLE=0 COMPARE_FOLDERS=1 PROCESSING_UPDATE=2 PROCESSING_UPDATE_WITH_ERROR=3 VERIFY_SYNC_RESULT=4
[8/22/04 21:27:04:686 EDT] 2e4c94c6 RuntimeCollab > sendNotification <-- send sync initiated notification websphere.nodesync.initiated[source=WebSphere:platform=common,cell=yiwen2Network,version=
5.0,name=nodeSync,mbeanIdentifier=nodeSync,type=NodeSync,node=yiwen2,process=nodeagent]
Step 2. Do sync:
[8/22/04 21:27:53:446 EDT] 2b3994ce NodeSyncTask d run <-- a new thread is spawn (notice the thread ID 2b3994ce is different from the sync init thread, 2e4c94c6)
[8/22/04 21:27:53:446 EDT] 2b3994ce NodeSyncTask > doSync <-- start doSync method, which does most of the work
[8/22/04 21:27:53:446 EDT] 2b3994ce NodeSync > getCellRepositoryEpoch <-- compare dmgr's repository epoch with node's repository epoch
(dmgr trace) [8/22/04 21:27:53:517 EDT] 55afd4cd SOAPConnectio > handleRequest <-- dmgr receives the JMX call
......
(dmgr trace) [8/22/04 21:27:53:567 EDT] 55afd4cd SOAPConnector d Parameter: getRepositoryEpoch(type: class java.lang.String)
......
(dmgr trace) [8/22/04 21:27:53:577 EDT] 55afd4cd SOAPConnectio < handleRequest <-- dmgr finishes processing the JMX call
Step 2.1 Make getModifiedFolders() JMX call:
[8/22/04 21:27:53:587 EDT] 2b3994ce NodeSync > invokeGetModifiedFolders <-- method entry
(dmgr trace) [8/22/04 21:27:53:627 EDT] 478594c2 SOAPConnectio > handleRequest <-- dmgr receives the JMX call
......
(dmgr trace) [8/22/04 21:27:53:647 EDT] 478594c2 SOAPConnector d Parameter: getModifiedFolders(type: class java.lang.String)
......
(dmgr trace) [8/22/04 21:27:54:878 EDT] 478594c2 SOAPConnectio < handleRequest <-- dmgr finishes processing the JMX call
.....
[8/22/04 21:27:54:878 EDT] 2b3994ce NodeSync d modifiedFolders <-- this is a list of folders changed in master repository in dmgr
cells/yiwen2Network/applications/perfServletApp.ear added
cells/yiwen2Network/applications/perfServletApp.ear/deployments added
cells/yiwen2Network/applications/perfServletApp.ear/deployments/perfServletApp added
cells/yiwen2Network/applications/perfServletApp.ear/deployments/perfServletApp/META-INF added
........
[8/22/04 21:27:54:878 EDT] 2b3994ce NodeSync < invokeGetModifiedFolders <-- method exit
Step 2.2.2 Make getFolderSyncUpdate() JMX call:
[8/22/04 21:27:54:899 EDT] 2b3994ce NodeSync > invokeGetFolderSyncUpdate <-- method entry
..........
(dmgr trace) [8/22/04 21:27:54:919 EDT] 478094c2 SOAPConnectio > handleRequest <-- dmgr receives the JMX call
.........
(dmgr trace) [8/22/04 21:27:54:949 EDT] 478094c2 SOAPConnector d Parameter: getFolderSyncUpdates(type: class java.lang.String)
..........
(dmgr trace) [8/22/04 21:27:55:179 EDT] 478094c2 SOAPConnectio < handleRequest <-- dmgr finishes processing the JMX call
..........
[8/22/04 21:27:55:229 EDT] 2b3994ce FileTransferC > downloadFile (5) to InputStream. srcPath=cells/yiwen2Network/applications/perfServletApp.ear/perfServletApp.ear53033.tmp, options=com.ibm.ws.management.filetransfer.client.FileTransferOptionsImpl@551d94c8 <-- changed document is downloaded to node
..........
[8/22/04 21:27:56:030 EDT] 2b3994ce NodeSync < invokeGetFolderSyncUpdate <-- method exit, a list of changed documents is returned
cells/yiwen2Network/applications/perfServletApp.ear
cells/yiwen2Network/applications/perfServletApp.ear/perfServletApp.ear created
cells/yiwen2Network/applications/perfServletApp.ear/deployments
cells/yiwen2Network/applications/perfServletApp.ear/deployments/perfServletApp
........
Step 2.3 Process each document:
[8/22/04 21:27:56:030 EDT] 2b3994ce NodeSyncTask d processFolderSyncUpdate: cells/yiwen2Network/applications/perfServletApp.ear <-- processing perfServletApp.ear folder
......
[8/22/04 21:27:56:190 EDT] 2b3994ce FileRepositor A ADMR0009I: Document cells/yiwen2Network/applications/perfServletApp.ear/perfServletApp.ear was created <-- create the file in perfServletApp.ear folder
.......
[8/22/04 21:27:56:210 EDT] 2b3994ce NodeSyncTask d processFolderSyncUpdate: cells/yiwen2Network/applications/perfServletApp.ear/deployments <-- processing deployments folder
.......
[8/22/04 21:27:56:230 EDT] 2b3994ce FileRepositor A ADMR0009I: Document cells/yiwen2Network/applications/perfServletApp.ear/deployments/perfServletApp/deployment.xml was created. <-- create the file in deployments folder
.......
Step 3 Post sync operations:
[8/22/04 21:27:56:971 EDT] 2b3994ce AppBinaryProc d xxxxxxxxxxx pre: -1 xxxxxxxxxx <--- app mgmt code gets invoked
......
[8/22/04 21:27:57:102 EDT] 2b3994ce AppBinaryProc < postProcess: -1 <-- app mgmt code gets invoked (do not pay attention to the -1, unless sync fails when around here, when AppBinaryProcessor is called)
......
[8/22/04 21:27:57:102 EDT] 2b3514ce AppBinThread > run <-- app mgmt spawn a new thread to expand the ear (note different threads: 2b3514ce 2b3994ce)
[8/22/04 21:27:57:102 EDT] 2b3514ce AppBinThread d ---- Expand cells/yiwen2Network/applications/perfServletApp.ear/perfServletApp.ear to [D:\was5\base\installedApps\yiwen2Network\perfServletApp.ear]
........
[8/22/04 21:27:57:102 EDT] 2b3994ce RuntimeCollab > sendNotification <-- sending out sync complete notification websphere.nodesync.complete[source=WebSphere:platform=common,cell=yiwen2Network,version=5.0,name=
nodeSync,mbeanIdentifier=nodeSync,type=NodeSync,node=yiwen2,process=nodeagent]
......
[8/22/04 21:27:57:242 EDT] 2b3994ce NodeSyncTask < doSync <-- existing doSync method, we are done
The following ZIP file contains the node agent and dmgr traces used in the preceding trace analysis:
For information on sending documentation to IBM click Submitting Diagnostic Information to IBM.
For a listing of all technotes, search the WebSphere Application Server support page.
Related Information
[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"System Management\/Repository","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"9.0;8.5.5;8.0;7.0","Edition":"Network Deployment","Line of Business":{"code":"LOB45","label":"Automation"}},{"Product":{"code":"SSNVBF","label":"Runtimes for Java Technology"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Java SDK","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]
Was this topic helpful?
Document Information
Modified date:
15 June 2018
UID
swg21199304