IBM Support

Problem Determination: Synchronization problems

Technote (troubleshooting)


Problem(Abstract)

This is a problem determination document to assist in the analysis of synchronization problems with the IBM WebSphere Application Server.

Resolving the problem

Tab navigation



Trace/Exception analysis:
To investigate the synchronization process, the traces from the MustGather are needed.

Review the following errors and links for solutions:

Review this technote SocketTimeoutException Read timed out seen during SyncNode request if you find the following error within the syncNode.log file:


[6/20/06 8:36:03:489 GMT+01:00] 405b06ac SOAPConnector < reconnect      
       [SOAPException: faultCode=SOAP-ENV:Client; msg=Read timed out;  
       targetException=java.net.SocketTimeoutException: Read timed out]
at org.apache.soap.transport.http.SOAPHTTPConnection.send(Unknown Source


Use annotated trace comparison for analysis. Reviewing the following trace, and then comparing against the trace gathered, greatly assists in pinpointing the problem area.

The following trace illustrates a synchronization operation from start to finish. The following trace level is set for both dmgr and node agent:
Admin=all=enabled:Sync=all=enabled

All traces are taken from the node agent trace file, except the ones starting with "(dmgr trace)".

Step 1. Initialization sync:
[8/22/04 21:27:04:686 EDT] 2e4c94c6 NodeSyncTask  > initiate: false
[8/22/04 21:27:04:686 EDT] 2e4c94c6 SyncResult    > initialize
[8/22/04 21:27:04:686 EDT] 2e4c94c6 SyncResult    < initialize
[8/22/04 21:27:04:686 EDT] 2e4c94c6 NodeSyncTask  d setState: 1 <-- sync state: IDLE=0 COMPARE_FOLDERS=1 PROCESSING_UPDATE=2 PROCESSING_UPDATE_WITH_ERROR=3 VERIFY_SYNC_RESULT=4
[8/22/04 21:27:04:686 EDT] 2e4c94c6 RuntimeCollab > sendNotification   <-- send sync initiated notification           websphere.nodesync.initiated[source=WebSphere:platform=common,cell=yiwen2Network,version=
5.0,name=nodeSync,mbeanIdentifier=nodeSync,type=NodeSync,node=yiwen2,process=nodeagent]


Step 2. Do sync:
[8/22/04 21:27:53:446 EDT] 2b3994ce NodeSyncTask  d run   <-- a new thread is spawn (notice the thread ID 2b3994ce is different from the sync init thread, 2e4c94c6)
[8/22/04 21:27:53:446 EDT] 2b3994ce NodeSyncTask  > doSync   <-- start doSync method, which does most of the work
[8/22/04 21:27:53:446 EDT] 2b3994ce NodeSync      > getCellRepositoryEpoch   <-- compare dmgr's repository epoch with node's repository epoch

(dmgr trace) [8/22/04 21:27:53:517 EDT] 55afd4cd SOAPConnectio > handleRequest   <-- dmgr receives the JMX call
......
(dmgr trace) [8/22/04 21:27:53:567 EDT] 55afd4cd SOAPConnector d Parameter: getRepositoryEpoch(type: class java.lang.String)
......
(dmgr trace) [8/22/04 21:27:53:577 EDT] 55afd4cd SOAPConnectio < handleRequest   <-- dmgr finishes processing the JMX call

Step 2.1 Make getModifiedFolders() JMX call:
[8/22/04 21:27:53:587 EDT] 2b3994ce NodeSync      > invokeGetModifiedFolders   <-- method entry

(dmgr trace) [8/22/04 21:27:53:627 EDT] 478594c2 SOAPConnectio > handleRequest   <-- dmgr receives the JMX call
......
(dmgr trace) [8/22/04 21:27:53:647 EDT] 478594c2 SOAPConnector d Parameter: getModifiedFolders(type: class java.lang.String)
......
(dmgr trace) [8/22/04 21:27:54:878 EDT] 478594c2 SOAPConnectio < handleRequest   <-- dmgr finishes processing the JMX call
.....
[8/22/04 21:27:54:878 EDT] 2b3994ce NodeSync      d modifiedFolders   <-- this is a list of folders changed in master repository in dmgr
                                 cells/yiwen2Network/applications/perfServletApp.ear added
                                 cells/yiwen2Network/applications/perfServletApp.ear/deployments added
                                 cells/yiwen2Network/applications/perfServletApp.ear/deployments/perfServletApp added
                                 cells/yiwen2Network/applications/perfServletApp.ear/deployments/perfServletApp/META-INF added
                                 ........
[8/22/04 21:27:54:878 EDT] 2b3994ce NodeSync      < invokeGetModifiedFolders     <-- method exit        

Step 2.2.2 Make getFolderSyncUpdate() JMX call:
[8/22/04 21:27:54:899 EDT] 2b3994ce NodeSync      > invokeGetFolderSyncUpdate     <-- method entry
..........
(dmgr trace) [8/22/04 21:27:54:919 EDT] 478094c2 SOAPConnectio > handleRequest     <-- dmgr receives the JMX call
.........
(dmgr trace) [8/22/04 21:27:54:949 EDT] 478094c2 SOAPConnector d Parameter: getFolderSyncUpdates(type: class java.lang.String)
..........
(dmgr trace) [8/22/04 21:27:55:179 EDT] 478094c2 SOAPConnectio < handleRequest     <-- dmgr finishes processing the JMX call
..........
[8/22/04 21:27:55:229 EDT] 2b3994ce FileTransferC > downloadFile (5) to InputStream. srcPath=cells/yiwen2Network/applications/perfServletApp.ear/perfServletApp.ear53033.tmp, options=com.ibm.ws.management.filetransfer.client.FileTransferOptionsImpl@551d94c8     <-- changed document is downloaded to node
..........
[8/22/04 21:27:56:030 EDT] 2b3994ce NodeSync      < invokeGetFolderSyncUpdate     <-- method exit, a list of changed documents is returned
                                 cells/yiwen2Network/applications/perfServletApp.ear
   cells/yiwen2Network/applications/perfServletApp.ear/perfServletApp.ear created
                                 cells/yiwen2Network/applications/perfServletApp.ear/deployments
                                 cells/yiwen2Network/applications/perfServletApp.ear/deployments/perfServletApp
                                  ........
 
Step 2.3 Process each document:
[8/22/04 21:27:56:030 EDT] 2b3994ce NodeSyncTask  d processFolderSyncUpdate: cells/yiwen2Network/applications/perfServletApp.ear   <-- processing perfServletApp.ear folder
......
[8/22/04 21:27:56:190 EDT] 2b3994ce FileRepositor A ADMR0009I: Document cells/yiwen2Network/applications/perfServletApp.ear/perfServletApp.ear was created   <-- create the file in perfServletApp.ear folder
.......
[8/22/04 21:27:56:210 EDT] 2b3994ce NodeSyncTask  d processFolderSyncUpdate: cells/yiwen2Network/applications/perfServletApp.ear/deployments   <-- processing deployments folder
.......
[8/22/04 21:27:56:230 EDT] 2b3994ce FileRepositor A ADMR0009I: Document cells/yiwen2Network/applications/perfServletApp.ear/deployments/perfServletApp/deployment.xml was created.     <-- create the file in deployments folder
.......

Step 3 Post sync operations:
[8/22/04 21:27:56:971 EDT] 2b3994ce AppBinaryProc d xxxxxxxxxxx pre: -1 xxxxxxxxxx     <--- app mgmt code gets invoked
......
[8/22/04 21:27:57:102 EDT] 2b3994ce AppBinaryProc < postProcess: -1   <-- app mgmt code gets invoked (do not pay attention to the -1, unless sync fails when around here, when AppBinaryProcessor is called)
......
[8/22/04 21:27:57:102 EDT] 2b3514ce AppBinThread  > run   <-- app mgmt spawn a new thread to expand the ear (note different threads: 2b3514ce 2b3994ce)
[8/22/04 21:27:57:102 EDT] 2b3514ce AppBinThread  d  ---- Expand cells/yiwen2Network/applications/perfServletApp.ear/perfServletApp.ear to [D:\was5\base\installedApps\yiwen2Network\perfServletApp.ear]
........
[8/22/04 21:27:57:102 EDT] 2b3994ce RuntimeCollab > sendNotification     <-- sending out sync complete notification                   websphere.nodesync.complete[source=WebSphere:platform=common,cell=yiwen2Network,version=5.0,name=
nodeSync,mbeanIdentifier=nodeSync,type=NodeSync,node=yiwen2,process=nodeagent]

......
[8/22/04 21:27:57:242 EDT] 2b3994ce NodeSyncTask  < doSync     <-- existing doSync method, we are done

The following ZIP file contains the node agent and dmgr traces used in the preceding trace analysis:

traces_sync.zip


For information on sending documentation to IBM click Submitting Diagnostic Information to IBM.

For a listing of all technotes, search the WebSphere Application Server support page.

Related information

MustGather: Synchronization problems

Cross reference information
Segment Product Component Platform Version Edition
Application Servers Runtimes for Java Technology Java SDK

Document information

More support for: WebSphere Application Server
System Management/Repository

Software version: 7.0, 8.0, 8.5.5, 9.0

Operating system(s): AIX, HP-UX, Linux, Solaris, Windows

Software edition: Network Deployment

Reference #: 1199304

Modified date: 30 March 2007