Portal server and Dynacache replication issues

Technote (FAQ)


Question

This technote discusses some common issues observed with a WebSphere Portal server cluster having performance, memory issues, and HAManager/DRS exceptions. You might see errors or warning such as follows:

(1) DCSV1051W/DCSV1052W: DCS Stack DefaultCoreGroup.DelCluster at Member del\del5\WebSphere_Portal: Raised a high severity congestion event for outgoing messages. Internal details are Total stored bytes: 67701476, Red mark is 41943040, Yellow mark is 37748736, Green mark is 8388608.

(2) HMGR0152W: CPU Starvation detected. Current thread scheduling delay is 109 seconds.

(3) CWWDR0010E: Replication instance ws/wp.te.transformationAssociationCache caught exception when sending/receiving messages : com.ibm.wsspi.hamanager.datastack.DataStackMembershipChangingException: The target member is not currently in view.
Caused by: com.ibm.ws.dcs.common.DCSMemberIsNotDefinedException : Illegal member is - IBM\IBM8\WebSphere_PortalD , legal members are: IBM\IBM5\WebSphere_Portal IBM\IBM7\WebSphere_PortalC IBM\IBM6\WebSphere_PortalB

(4) Other suggested WebSphere Application server DRS cache replication configuration

Answer

An explanation and suggested steps for each message follow.

(1) DCSV1051W/DCSV1052W

This message is due to the DRS Dynacache replication facing congestion in the transport layer. This cache needs more memory requirement for HAmanager cache and transport buffers. A Portal server can have 50+ cache instances.

To address, tune the HA Manager transport memory size (excerpts from the technote "Tune High Availability (HA) Manager configuration for large cell environments"):

a. Change the value of the IBM_CS_DATASTACK_MEG custom property.
This setting controls the peak amount of dynamic memory that can be used by the HA Manager for caching in-flight messages. The default value for this property is 50 megabytes. The maximum value for this property is 256 megabytes. For large topologies, the default setting is inefficient and should be increased. A setting of 100 megabytes is sufficient for most large topologies. If the core group size is large (50 members or more) and memory-to-memory replication is enabled, consider increasing the setting to 200.

b. Change the size of the transport buffer.
The transport buffer size controls the peak amount of memory that can be used by messages that have been sent and are waiting to have their transmission acknowledged. It is recommended that the transport buffer size be configured to be the same size as the datastack memory size above. The setting for the transport buffer size should be the same for all processes in the core group.

See topic Configuring a core group for replication for instructions.

Back to Top



(2) HMGR0152W

This message does not imply that HAManager is causing high CPU. The HAManager threads scheduling is delayed due to lack of CPU cycles. The CPU cycles might be blocked by High CPU, GC or I/O operation(Memory Swap). Analyze VerboseGC, CPU Stats (vmstat, sar), and so on.

For more information, refer to technote HMGR0152W: CPU Starvation detected messages in SystemOut.log

Back to Top


(3) CWWDR0010E

This message is caused by HAmanager view instability. DCSV1111W and DCSV1113W messages are seen showing HAManager connections closed. The above two situations can cause this problem.

Also, there is a memory leak in the DRSGROUPS. This leak might not be huge but certainly can cause heap fragmentation in 1.4.2.x JDK. This is just a membership cache that grows unbounded (supposed to have only few entries):

293,324,864 (22%) [72] 11 com/ibm/ws/drs/model/DRSGroups 0x1ccd92e8
171,044,456 (12%) [48] 1 java/util/HashMap 0x1cc83178
171,044,408 (12%) [80] 3 array of java/util/HashMap$Entry 0x1cc83020
66,863,200 (5%) [32] 2 java/util/HashMap$Entry 0x38f37b10
66,863,056 (5%) [24] 1 java/util/ArrayList 0x24a18ea8
66,863,032 (5%) [ 20,357,528] 4,491,849 array of java/lang/Object 0x591c5460
1,440 (0%) [32] 1 java/util/Collections$UnmodifiableMap 0x383edad0
1,440 (0%) [32] 1 java/util/Collections$UnmodifiableMap 0x383edad0
1,440 (0%) [32] 1 java/util/Collections$UnmodifiableMap 0x383edad0
1,440 (0%) [32] 1 java/util/Collections$UnmodifiableMap 0x383f0310
1,440 (0%) [32] 1 java/util/Collections$UnmodifiableMap 0x383f0310

To resolve this memory leak, install PK77234. The fix for this APAR is currently targeted for inclusion in WebSphere Application Server fix packs 6.0.2.35, 6.1.0.23, and 7.0.0.5.

Back to Top


(4) Other suggested WebSphere Application Server DRS cache replication configuration

See Dynamic cache service settings and verify Enable cache replication is selected and the Replication type is Not Shared for all WebSphere Portal application servers.

See Object cache instance settings and verify Enable cache replication is selected and the Replication type is Not Shared for all object cache instances:
  • abspath
  • abspathreverse
  • global
  • libparent
  • menu
  • module
  • nav
  • processing
  • session
  • simple_sdo
  • site
  • strategy
  • summary

See Servlet cache instance settings and verify Enable cache replication is selected and the Replication type is Not Shared for all servlet cache instances.

Install the latest Member Manager cumulative fix for WebSphere Portal version 6.0, PK62457 (WMM CACHE CHANGED FROM SHARED_PUSH TO SHARED_NONE).

Install WebSphere Portal PK64925 (sharing policy for baseCache is set to SHARED_PULL instead to NOT_SHARED) available from Fix Central.

See Java virtual machine custom properties.

Install WebSphere Application Server PK15310 (included in WebSphere Application Server 6.0.2.9 and higher) and configure JVM custom property:

com.ibm.ws.cache.CacheConfig.useServerClassLoader=true

Install WebSphere Application Server PK24146 (included in WebSphere Application Server 6.0.2.13 and higher) and configure JVM custom properties:

com.ibm.ws.cache.CacheConfig.filterLRUInvalidation=true
com.ibm.ws.cache.CacheConfig.filterTimeOutInvalidation=true

Install WebSphere Application Server PK65550 (included in WebSphere Application Server 6.0.2.31 and higher)

Install WebSphere Application Server PK77234 (included in WebSphere Application Server 6.0.2.35 and higher)

Back to Top

Rate this page:

(0 users)Average rating

Add comments

Document information


More support for:

WebSphere Portal
General

Software version:

6.0, 6.1

Operating system(s):

AIX, HP-UX, Linux, Solaris, Windows, i5/OS, z/OS

Software edition:

Enable, Express, Extend, Server

Reference #:

1322640

Modified date:

2014-01-15

Translate my page

Machine Translation

Content navigation