Direct links to fixes
APAR status
Closed as program error.
Error description
Process Inspector allows you to addor remove groups to/from a Participant Group by editing Team Bindings (Role bindings page in BPM Process Admin console). When adding/removing a large LDAP based group containing many users (several 10k users), the system attempts to load the members of the group from LDAP which results in very long processing times. Java core thread dumps taken during the time of loading the role bindings page makes many calls (threads) which contain com/ibm/ws/security/registry/UserRegistryImpl.getUsersForGroup(U serRegistryImpl.java:1094), which is called from com/lombardisoftware/server/core/cache/GroupMemberCache.loadCach eData This can also occur during other operations that need to load the group cache. For example, the call for a login can be affected by this same issue. A login invokes a method called initializeNewLogin(). Due to this problem, the login took 36 seconds to complete, it started at 9:21:45:912 and ended at: 9:22:21:526. Most of the time in that block was caused by a large number of ldap calls which also had slow response time. Here's an example of one of the LDAP calls that was made. Start of ldapsearch which takes 1.2 seconds to complete. [11/15/14 9:21:52:877 SGT] 0000011b LdapConnectio > com.ibm.ws.wim.adapter.ldap.LdapConnection searchEntities ENTRY ou=groups,dc=ibm,dc=com,dc=sg (&(objectClass=groupofnames)(member=cn=myid_mo npaym,ou=groups,dc=ibm,dc=com,dc=sg)) null 2 [Group] [cn] false false Return of ldapsearch: [11/15/14 9:21:54:026 SGT] 0000011b LdapConnectio < com.ibm.ws.wim.adapter.ldap.LdapConnection searchEntities RETURN [] The java stack of an affected call might look like this in a javacore 3XMTHREADINFO "WebContainer : 0" J9VMThread:0x0000000006D3E000, j9thread_t:0x000003FFCB963D90, java/lang/Thread:0x00000000A7E3C748, state:CW, prio=5 3XMJAVALTHREAD (java/lang/Thread getId:0x16E, isDaemon:true) 3XMTHREADINFO1 (native thread ID:0x8C20, native priority:0x5, native policy:UNKNOWN) 3XMTHREADINFO2 (native stack address range from:0x000003FFBA47E000, to:0x000003FFBA4BE000, size:0x40000) 3XMCPUTIME CPU usage total: 144.374163222 secs 3XMHEAPALLOC Heap bytes allocated since last GC cycle=617408 (0x96BC0) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at javax/naming/directory/BasicAttribute.contains(BasicAttribute.ja va:299) 4XESTACKTRACE at javax/naming/directory/BasicAttribute.add(BasicAttribute.java:27 3) 4XESTACKTRACE at com/ibm/ws/wim/adapter/ldap/LdapConnection.supportRangeAttribute s(LdapConnection.java:3940) 4XESTACKTRACE at com/ibm/ws/wim/adapter/ldap/LdapConnection.getAttributesByExtId( LdapConnection.java:2595(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/wim/adapter/ldap/LdapConnection.getEntityByIdentifier (LdapConnection.java:2903(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/wim/adapter/ldap/LdapConnection.getEntityByIdentifier (LdapConnection.java:2825(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/wim/adapter/ldap/LdapAdapter.get(LdapAdapter.java:159 8(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/wim/ProfileManager.getImpl(ProfileManager.java:1747(C ompiled Code)) 4XESTACKTRACE at com/ibm/ws/wim/ProfileManager.genericProfileManagerMethod(Profil eManager.java:365(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/wim/ProfileManager.get(ProfileManager.java:418(Compil ed Code)) 4XESTACKTRACE at com/ibm/websphere/wim/ServiceProvider.get(ServiceProvider.java:4 09(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/wim/registry/util/MembershipBridge.getUsersForGroup(M embershipBridge.java:426) 4XESTACKTRACE at com/ibm/ws/wim/registry/WIMUserRegistry$16.run(WIMUserRegistry.j ava:1260) 4XESTACKTRACE at com/ibm/ws/security/auth/ContextManagerImpl.runAs(ContextManager Impl.java:5474(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/security/auth/ContextManagerImpl.runAsSystem(ContextM anagerImpl.java:5600(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/wim/security/authz/jacc/JACCSecurityManager.runAsSupe rUser(JACCSecurityManager.java:438(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/wim/env/was/JACCAuthorizationService.runAsSuperUser(J ACCAuthorizationService.java:1086(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/wim/security/authz/ProfileSecurityManager.runAsSuperU ser(ProfileSecurityManager.java:285(Compiled Code)) 4XESTACKTRACE at com/ibm/ws/wim/registry/WIMUserRegistry.getUsersForGroup(WIMUser Registry.java:1251) 4XESTACKTRACE at com/ibm/ws/security/registry/UserRegistryImpl.getUsersForGroup(U serRegistryImpl.java:1094) 4XESTACKTRACE at com/ibm/websphere/security/_UserRegistry_Stub.getUsersForGroup(_ UserRegistry_Stub.java:900) 4XESTACKTRACE at com/lombardisoftware/userorg/WSAbstractUserRegistryModule.getRol eMembers(WSAbstractUserRegistryModule.java:376) 4XESTACKTRACE at com/lombardisoftware/userorg/AbstractAccessControllerManager.get RoleMembers(AbstractAccessControllerManager.java:258) 4XESTACKTRACE at com/lombardisoftware/userorg/UserOrgModule.getRoleMembers(UserOr gModule.java:500) 4XESTACKTRACE at com/lombardisoftware/server/core/cache/GroupMemberCache.getMembe rs(GroupMemberCache.java:87) 4XESTACKTRACE at com/lombardisoftware/server/core/cache/GroupMemberCache.loadCach eData(GroupMemberCache.java:68) 4XESTACKTRACE at com/lombardisoftware/server/core/cache/GroupMemberCache.loadCach eData(GroupMemberCache.java:23) 4XESTACKTRACE at com/lombardisoftware/core/cache/GenericCache.getCacheData(Generi cCache.java:189(Compiled Code)) 4XESTACKTRACE at com/lombardisoftware/server/core/GroupCore.accessGroupMemberCach e(GroupCore.java:681) 4XESTACKTRACE at com/lombardisoftware/server/core/GroupCore.getDirectUserGroupMem berIds(GroupCore.java:671)
Local fix
Problem summary
When adding a large LDAP based group (containing thousands of users) to a participant group definition using Process Admin Console or Process Designer the system apparently hangs. Along a separate line, high concurrent user login rates per minute are expected, but seem to be limited when employing an LDAP directory. PROBLEM DETAILED DESCRIPTION: An in-memory Group Member Cache is employed by the system to keep track of the members of a group. Upon using a group, e.g. for adding it to a Participant Group, the members of the group are retrieved from both LDAP and DB. For large groups, LDAP access for retrieving group members is in the range of many minutes making the system appear to hang. In case LDAP connection time-out values are not high enough, the connection may terminate without data retrieval. In addition, the Group Member Cache does not have a per group reset mechanism. Instead, it resets the whole cache content implying frequent cache re-loads. During user login, the group memberships of the user logging in are retrieved from the LDAP server and refreshed in the BPM DB. The retrieval makes use of the WebSphere UserRegistry interface implying multiple LDAP queries (one per group member). In highly concurrent situations, this can lead to a high number of LDAP queries being executed concurrently.
Problem conclusion
A new configuration setting is defined allowing to specify that the members of a group are to be determined from the DB only. Include in 100Custom.xml (in a cluster, on the Deployment Manager for every server of the cluster) the following setting: <server> <group-member-cache-source>DB</group-member-cache-source> </server> The file is located at <install-root>/profiles/<profile>/config/cells/<cell>/nodes/ <node>/servers/<server>/process-center| process-server/config. The setting avoids LDAP access. As a result, the speed of loading group members into the cache is dramatically increased. In addition, a mechanism for resetting the cache for a specific entry (i.e. group) is included, thus avoiding frequent cache re-loads. Additionally, the above work around resolves Performance issues for customers who face long wait times for Process Portal Login and Upgrading of Toolkit dependencies for BPM Applications. A companion APAR will be provided to complement above setting. The APAR includes administrative scripts by which bulk synchronization can be done for users and their group memberships between LDAP and BPM DB. The user login rate is increased by using Federated Repositories (aka VMM) calls instead of the WAS UserRegistry interface. The optimization requires that the LDAP server configured for Federated Repositories supports for a user entry an attribute exposing the list of groups the user is member of. For instance, the Tivoli LDAP Directory Server supports the ibm-allGroups attribute for this purpose. This attribute is exploited to determine the set of groups a user is member of within one LDAP query. NOTE: employ the login optimization only, if all required group memberships of a user can be computed via the the additional VMM property "memberof" (see below). This is the case for LDAP directories appropriately configured for VMM. For other repositories this may not hold. For instance, groups defined in the VMM file registry are not reflected via the "memberof" property. Assuming your set-up is configured for Federated Repositories and the attached LDAP directory exposes the ibm-allGroups or a similar attribute, apply the following steps to optimize group retrieval: 1. Extend the VMM entity type PersonAccount to include an additional property with name "memberof". For this, include a file wimxmlextension.xml (in a cluster, on the Deployment Manager for every server of the cluster) at the location <install-root>/profiles/<profile>/config/cells/<cell>/wim/model The file is to contain the extension definition: <sdo:datagraph xmlns:sdo="commonj.sdo" xmlns:wim="http://www.ibm.com/websphere/wim"> <wim:schema> <wim:propertySchema nsURI="http://www.ibm.com/websphere/wim" dataType="STRING" multiValued="true" propertyName="memberof"> <wim:applicableEntityTypeNames>PersonAccount </wim:applicableEntityTypeNames> </wim:propertySchema> </wim:schema> </sdo:datagraph> 2. Define the mapping between the VMM property name "memberof" and the available LDAP attribute, e.g. "ibm-allGroups". For this, include in <install-root>/profiles/<profile>/config/cells/<cell>/wim/config /wimconfig.xml (in a cluster, on the Deployment Manager for every server of the cluster) the entry: <config:repositories xsi:type="config:LdapRepositoryType" ...> ... <config:attributeConfiguration> ... <config:attributes name="ibm-allGroups" propertyName="memberof"> <config:entityTypes>PersonAccount</config:entityTypes> </config:attributes> ... </config:attributeConfiguration> </config:repositories> 3. Tune your LDAP configuration in wimconfig.xml to allow for potentially retrieving all groups in one VMM query. Consult the VMM tuning documents. In particular, select an appropriate setting for configurationProvider->maxSearchResults. 4. Enable using the "memberof" property by BPM. Include in 100Custom.xml (in a cluster, on the Deployment Manager for every server of the cluster): <common merge="mergeChildren"> <security> <vmm-options> <member-of-group-prop>memberof</member-of-group-prop> </vmm-options> </security> </common> The file is located at <install-root>/profiles/<profile>/config/cells/<cell>/nodes/<nod e>/servers/<server>/process-center|process-server/config. FIX AVAILABILITY: <on which releases is an iFix available and in which fixpacks is this going to be> iFix for 8.0.1.1 is available on Fix Central, search for APAR JR47596 at http://www.ibm.com/support/fixcentral/ iFix for 8.0.1.2 is available on Fix Central, search for APAR JR47596 at http://www.ibm.com/support/fixcentral/ iFix for 8.5.0.1 is available on Fix Central, search for APAR JR47596 at http://www.ibm.com/support/fixcentral/ Fix is also targetted for inclusion in next fixpack for BPM 8.0.1 When obtaining any of the above fixes, be sure to download the accompanying readme, for itself, and any prerequisite fixes, and review them thorougly.
Temporary fix
Not applicable
Comments
APAR Information
APAR number
JR47596
Reported component name
BPM STANDARD
Reported component ID
5725C9500
Reported release
801
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2013-08-22
Closed date
2015-08-17
Last modified date
2015-08-17
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
BPM STANDARD
Fixed component ID
5725C9500
Applicable component levels
R801 PSY
UP
R850 PSY
UP
[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"SSFTDH","label":"IBM Business Process Manager Standard"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.0.1","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
11 October 2021