IBM Support

JR47596: ADDING or REMOVING LARGE (LDAP) GROUPS TO/FROM PARTICIPANT GROUPS TAKES VERY LONG.

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • Process Inspector allows you to addor remove groups to/from a
    Participant Group by editing Team Bindings (Role bindings page
    in BPM Process Admin console).
    When adding/removing a large LDAP based group containing many
    users (several 10k users), the system attempts to load the
    members of the group from LDAP which results in very long
    processing times.
    Java core thread dumps taken during the time of loading the
    role bindings page makes many calls (threads) which contain
    com/ibm/ws/security/registry/UserRegistryImpl.getUsersForGroup(U
    serRegistryImpl.java:1094), which is called from
    com/lombardisoftware/server/core/cache/GroupMemberCache.loadCach
    eData
       This can also occur during other operations that need to
    load the group cache.  For example, the call for a login can be
    affected by this same issue.  A login invokes a method called
    initializeNewLogin().
        Due to this problem, the login took 36 seconds to complete,
    it started at  9:21:45:912 and ended at: 9:22:21:526.
    Most of the time in that block was caused by a large number of
    ldap calls which also had slow response time.
    Here's an example of one of the LDAP calls that was made.
    Start of ldapsearch which takes 1.2 seconds to complete.
    [11/15/14 9:21:52:877 SGT] 0000011b LdapConnectio >
    com.ibm.ws.wim.adapter.ldap.LdapConnection searchEntities ENTRY
    ou=groups,dc=ibm,dc=com,dc=sg
    (&(objectClass=groupofnames)(member=cn=myid_mo
    npaym,ou=groups,dc=ibm,dc=com,dc=sg)) null 2 [Group] [cn] false
    false
    Return of ldapsearch:
    [11/15/14 9:21:54:026 SGT] 0000011b LdapConnectio <
    com.ibm.ws.wim.adapter.ldap.LdapConnection searchEntities
    RETURN []
        The java stack of an affected call might look like this in
    a javacore
    3XMTHREADINFO      "WebContainer : 0"
    J9VMThread:0x0000000006D3E000, j9thread_t:0x000003FFCB963D90,
    java/lang/Thread:0x00000000A7E3C748, state:CW, prio=5
    3XMJAVALTHREAD            (java/lang/Thread getId:0x16E,
    isDaemon:true)
    3XMTHREADINFO1            (native thread ID:0x8C20, native
    priority:0x5, native policy:UNKNOWN)
    3XMTHREADINFO2            (native stack address range
    from:0x000003FFBA47E000, to:0x000003FFBA4BE000, size:0x40000)
    3XMCPUTIME               CPU usage total: 144.374163222 secs
    3XMHEAPALLOC             Heap bytes allocated since last GC
    cycle=617408 (0x96BC0)
    3XMTHREADINFO3           Java callstack:
    4XESTACKTRACE                at
    javax/naming/directory/BasicAttribute.contains(BasicAttribute.ja
    va:299)
    4XESTACKTRACE                at
    javax/naming/directory/BasicAttribute.add(BasicAttribute.java:27
    3)
    4XESTACKTRACE                at
    com/ibm/ws/wim/adapter/ldap/LdapConnection.supportRangeAttribute
    s(LdapConnection.java:3940)
    4XESTACKTRACE                at
    com/ibm/ws/wim/adapter/ldap/LdapConnection.getAttributesByExtId(
    LdapConnection.java:2595(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/wim/adapter/ldap/LdapConnection.getEntityByIdentifier
    (LdapConnection.java:2903(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/wim/adapter/ldap/LdapConnection.getEntityByIdentifier
    (LdapConnection.java:2825(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/wim/adapter/ldap/LdapAdapter.get(LdapAdapter.java:159
    8(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/wim/ProfileManager.getImpl(ProfileManager.java:1747(C
    ompiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/wim/ProfileManager.genericProfileManagerMethod(Profil
    eManager.java:365(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/wim/ProfileManager.get(ProfileManager.java:418(Compil
    ed Code))
    4XESTACKTRACE                at
    com/ibm/websphere/wim/ServiceProvider.get(ServiceProvider.java:4
    09(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/wim/registry/util/MembershipBridge.getUsersForGroup(M
    embershipBridge.java:426)
    4XESTACKTRACE                at
    com/ibm/ws/wim/registry/WIMUserRegistry$16.run(WIMUserRegistry.j
    ava:1260)
    4XESTACKTRACE                at
    com/ibm/ws/security/auth/ContextManagerImpl.runAs(ContextManager
    Impl.java:5474(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/security/auth/ContextManagerImpl.runAsSystem(ContextM
    anagerImpl.java:5600(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/wim/security/authz/jacc/JACCSecurityManager.runAsSupe
    rUser(JACCSecurityManager.java:438(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/wim/env/was/JACCAuthorizationService.runAsSuperUser(J
    ACCAuthorizationService.java:1086(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/wim/security/authz/ProfileSecurityManager.runAsSuperU
    ser(ProfileSecurityManager.java:285(Compiled Code))
    4XESTACKTRACE                at
    com/ibm/ws/wim/registry/WIMUserRegistry.getUsersForGroup(WIMUser
    Registry.java:1251)
    4XESTACKTRACE                at
    com/ibm/ws/security/registry/UserRegistryImpl.getUsersForGroup(U
    serRegistryImpl.java:1094)
    4XESTACKTRACE                at
    com/ibm/websphere/security/_UserRegistry_Stub.getUsersForGroup(_
    UserRegistry_Stub.java:900)
    4XESTACKTRACE                at
    com/lombardisoftware/userorg/WSAbstractUserRegistryModule.getRol
    eMembers(WSAbstractUserRegistryModule.java:376)
    4XESTACKTRACE                at
    com/lombardisoftware/userorg/AbstractAccessControllerManager.get
    RoleMembers(AbstractAccessControllerManager.java:258)
    4XESTACKTRACE                at
    com/lombardisoftware/userorg/UserOrgModule.getRoleMembers(UserOr
    gModule.java:500)
    4XESTACKTRACE                at
    com/lombardisoftware/server/core/cache/GroupMemberCache.getMembe
    rs(GroupMemberCache.java:87)
    4XESTACKTRACE                at
    com/lombardisoftware/server/core/cache/GroupMemberCache.loadCach
    eData(GroupMemberCache.java:68)
    4XESTACKTRACE                at
    com/lombardisoftware/server/core/cache/GroupMemberCache.loadCach
    eData(GroupMemberCache.java:23)
    4XESTACKTRACE                at
    com/lombardisoftware/core/cache/GenericCache.getCacheData(Generi
    cCache.java:189(Compiled Code))
    4XESTACKTRACE                at
    com/lombardisoftware/server/core/GroupCore.accessGroupMemberCach
    e(GroupCore.java:681)
    4XESTACKTRACE                at
    com/lombardisoftware/server/core/GroupCore.getDirectUserGroupMem
    berIds(GroupCore.java:671)
    

Local fix

Problem summary

  • When adding a large LDAP based group (containing thousands of
    users) to a participant group definition using Process Admin
    Console or Process Designer the system apparently hangs. Along
    a separate line, high concurrent user login rates per minute
    are expected, but seem to be limited when employing an LDAP
    directory.
    
    PROBLEM DETAILED DESCRIPTION:
    An in-memory Group Member Cache is employed by the system to
    keep track of the members of a group. Upon using a group, e.g.
    for adding it to a Participant Group, the members of the group
    are retrieved from both LDAP and DB. For large groups, LDAP
    access for retrieving group members is in the range of many
    minutes making the system appear to hang. In case LDAP
    connection time-out values are not high enough, the connection
    may terminate without data retrieval. In addition, the Group
    Member Cache does not have a per group reset mechanism.
    Instead, it resets the whole cache content implying frequent
    cache re-loads.
    
    During user login, the group memberships of the user logging in
    are retrieved from the LDAP server and refreshed in the BPM DB.
    The retrieval makes use of the WebSphere UserRegistry interface
    implying multiple LDAP queries (one per group member). In highly
    concurrent situations, this can lead to a high number of LDAP
    queries being executed concurrently.
    

Problem conclusion

  • A new configuration setting is defined allowing to specify that
    the members of a group are to be determined from the DB only.
    Include in 100Custom.xml (in a cluster, on the Deployment
    Manager for every server of the cluster) the following setting:
    <server>
     <group-member-cache-source>DB</group-member-cache-source>
    </server>
    The file is located at
      <install-root>/profiles/<profile>/config/cells/<cell>/nodes/
       <node>/servers/<server>/process-center|
       process-server/config.
    
    The setting avoids LDAP access. As a result, the speed of
    loading group members into the cache is dramatically increased.
    In addition, a mechanism for resetting the cache for a specific
    entry (i.e. group) is included, thus avoiding frequent cache
    re-loads.
    
    Additionally, the above work around resolves Performance issues
    for customers who face long wait times for Process Portal Login
    and Upgrading of Toolkit dependencies for BPM Applications.
    
    A companion APAR will be provided to complement above setting.
    The APAR includes administrative scripts by which bulk
    synchronization can be done for users and their group
    memberships between LDAP and BPM DB.
    
    The user login rate is increased by using Federated Repositories
    (aka VMM) calls instead of the WAS UserRegistry interface. The
    optimization requires that the LDAP server configured for
    Federated Repositories supports for a user entry an attribute
    exposing the list of groups the user is member of. For
    instance, the Tivoli LDAP Directory Server supports the
    ibm-allGroups attribute for this purpose.  This attribute is
    exploited to determine the set of groups a user is member of
    within one LDAP query.
    
    NOTE: employ the login optimization only, if all required group
    memberships of a user can be computed via the the additional VMM
    property "memberof" (see below). This is the case for LDAP
    directories appropriately configured for VMM. For other
    repositories this may not hold. For instance, groups defined in
    the VMM file registry are not reflected via the "memberof"
    property.
    
    Assuming your set-up is configured for Federated Repositories
    and the attached LDAP directory exposes the ibm-allGroups or a
    similar attribute, apply the following steps to optimize group
    retrieval:
    
    1. Extend the VMM entity type PersonAccount to include an
    additional property with name "memberof". For this, include a
    file wimxmlextension.xml (in a cluster, on the Deployment
    Manager for every server of the cluster) at the location
    <install-root>/profiles/<profile>/config/cells/<cell>/wim/model
    
    The file is to contain the extension definition:
    <sdo:datagraph xmlns:sdo="commonj.sdo"
        xmlns:wim="http://www.ibm.com/websphere/wim">
      <wim:schema>
       <wim:propertySchema nsURI="http://www.ibm.com/websphere/wim"
        dataType="STRING" multiValued="true"
        propertyName="memberof">
        <wim:applicableEntityTypeNames>PersonAccount
        </wim:applicableEntityTypeNames>
       </wim:propertySchema>
      </wim:schema>
    </sdo:datagraph>
    
    2. Define the mapping between the VMM property name "memberof"
    and the available LDAP attribute, e.g. "ibm-allGroups".  For
    this, include in
    <install-root>/profiles/<profile>/config/cells/<cell>/wim/config
    /wimconfig.xml
    (in a cluster, on the Deployment Manager for every server of the
    cluster) the entry:
    
     <config:repositories xsi:type="config:LdapRepositoryType" ...>
      ...
      <config:attributeConfiguration>
      ...
      <config:attributes name="ibm-allGroups"
    propertyName="memberof">
              <config:entityTypes>PersonAccount</config:entityTypes>
            </config:attributes>
      ...
          </config:attributeConfiguration>
        </config:repositories>
    
    3. Tune your LDAP configuration in wimconfig.xml to allow for
    potentially retrieving all groups in one VMM query.
    Consult the VMM tuning documents. In particular, select an
    appropriate setting for configurationProvider->maxSearchResults.
    
    4. Enable using the "memberof" property by BPM.
    Include in 100Custom.xml (in a cluster, on the Deployment
    Manager for every server of the cluster):
    <common merge="mergeChildren">
     <security>
      <vmm-options>
       <member-of-group-prop>memberof</member-of-group-prop>
      </vmm-options>
     </security>
    </common>
    The file is located at
    <install-root>/profiles/<profile>/config/cells/<cell>/nodes/<nod
    e>/servers/<server>/process-center|process-server/config.
    
    FIX AVAILABILITY:
    <on which releases is an iFix available  and in which fixpacks
    is this going to be>
    iFix for 8.0.1.1 is available on Fix Central, search for APAR
    JR47596 at http://www.ibm.com/support/fixcentral/
    iFix for 8.0.1.2 is available on Fix Central, search for APAR
    JR47596 at http://www.ibm.com/support/fixcentral/
    iFix for 8.5.0.1 is available on Fix Central, search for APAR
    JR47596 at http://www.ibm.com/support/fixcentral/
    Fix is also targetted for inclusion in next fixpack for BPM
    8.0.1
    
    When obtaining any of the above fixes, be sure to download the
    accompanying readme, for itself, and any prerequisite fixes, and
    review them thorougly.
    

Temporary fix

  • Not applicable
    

Comments

APAR Information

  • APAR number

    JR47596

  • Reported component name

    BPM STANDARD

  • Reported component ID

    5725C9500

  • Reported release

    801

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2013-08-22

  • Closed date

    2015-08-17

  • Last modified date

    2015-08-17

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    BPM STANDARD

  • Fixed component ID

    5725C9500

Applicable component levels

  • R801 PSY

       UP

  • R850 PSY

       UP



Document information

More support for: IBM Business Process Manager Standard

Software version: 8.0.1

Reference #: JR47596

Modified date: 17 August 2015