IBM Support

JR39837: CMVC 209154 - IMPROVE THE PROCESSEVENT JOB TO PROCESS EVENT FASTER

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • The three scheduler jobs used for the ECEvent framework have
    been updated to process the ECEvents more efficiently:
    RaiseECEvent, EventProcessor and CleanECEvent.
    
    Without this APAR, after an event is created and raised with the
    RaiseECEvent job, the event is
    set to -10 state (BUSEVENT.CHECKED column). -10 events are
    processed further with the EventProcessor command, which has
    a default batch size of 200 (configurable in wc-server.xml).
    After the events are processed by the EventProcessor,
    they are set to -20 state, so they can be cleaned with the
    CleanECEvent job.  Since there are several events enabled by
    default, for example OrderCreation and OrderItemUpdate, the
    BUSEVENT grows very large and the EventProcessor job is not able
    to catch up.
    
    With the fix associated to this APAR, enhancements are made to
    the three commands so that they update and delete rows in the
    BUSEVENT table using multiple transactions, to allow it to
    update and delete millions of records, over time, without
    running into transaction time-outs or database locking.
    
    RaiseECEvent:
    -------------
    
    This scheduled job, that runs by default every five minutes,
    raises a common base event for each of the events in the
    BUSEVENT table that
    have not been  raised (CHECKED = 0). Raising the  event will
    notify the registered event listeners. After the event is
    raised, the CHECKED
    column is updated to -10.
    
    It is the responsibility of the EventProcessor scheduled job to
    do further processing of events in -10 state:  analytics events
    are sent to
    an external analytics system and then updated to -20 state,
    while non-analytics events are simply updated to the -20 state.
    Finally, the
    CleanECEvent job deletes the events with CHECKED = -20 from the
    BUSEVENT table.
    
    As an optimization, if there are no analytics events that need
    to be processed by EventProcessor, the "Analytics Event
    Configuration" component
    can be disabled in the instance configuration file and the
    EventProcessor scheduled job can be deleted.  Then the parameter
    "noEventProcessing=true"
    must be passed to the RaiseECEvent command so that events are
    placed in state -20 after they are raised. A maximum of 5000
    events are changed from state
    -10 to -20, per command execution.
    
    To determine if EventProcessor is sending analytics events to an
    external analytics system, look at the "Analytics Even
    Configuration" in the instance
    configuration file.  If the "enable" attribute of the component
    node is set to false, or if there are no stores enabled, the
    Analytics component is not
    being used.
    
    <component
    compClassName="com.ibm.commerce.bi.events.AnalyticsComponentConf
    iguration"
    enable="true"
    name="Analytics Event Configuration">
    <property display="false">
     <supportedEvents inProcessState="-15"
      mapper="mapCommonEvents.xml"
      maxBatchSize="200"
      maxProcessTime="86400"
      processState="-10"
      purgeState="-20">
     ...
     <stores>
      <store enabled="false" storeId="-1">
       <configuration cmClientID=""
        password=""
        serviceURL="https://wscreceiver.coremetrics.com/Receiver/sen
    dEventData"
        sslKeyPassphrase="WebAS"
        sslKeyStore="/usr/WebSphere/AppServer/profiles/demo/etc/Dumm
    yServerKeyFile.jks"
        sslTrustPassphrase="WebAS"
        sslTrustStore="/usr/WebSphere/AppServer/profiles/demo/etc/Du
    mmyServerTrustFile.jks"
        transmitClassName="com.ibm.commerce.bi.events.transmit.CMWeb
    ServiceTransmitter"
        transmitEnabled="false"
       username="" />
      </store>
     </stores>
    </property>
    </component>
    
    EventProcessor:
    ---------------
    This scheduled job, that runs by default every hour, does
    further processing on analytics events that have been
    raised and persisted in the BUSEVENT table so they can be
    transmitted in a batch to an external
    (non-WebSphere Commerce) analytics system.
    
    If the "Analytics Event Configuration" component is enabled, 200
    events are fetched and updated from
    'process' state (default -10) to 'inProcess' state (default
    -15). Any qualifying analytics events (events
    for stores that are enabled, with the correct type, transport
    and channel ID's) in this batch are transformed
    and then transmitted to the analytics system, and then updated
    to purge state.  Any non-qualifying events in
    this batch are  updated to purge state (default -20). This
    process of fetching and processing 200 events is
    repeated until the maxBatchSize analytics events are processed.
    If maxBatchSize is set to -1, then the batching
    will continue until all the events in the BUSEVENT table are
    processed.
    
    If the "Analytics Event Configuration" component is disabled or
    its stores are disabled, up 500,000 events in the
    process state (-10) are updated to be in the purge state
    (default -20). These are committed in batches of 5,000.
    
    CleanECEvent:
    ---------------
    This scheduled job, that runs by default every 30 minutes,
    deletes up to the specified maximum events from the BUSEVENT
    table that are in the purge state and are older than the
    specified expire time (default 30 minutes). Records are deleted
    and committed in batches of 5,000. The default maximum is
    500,000.
    
    If "Analytics Event Configuration" component is enabled, any
    events that are older than maxProcessTime (default 1 day) and
    stuck
    in the -15 state are re-activated by changing them back to -10
    state, so that they can be processed by the EventProcessor job.
    
    Before V6 FEP2, the default BUSEVENT.CHECKED value to indicate
    an event is ready to be purged was -10.  However, as of V6 FEP2
    with
    the cm-salescenter feature and in V7 base, the scheduler
    configuration  of the CleanECEvent job has been changed to
    specify that -20
    is the purge state:  SCHCONFIG.SCCQUERY='purgeState=-20'. This
    was done so that the EventProcessor scheduled job could do
    further
    processing of -10 events, transmit the data to an external
    analytics system, and then put the event in -20 state.
    

Local fix

Problem summary

  • USERS AFFECTED:
    WebSphere Commerce 7.0 users
    
    PROBLEM ABSTRACT:
    Performance improvements for the ECEvent processing framework
    (BUSEVENT table)
    
    BUSINESS IMPACT:
    Without this APAR, and using the default configuration for the
    Business Event framework, the rows in the BUSEVENT table will
    grow unbounded. Once the table reaches a large size, it is
    possible for the site to experience high memory utilization or
    database locking problems resulting from the Event Framework
    commands trying to operate on millions of rows
    
    RECOMMENDATION:
    

Problem conclusion

  • Install the iFix associated to this APAR to obtain the code
    changes which make the Event Processor framework
    more efficient as described above
    
    -------------------------------------------------------------
    The latest available maintenance information can be obtained
    from the Recommended Fixes for WebSphere Commerce technote:
    http://www.ibm.com/support/docview.wss?rs=3046&uid=swg21261296
    

Temporary fix

Comments

APAR Information

  • APAR number

    JR39837

  • Reported component name

    WC BUS EDITION

  • Reported component ID

    5724I3800

  • Reported release

    700

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    YesSpecatt / Pervasive

  • Submitted date

    2011-06-13

  • Closed date

    2011-08-09

  • Last modified date

    2011-08-09

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WC BUS EDITION

  • Fixed component ID

    5724I3800

Applicable component levels

  • R700 PSY

       UP

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSYSYL","label":"WebSphere Commerce Enterprise"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0","Edition":"","Line of Business":{"code":"","label":""}}]

Document Information

Modified date:
09 August 2011