IT19859: IBM STERLING FILE GATEWAY BPS STAY ACTIVE FOR EVER WAITING FOR VISIBILITY EVENTWAREHOUSE

APAR status

• Closed as program error.

Error description

• Many BPs are active for hours on one node.
  Thread dumps show the BP threads all waiting (parking) for the
  event framework across the thread dumps:
  
  at sun/misc/Unsafe.park(Native Method)
  at
  java/util/concurrent/locks/LockSupport.park(LockSupport.java:198
  (Compiled Code))
  at
  java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionO
  bject.await(AbstractQueuedSynchronizer.java:2055(Compiled Code))
  at
  java/util/concurrent/ArrayBlockingQueue.put(ArrayBlockingQueue.j
  ava:336(Compiled Code))
  at
  com/sterlingcommerce/fg/visibility/evf/EventWarehouse.storeEvent
  (EventWarehouse.java:114(Compiled Code))
  at
  com/sterlingcommerce/fg/visibility/evf/EventFrameworkListener.ha
  ndleEvent(EventFrameworkListener.java:48(Compiled Code))
  at
  com/sterlingcommerce/woodstock/event/EventProcessor.fireEvent(Ev
  entProcessor.java:325(Compiled Code))
  at
  com/sterlingcommerce/fg/visibility/evf/EventFactoryBase.fireEven
  t(EventFactoryBase.java:128(Compiled Code))
  at
  com/sterlingcommerce/fg/visibility/FGEventFactory.fireRouteEvent
  (FGEventFactory.java:477(Compiled Code))
  at
  com/sterlingcommerce/fg/services/FileGatewayRouteEventService.pr
  imitiveProcessData(FileGatewayRouteEventService.java:76(Compiled
  Code))
  at
  com/sterlingcommerce/fg/services/TransformingRouterService.proce
  ssData(TransformingRouterService.java:79(Compiled Code))
  at
  com/sterlingcommerce/woodstock/workflow/activity/engine/Activity
  EngineHelper.invokeService(ActivityEngineHelper.java:1818(Compil
  ed Code))
  at
  com/sterlingcommerce/woodstock/workflow/activity/engine/Activity
  EngineHelper.nextMainLogic(ActivityEngineHelper.java:631(Compile
  d Code))
  at
  com/sterlingcommerce/woodstock/workflow/activity/engine/Activity
  EngineHelper.next(ActivityEngineHelper.java:362(Compiled Code))
  at
  com/sterlingcommerce/woodstock/workflow/queue/WorkFlowQueueListe
  ner.doWork(WorkFlowQueueListener.java:461(Compiled Code))
  at
  com/sterlingcommerce/woodstock/workflow/queue/WorkFlowQueueListe
  ner.run(WorkFlowQueueListener.java:240(Compiled Code))
  at
  com/sterlingcommerce/woodstock/workflow/queue/WorkFlowQueueListe
  ner.onMessage(WorkFlowQueueListener.java:197(Compiled Code))
  at
  com/sterlingcommerce/woodstock/workflow/queue/WorkFlowQueueListe
  ner.onMessage(WorkFlowQueueListener.java:184(Compiled Code))
  at
  com/sterlingcommerce/woodstock/workflow/queue/wfTransporter.run(
  wfTransporter.java:444(Compiled Code))
  at
  com/sterlingcommerce/woodstock/workflow/queue/BasicExecutor$Work
  er.run(BasicExecutor.java:508(Compiled Code))
  at java/lang/Thread.run(Thread.java:798(Compiled Code))
  
  FileGateway event worker threads were all Parked.
  he last error in the filegatway.log on node 1 shows:
  
  [2017-03-06 16:56:54.472] ERROR EventWarehouse.storeEvent(E) -
  Was interrupted while trying to put an event on the queue:
  Event: (Fg.Visibility.SFGError.0) - Abnormal
      TimeStamp    = 1488764927628
      Message    = Error connecting to FTP server : Response not
  received before specified timeout reached. Closing connections.
      EntityKey    = 201703060298470733724020
      FileGatewayMetadata    =
  201703060230470733724018,201703060285470733724019,20170306029847
  0733724020,FTB-INT-DEV-FTR,IAP_LOOP0004,803_IAP_N1_FTP_LOOP_2017
  0306024705
      DataflowId    = 13769913
      EntityType    = DELIVERY
      FGArrivedFileKey    = 201703060230470733724018
      EventCode    = CUST_0750
      EVFCounter    = 367265
  
  
  Expected behavior: BPs should not remain active for ever.
  

Local fix

• MB / EJ
  RTC 532079
  No workaround
  

Problem summary

• Users Affected:
  All
  
  Problem Description:
  Property "pushEventsToNextQueue" in visibility.properties is
  set to ?false? by default causing the events to be queued up in
  the system.
  
  Platforms Affected:
  All protocol communications using visibility framework
  in 5020500_15
  

Problem conclusion

• Resolution Summary:
  
  A code fix is provided.
  
  
  Delivered In:
  5020500_16
  5020601_8
  5020603_3
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  STR FILE GATEWA

• Fixed component ID

  5725D0700

Applicable component levels