IT20212: VERY SLOW CRASH RECOVERY CAUSED BY DB2 APAR IT18691

8.1.2.000-IBM-SPSRV-AIX
8.1.2.000-IBM-SPSRV-Linuxppc64le
8.1.2.000-IBM-SPCMS-Linuxx86_64
8.1.2.000-IBM-SPCMS-WindowsI32
8.1.2.000-IBM-SPSRV-Linuxs390x
8.1.2.000-IBM-SPCMS-WindowsX64
8.1.2.000-IBM-SPSRV-Linuxx86_64
8.1.2.000-IBM-SPSRV-WindowsX64
8.1.2.000-IBM-SPOC-AIX
8.1.2.000-IBM-SPOC-LinuxPPC64le
8.1.2.000-IBM-SPOC-Linuxs390x
8.1.2.000-IBM-SPOC-Linuxx86_64
8.1.2.000-IBM-SPOC-WindowsX64
7.1.8.000-TIV-TSMSRV-AIX
7.1.8.000-TIV-TSMSRV-HP-UX
7.1.8.000-TIV-TSMSRV-Linuxppc64
7.1.8.000-TIV-TSMSRV-Linuxs390x
7.1.8.000-TIV-TSMSRV-Linuxx86_64
7.1.8.000-TIV-TSMSRV-SolarisSPARC
7.1.8.000-TIV-TSMSRV-WIN
7.1.8.000-TIV-TSMOC-AIX
7.1.8.000-TIV-TSMOC-Linuxx64
7.1.8.000-TIV-TSMOC-Windows
7.1.8.000-TIV-TSMOC-LinuxPPC64
7.1.8.000-TIV-TSMOC-LinuxPPC64le
7.1.8.000-TIV-TSMOC-LinuxS390
IBM Spectrum Protect Server V8.1 Fix Pack 2 (V8.1.2) Downloads
IBM Spectrum Protect Server V7.1 Fix Pack 8 (7.1.8.000) Downloads

APAR status

• Closed as program error.

Error description

• DB2 APAR IT18691 can cause very slow rollback.  This issue can
  happen When a log record needed for rollback is not available in
   a memory buffer, the agent performing the rollback needs to
  retrieve this from disk. Due to poor caching, this can lead to a
   bottleneck with many reads having to be performed by the
  db2loggr.
  
   .
  
  L2/customer diagnostics:
  
   Spectrum Protect users can see this happen  when some very
  large transactions got cut because of num_log_span, or cancel
  process/session.  the  rollback of such transaction can be very
  slow.  The same slowness can also be observed during  crash
  recovery
  
  When the slow rollback is observed,  execute the following
  command as instance owner:
  
  db2pd -latches | grep SQLO_LT_SQLP_DBCB__ReadBufSem
  0x0A00020000061940 793497     273710     sqlpgrlg.C 1553
  SQLO_LT_SQLP_DBCB__ReadBufSem 1
  0x0A00020000061940 793497     110666     sqlpgrlg.C 1553
  SQLO_LT_SQLP_DBCB__ReadBufSem 1
  0x0A00020000061940 793497     471206     sqlpgrlg.C 1553
  SQLO_LT_SQLP_DBCB__ReadBufSem 1
  0x0A00020000061940 793497     641753     sqlpgrlg.C 1553
  SQLO_LT_SQLP_DBCB__ReadBufSem 1
  
  
  3rd column of above output present the number of waiters , a
  high value of waiter hint the  latch contention
  .
  Platform /Version affected:
  IBM Spectrum protect users on all supported platform and
  versions
  .
  Additional keywords
  rollback slow  IT18691
  

Local fix

• preventing very slow long running transaction can help to reduce
   the in-flight transaction size and preventing the situation
  from happening.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All Tivoli Storage Manager and IBM Spectrum Protect server   *
  * users.                                                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 7.1.8 and 8.1.2. Note that   *
  * this is subject to change at the discretion of IBM.          *
  ****************************************************************
  

Problem conclusion

• Problem Conclusion:
  This problem was fixed.
  
  Affected platforms: AIX, Linux, Windows, HP-UX, and Solaris.
  Platforms fixed: AIX, Linux, Windows, HP-UX, and Solaris.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels