IBM Support

IT17928: SERVER CRASHES WITH SEGMENTATION FAULT AFTER IMPLEMENTING OPERATION CENTER.

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • After implementing the Operation Center, the Tivoli Storage
    Manager server can crash over time as millions of alerts
    are stored in the server database. The core dump will be similar
    to the following:
    .
    Program terminated with signal 11, Segmentation fault.
    #0  0x00007ffff41b7ac0 in memcpy () from /lib64/libc.so.6
    .
    ===================================================
    .
    #0  memcpy () from /lib64/libc.so.6
    #1  GrowBuf (bufP=<value optimized out>,
          bufLeftP=0x7fffe8f15c4c, bufUsedP=0x7fffe8f15c48,
          fmt=0x131771e ",") at outjson.c:285
    #2  jsonAddToBuf (bufP=<value optimized out>,
          bufLeftP=0x7fffe8f15c4c, bufUsedP=0x7fffe8f15c48,
          fmt=0x131771e ",") at outjson.c:339
    #3  jsonAddIntToBuf (bufP=0x7fffd4c68f98 "\":\"\",\"msgexpl\":
          \"\",\"msgsysa\":\"\",\"msguresp\":\"\"},{\"TSMJSONID\":
          1,\"servername\":\"PSBRBRML90\",\"alertid\":11744,\""
          <Address 0x7fffd4c69000 out of bounds>,
          bufLeftP=0x7fffe8f15c4c, bufUsedP=0x7fffe8f15c48,
          name=0x11c2b75 "lasttime_sec", value=1476378555,
          comma=True) at outjson.c:523
    #4  BuildAlertJSON (alertP=0x656d616e72657672 <Address
          0x656d616e72657672 out of bounds>, bufLeftP=0x7fffe8f15c4c
          bufUsedP=0x7fffe8f15c48, infoP=0x7fffd4b996f8) at
          monalert.c:8313
    #5  DoQueAlertSummary () at monalert.c:8794
    #6  AlertMonitorThread (notused=<value optimized out>) at
        monalert.c:5882
    #7  StartThread (startInfoP=0x0) at pkthread.c:3436
    #8  start_thread () from /lib64/libpthread.so.0
    #9  clone () from /lib64/libc.so.6
    .
    Initial Impact:
    Medium
    .
    Tivoli Storage Manager Versions Affected:
    Applies to all 7.1 server versions.
    .
    Additional Keywords:
    Crash core dump Segmentation fault signal 11 OC Operation Center
    

Local fix

  • To prevent the server from crashing until the server can be
    upgraded to the patch level, use the following instructions:
    .
    1. Turn off the Operation Center web server service
    2. SET ALERTMONITOR off
    3. Restart the server
    .
    Reduce the number of alerts you have on your server.
    Use QUERY ALERTSTATUS to view the current alerts you have.
    Use SET ALERTACTIVEDURATION, SET ALERTINACTIVEDURATION,
    and SET ALERTCLOSEDURATION to shorten the amount of time
    alerts are kept on your server.
    .
    For example:
    To delete existing inactive and closed alerts
    you can configure your server as follows. This will
    have the effect of:
      - Inactive alerts change to closed status after
        approximately one minute.
      - Closed alerts deleted immediately.
    .
    1.  Set the Operation Center Web server service off
    2.  Set the alert monitor off temporarily on Spoke server:
              SET ALERTMONITOR OFF
    3.  Set the duration to keep 'inactive' alerts to keep for the
            minimum setting of 1 minute:
              SET ALERTINACTIVEDUR 1
    4.  Set the duration to keep 'closed' alerts to zero minutes:
              SET ALERTCLOSEDDUR 0
    

Problem summary

  • ****************************************************************
    * USERS AFFECTED:                                              *
    * All IBM Tivoli Storage Manager and IBM Spectrum Protect      *
    * server users.                                                *
    ****************************************************************
    * PROBLEM DESCRIPTION:                                         *
    * See error description.                                       *
    ****************************************************************
    * RECOMMENDATION:                                              *
    * Apply fixing level when available. This problem is currently *
    * projected to be fixed in levels 7.1.8 and 8.1.1. Note that   *
    * this is subject to change at the discretion of IBM.          *
    ****************************************************************
    

Problem conclusion

  • This problem was fixed.
    Affected platforms for reported release:  AIX, HP-UX, Solaris,
    Linux, and Windows.
    Platforms fixed:  AIX, HP-UX, Solaris, Linux, and Windows.
    

Temporary fix

Comments

APAR Information

  • APAR number

    IT17928

  • Reported component name

    TSM SERVER

  • Reported component ID

    5698ISMSV

  • Reported release

    71L

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt / Xsystem

  • Submitted date

    2016-11-12

  • Closed date

    2016-12-09

  • Last modified date

    2016-12-09

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    TSM SERVER

  • Fixed component ID

    5698ISMSV

Applicable component levels

  • R71A PSY

       UP

  • R71H PSY

       UP

  • R71L PSY

       UP

  • R71S PSY

       UP

  • R71W PSY

       UP

  • R81A PSY

       UP

  • R81L PSY

       UP

  • R81W PSY

       UP



Document information

More support for: Tivoli Storage Manager

Software version: 7.1.3

Reference #: IT17928

Modified date: 09 December 2016