HUB TEMS crashes every few hours and generates coredump. The HUB TEMS log displays the "malloc failed" error.
The below trace message is written in the HUB TEMS default trace log
(4B82C165.0001-14:kfastgcr.c,101,"KO4ST_GetCurrentStatusRecord") malloc failed (4B82C165.0002-14:kfansins.c,381,"NodeStatusInsertProcess") Failed accessing node status index for key: < REPORTING:VAVCENTSQL01:MSS > during node status processing.
(4B82C165.0003-14:kdsvws1.c,2420,"ManageView") ProcessTable TNODESTS Insert Error status = ( 1000 ). SRVR01 ip.spipe:#xx.xxx.xx.xx[ 3660]
(4B82C166.0000-18:kdsrqc1.c,2620,"AccessRowsets") Could not obtain rowset 2, status code = 1000 (4B82C168.0000-31:kfastgcr.c,101,"KO4ST_GetCurrentStatusRecord") malloc failed (4B82C168.0001-31:ko4bkgnd.cpp,480,"BackgroundController::nodeStatusUpdate") TEMS heartbeat insert failed with status 1000
The above error message is due to lack of memory
Resolving the problem
The reason for the HUB TEMS crash is due to lack memory. To resolve the issue, execute the below command:
tacmd getdeploystatus to view the deployment entries and
tacmd cleardeploystatus -a to remove the accumulated deployment status messages maintained in the TEMS table
The current design of Remote Deploy 6.2.1 and greater, retains the completion status queue in memory. Unless that queue is cleared it will continue to increase in size indefinitely. By removing the queue entries that have completed their processing, the memory is released.
If the status queue is cleansed automatically without user intervention, then there would be no way for the user to know the status of their deployment requests since those requests may have already been removed from the queue.
To release the memory, it is advised to cleanse the queue periodically of only successful updates. Leave the failed deployment so that it be attempted again or retried.
Rate this page:
Copyright and trademark information
IBM, the IBM logo and ibm.com are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml.