HUB TEMS crashes every few hours and generates a core dump

The below trace message is written in the HUB TEMS default trace log

(4B82C165.0001-14:kfastgcr.c,101,"KO4ST_GetCurrentStatusRecord") malloc failed (4B82C165.0002-14:kfansins.c,381,"NodeStatusInsertProcess") Failed accessing node status index for key: < REPORTING:VAVCENTSQL01:MSS > during node status processing.
(4B82C165.0003-14:kdsvws1.c,2420,"ManageView") ProcessTable TNODESTS Insert Error status = ( 1000 ). SRVR01 ip.spipe:#xx.xxx.xx.xx[ 3660]
(4B82C166.0000-18:kdsrqc1.c,2620,"AccessRowsets") Could not obtain rowset 2, status code = 1000 (4B82C168.0000-31:kfastgcr.c,101,"KO4ST_GetCurrentStatusRecord") malloc failed (4B82C168.0001-31:ko4bkgnd.cpp,480,"BackgroundController::nodeStatusUpdate") TEMS heartbeat insert failed with status 1000

The above error message is due to lack of memory

The reason for the HUB TEMS crash is due to lack memory. To resolve the issue, execute the below command:

tacmd getdeploystatus to view the deployment entries and

tacmd cleardeploystatus -a to remove the accumulated deployment status messages maintained in the TEMS table

The current design of Remote Deploy 6.2.1 and greater, retains the completion status queue in memory. Unless that queue is cleared it will continue to increase in size indefinitely. By removing the queue entries that have completed their processing, the memory is released.

If the status queue is cleansed automatically without user intervention, then there would be no way for the user to know the status of their deployment requests since those requests may have already been removed from the queue.

To release the memory, it is advised to cleanse the queue periodically of only successful updates. Leave the failed deployment so that it be attempted again or retried.