Troubleshooting
Problem
Customers that patched from QRadar version 7.2.8 to the latest see the original opt (dev/mapper/rootrhel-opt) size of 7 GB instead of the newer rezised13 GB. This may lead to services stopping when the opt partition is 95% full or greater.
Cause
When you patch from QRadar 7.2.8 to a newer version, the upgrade of the underlying operating system resizes partitions as part of the upgrade process. Administrators who upgrade from QRadar 7.2.8 might notice that their opt partition is smaller than expected when partitions are re-created for the Console. This smaller partition can lead to system notifications about disk space issues as X-Force Threat Intelligence data is kept in the opt partition and the opt partition was not resized from 7 - 13 GB. In 7.3.1 and later, we use opt more often, and unless you did a fresh install it wouldn’t use the new LVM (logical volume manager) partition size of 13 GB. Rebuilding the system with the latest ISO is the recommended long-term solution.
Diagnosing The Problem
- Run "df -h" and check utilization of /opt
- Run "du -sh * /opt|grep G" check for directories GigaBytes in size
- Run "ll /opt/ibm/si/services/ecs-e*" check for any old associated older version service directories for the ecs-ec, ecs-ec-ingress and ecs-ep services.
Resolving The Problem
Backup or remove unwanted files found taking up /opt space. For example, cleaning up /opt/ibm/si/services/ecs-e* directory structure leaving only the latest rpm directories and their contents. Please keep the existing associated symbolic links as well as any other relevant directories such as "eventnosis" as follows.
[root@xxxxxx ~]# ll /opt/ibm/si/services/ecs-e*
/opt/ibm/si/services/ecs-ec:
Total 0
drwxr-xr-x 5 root root 59 Jun 2 2020 2020.3.0.20200526173939
lrwxrwxrwx 1 root root 51 Jun 2 2020 current -> /opt/ibm/si/services/ecs-ec/2020.3.0.20200526173939
/opt/ibm/si/services/ecs-ec-ingress:
Total 0
drwxr-xr-x 5 root root 59 Jun 2 2020 2020.3.0.20200526173939
lrwxrwxrwx 1 root root 59 Jun 2 2020 current -> /opt/ibm/si/services/ecs-ec-ingress/2020.3.0.20200526173939
drwxr-xr-x 3 root root 17 Jul 12 2018 eventgnosis
/opt/ibm/si/services/ecs-ep:
Total 0
drwxr-xr-x 5 root root 59 Jun 2 2020 2020.3.0.20200526173939
lrwxrwxrwx 1 root root 51 Jun 2 2020 current -> /opt/ibm/si/services/ecs-ep/2020.3.0.20200526173939
You have mail in /var/spool/mail/root
[root@xxxxxxx ~]#
- If applicable, move files and create a symlink for /opt/qradar/dca to /store/dca to prevent X-Force updates from consuming space in the /opt directory.
- Run "df -h" and check to see that utilization of /opt dropped
- Note: Also see our knowledge base article "Disk Space 101" https://www.ibm.com/community/qradar/home/qradar-disk-space/
Results
Once the older files, no longer needed are removed opt utilization drops below the warning threshold. See reference article below.
"Disk usage system notifications" https://www.ibm.com/docs/en/qsip/7.3.3?topic=notifications-disk-usage-system
Document Location
Worldwide
Was this topic helpful?
Document Information
Modified date:
28 September 2022
UID
ibm10716207