Question & Answer
Question
Cause
The "/" partition is the root directory of the file system. The bulk of this partition falls under /usr and /etc directories.
By default, the QRadar disk sentry check runs every 60 seconds and looks for high disk usage across the "/" partition. If the "/" partition fills up above 95%, it will stop the QRadar critical services.
To verify the current partition usage, you can use the df -h command:
[root@hostname ~]# df -h FilesystemSize Used Avail Use% Mounted on /dev/mapper/rootrhel-root 20G 20G 258M 99% /
By far, the most common cause for the "/" partition filling up, is when a particular file system (e.g., remote nfs partition that is full or unavailable) is not available when a file (e.g., backup file) is needing to be written to, it may end up being written to "/" directory which could lead to partition space exceeding the threshold of 95% and stopping critical services.
Sometimes files are written to the mount points before other partitions are mounted, for example, if files are written to /store, /store/transient, /store/backups but the partitions are not yet mounted, they will be written to the "/" partition under the /nfs/backups, /store/transient, /store/backups directories.
To find out what files or directories are filling up a partition, see the Troubleshooting Disk Space Problems technote:
Technote 0881013 - QRadar: Troubleshooting Disk Space Problems
Answer
Quick Links
- 1. Troubleshooting / space issues
- 2. Defects around / partition
- 3. General Information about the sizing of / partition
1. Troubleshooting / space issues
The most common issue that affects the free space on the "/" partition is the files written to it instead of the original destination of an unavailable partition or directory.
Resolving disk space usage on "/" partition due to hidden files in unmounted directories.
You can check all the hidden files under the root "/" partition without the need to stop all services and unmount all partitions. Hidden files might not show up when using the du or ls commands and switches, even with -a. To investigate the contents of the underlying directory, the root (/) partition needs to be mounted again to a different mount point with the --bind option.
The following instructions allow us to inspect the contents of an unavailable partition, due to it being unmounted or full:
-
Create a directory that will be used as a temporary mount point:
mkdir /root/root_tmp -
Locate the block device for "/" using the df -h command:
[root@hostname ~]# df -h Filesystem
Size Used Avail Use% Mounted on /dev/mapper/rootrhel-root 20G 20G 258M 99% / Typically the block device for "/" is
/dev/mapper/rootrhel-rootfor QRadar 7.3.x or/dev/sda7for QRadar 7.2.8. -
Mount the root partition to the newly created temporary directory:
mount -o --bind <block_device for "/"> /root/root_tmp
Examples:- QRadar 7.3.x Versions:
mount -o --bind /dev/mapper/rootrhel-root /root/root_tmp - QRadar 7.2.8 Versions:
mount -o --bind /dev/sda7 /root/root_tmp
- QRadar 7.3.x Versions:
-
Navigate to the directories:
cd /root/root_tmp/ -
The following commands list the contents or check disk usage in those directories:
du -xch /root/root_tmp/ | sort -h -
If you identify certain files on the bind mount that you did not see on the original mount point, these files can be moved off to free up space. Alternatively, if the file is no longer needed, they can be deleted. Again, this situation most commonly occurs for a /store/backup NFS mount, when the NFS mount is not available for some reason.
-
Once completed with the bind mount, ensure that it is unmounted with the command below. Be sure that you are not within this partition when unmounting; otherwise, it may say that the partition is busy:
umount /root/root_tmp/
Important:
-
Any changes made to the bind mount also affect the files on the original mount (which is "/" in the above example).
-
Do not remove any files from "/" unless you are certain they are not needed. If you have questions about if a file is safe to remove or not, contact Technical Support for further guidance.
Notes:
-
If files are found inside the root partition that should be on the other partitions, these files should be moved to release disk space.
-
This can be applied to other partitions and mount points as well.
2. Defects around / partition
This is a summary list of defects encountered on /opt partition:
- No current known issues around the / partition.
3. General Information about the sizing of / partition
Partition requirements and recommendations when upgrading:
During the upgrade process, partition requirements and recommendations are generated and stored in the /root/partition_instructions.txt file. This file is deleted during QRadar® setup on the new operating system. If you choose not to use the partitions recommendations, make sure that you meet these partition requirements.
QRadar 7.3.1 Partition requirements and recommendations documentation
Note: Once on the IBM Knowledge Center, you can change the version from the drop-down for 7.2.8/7.3.0/7.3.2
Linux operating system partition properties for QRadar installations on your hardware:
If you use your appliance hardware, you can delete and re-create partitions on your Red Hat Enterprise Linux operating system rather than modify the default partitions.
Note: Once on the IBM Knowledge Center, you can change the version from the drop-down for 7.2.8/7.3.0/7.3.2
This technote details the changes in RHEL 7 in regards to partitioning changes
QRadar: What's new about the RHEL 7 Operating System
Was this topic helpful?
Document Information
Modified date:
07 January 2021
UID
ibm10881470