IBM Support

QRadar: Upgrades from v7.2.8 to v7.3.1 can result in the /opt partition being less than 13 GB (Updated)

Troubleshooting


Problem

After an administrator upgrades from QRadar version 7.2.8 to 7.3.1, partitions are resized and /opt (/dev/mapper/rootrhel-opt) may not be converted from 7 GB to 13 GB. This can lead to services stopping when the /opt partition is 95% full or greater. A new support utility partitionDiagnostic has been released to assist with space issues in the /opt partition. This script is designed to clean up unused service versions and free up partitions clearing away any unused data.

  • Clean up legacy files that consume space for older versions of the ecs-ec-ingress service.
  • Move files and create a symlink for /opt/qradar/dca  to /store/dca to prevent X-Force updates from consuming space in the /opt directory.
     
Option flags
  -d, --delete        Delete the files and folders
  -p, --dir string    scan partition for large unused files :: future feature not available yet (default "/opt/")
  -n, --dry-run       Don't actually remove anything, just show what would be done.
  -h, --help          help for partitionDiagnostic
  -s, --save-delete   Backup all the Files and Folders, before the deletion, will fail if the backups do NOT complete

Cause

When you upgrade from QRadar 7.2.8 to a newer version (7.3.0 or 7.3.1), the upgrade of the underlying operating system resizes partitions as part of the upgrade process. Administrators who upgrade from QRadar 7.2.8 might notice that their /opt partition is smaller than expected when partitions are recreated for the Console. This smaller partition can lead to system notifications about disk space issues as X-Force Threat Intelligence data is kept in the opt partition and the resized /opt partition was not resized from 7 GB to 13 GB. In 7.3.1 we utilize /opt more often, and unless you did a fresh install it wouldn’t use the new LVM partition size of 13 GB. Rebuilding the system with the latest patch of 7.3.1 is the recommended long term solution.


Diagnosing The Problem

How to complete a dry run before removing files

  1. Using SSH, log in to the QRadar Console as the root user.
  2. Navigate to /opt/qradar/support.
  3. To to run the partitionDiagnostic utility, type: ./partitionDiagnostic -n
  4. The utility runs and evaluates the files to be cleaned up and potential changes are written to the screen.

    Example output
    [root@SupportLAB support]# ./partitionDiagnostic -n
    2019/01/23 10:07:31 --------   ecs-ec   --------
    2019/01/23 10:07:33 Not loading "rhnplugin" plugin, as it is disabled
    Loading "product-id" plugin
    Loading "search-disabled-repos" plugin
    Loading "subscription-manager" plugin
    Updating Subscription Management repositories.
    Unable to read consumer identity
    This system is not registered with an entitlement server. You can use subscription-manager to register.
    Config time: 0.123
    Yum version: 3.4.3
    rpmdb time: 0.000
    Resolving Dependencies
    --> Running transaction check
    ---> Package ecs-ec-0.0.804.noarch 0:0.0.804-1 will be erased
    Checking deps for ecs-ec-0.0.804.noarch 0:0.0.804-1 - e
    --> Finished Dependency Resolution
    Dependency Process ending
    Depsolve time: 1.311
    Dependencies Resolved
    ================================================================================
     Package                Arch           Version             Repository      Size
    ================================================================================
    Removing:
     ecs-ec-0.0.804         noarch         0.0.804-1           @local         149 M
    Transaction Summary
    ================================================================================
    Remove  1 Package
    Installed size: 149 M
    Exiting on user command
    2019/01/23 10:07:39 Moving: /opt/qradar/dca/ To: /store/dca/ & Creating a symlink
    2019/01/23 10:07:39 Services that will be shutdown: scaserver

Resolving The Problem

Run the partitionDiagnostic utility to remove unwanted files

  1. Using SSH, log in to the QRadar Console as the root user.
  2. Navigate to /opt/qradar/support.
  3. To to run the partitionDiagnostic utility, type: ./partitionDiagnostic -d

    Results
    The partitionDiagnostic utility runs and provides an output of the files removed.



Troubleshooting 'Service did not shutdown properly' errors 
Users who have issues running partitionDiagnostic.sh can stop the scasever if you experience an issue where the utility prints the 'Service did not shutdown properly' error message. An example error message is shown in the output below. The rows marked ** show the error messages when a failure to stop the scaserver occurs: 

[root@qradarlab_environ /]# /opt/qradar/support/partitionDiagnostic -d
2019/03/13 15:38:59 -------- ecs-ec --------
2019/03/13 15:39:00 -------- ecs-ec-ingress --------
2019/03/13 15:39:00 tar -czf dca-symlinkBackups.tar.gz /opt/qradar/dca/ to location: /store/support
2019/03/13 15:40:38 Shutting down the services: scaserver
2019/03/13 15:40:38 Stopping: scaserver
**2019/03/13 15:41:32 [ERROR] Failed with exit status 1**
**2019/03/13 15:41:32 [ERROR] Service did not shutdown properly: exit status 1**
2019/03/13 15:41:32 Attempting to start services
2019/03/13 15:41:32 Starting: scaserver
2019/03/13 15:42:09
2019/03/13 15:42:09 Starting: scaserver
2019/03/13 15:42:09
[root@qradarlab_environ /]#



Workaround
1. Using SSH, log in to the QRadar Console as the root user.
2. To stop the sca server, type: systemctl stop scaserver
3. Run ./opt/qradar/support/partitionDiagnostic.sh
4. Wait for the utility to complete.
5. To start the sca server, type: systemctl start scaserver

Results
If you continue to experience issues with the partitionDiagnostic.sh utility, ensure you have install the latest QRadar weekly auto update. QRadar support tools updates are provided via our weekly update utility. If your QRadar Console is air-gapped or you do not have Internet access to your Console, see: How to manually install the QRadar weekly auto update. If you continue to experience issues with partitionDiagnostic.sh or you are still having space issues after partitionDiagnostic completes, contact QRadar Support for assistance.












Where do you find more information?

Document information

More support for: IBM QRadar SIEM

Software version: 7.3.0, 7.3.1, 7.3.2

Operating system(s): Linux

Reference #: 0716207

Modified date: 27 March 2019