IBM Support

QRadar: How to update appliances in parallel

Question & Answer


Question

Updating in parallel allows administrators to save on downtime by first patching the Console, then applying the update to all other appliances simultaneously. This article walks through process of how to update appliances in parallel.

Answer

Update methods supported by QRadar - Overview
QRadar has two methods of completing updates by using SFS files:
  • Patch All - Patch all is an easy method of upgrading QRadar as administrators can mount and run the installer on only the console and select the Patch All option. This option upgrades the Console appliance first, then copies the necessary files to each managed host and starts the installation in a serial method.
    • Pro: Easy to deploy, run from the Console, and walk away. A summary of the update is provided to the administrator on the Console.
    • Pro: Appliances are only down one at a time, so you only that one appliance that is patching is not collecting data.
    • Con: This upgrade method can take a long time for larger deployments.
    • Con: The all option uses deployment information to update appliances and there is no method of applying an order to how appliances are updated. The system walks through the deployment and there is no way to tell which appliance is going to be updated next.
  • Updating in Parallel - Parallel updates take more manual intervention from an administrator, but can drastically reduce the amount of time required to update a large deployment to the latest version. In this scenario, parallel updates allow administrators to control the order that the appliances are patched and can start a number of updates simultaneously to prevent extended downtime. For example, the administrator might choose to simultaneously update the event processors EP1, EP2, and EP3, and set the load balancer to forward all data to EP4.
    • Pro: The fastest method of upgrading a deployment.
    • Pro: Fairly easy to distribute the SFS and mount files by using the all_servers.sh support tool.
    • Pro: Administrators can decide the order in which appliances are updated or start updates in parallel for groups or regions to reduce downtime.
    • Con: When you update in parallel, all appliances you update simultaneously cannot collect data.

Here is a visual example of the difference in process:
Installation type Step 1 Step 2 Step 3 Step 4 Step 5
Patch All Mount file and install on the Console (1h - 1.25 hours) File copied automatically to EP1 and EP1 installs (45m) File copied automatically to EP2 and EP2 installs (45m) File copied automatically to EP3 and EP3 installs (45m) File copied automatically to EP4 and EP4 installs (45m)
Updating in Parallel Mount file and install on the Console (1h - 1.25 hours) Copy files to all hosts and start updates simultaneously for all managed hosts (45m)  
Time savings: Over 2 hours


IMPORTANT: The Console expects all appliances to be on the same version, meaning that they can stay at a different version temporarily, but administrators are expected to ensure that the entire deployment is updated. When you upgrade a QRadar deployment, all appliances are intended to be at the same software version. If you update the Console and leave managed hosts in the network at the old version they still collect data, however, the deployment might not work as expected. For example, offenses sent from the managed host to the Console might not trigger rules properly, searches might not complete as expected, and several other issues can occur. 
Restrictions
This procedure does not apply to HA secondary appliances, as these systems are updated by the primary host. This procedure also does not apply to QRadar Packet Capture appliances, as they use a different SFS file for their appliance updates.

Upgrade a QRadar Deployment in Parallel - Procedure

Before you begin
Ensure that you take the following precautions:

  • Back up your data before you begin any software upgrade.
  • To avoid access errors in your log file, close all open QRadar sessions.
  • Verify there are no Deploy Changes pending on the Console.
  • The file referred to in this procedure is an .SFS patch file, which is only capable of performing upgrades or patches to existing QRadar installations. Administrators who want to complete a new appliance installation of QRadar need to review the QRadar Installation Guide.
  • HA secondaries are upgraded by the primary host. There is not need to copy or run the patch on an HA secondary. Before Administrators upgrade the environment, confirm that the primary HA hosts are active, and secondaries are in Standby mode and not offline.
Part 1: Update the Console
The instructions guide administrators through the process of upgrading an existing QRadar version at 7.3.3 or later to the newest software version.
  1. Download the update from IBM Fix Central.
  2. SSH in to your QRadar Console as the root user.
  3. Copy the update to a directory on the QRadar Console that has sufficient space for the SFS file.
    QRadar 7.3.3 and later: The best directories for the SFS file are as follows:
    • /store/tmp (best)
      The storetmp directory is available on all appliance types. This directory can be cleaned by tmpwatch the tmpwatch cron job if you postpone your update.
    • /tmp (better)
      This directory is available on all appliances, but moving a file to /tmp can cause services to stop if it fills the directory. If you leave a file in /tmp for 10 days without completing the SFS update, it might get cleaned up by Red Hat's tmpwatch cron job.
    • store/transient
      The store/transient directory is allocated at 10% of the overall /store directory. However, this directory does not exist on all appliances, such as QFlow or QRadar Network Insights and might not be an actual partition on all appliances.
  4. To create the /media/updates directory, type the following command: 
    mkdir -p /media/updates
  5. Open the directory where you copied the patch file.
  6. To mount the patch file to the /media/updates directory, type the following commands: 
    mount -o loop -t squashfs /store/tmp/<QRadar file name>.sfs /media/update
  7. To run the patch installer, type the following command: 
    /media/updates/installer
    NOTE: The first time that you run the update, there might be a delay before the update installation menu is displayed.
  8. Select the option to patch the Console.
    NOTE: If the SSH session is disconnected, reconnect to the host you are upgrading and rerun the installer. The system takes you back to the current step in the patch process.
  9. After the patch completes and you exited the installer, type the following command: 
    umount /media/updates
  10. Before users log in to the Console, they need to clear their browser cache.

    Results
    The Console appliance is updated. Administrators can now proceed to Part 2 to learn how to distribute the SFS and mount files by using the all_servers support tool.
Part 2: Update your managed hosts
This section informs administrators how to parallel patch a QRadar deployment. It is critical that customers verify they have space to install the update. You can use the Console to stage the SFS file to all managed hosts and verify that you have disk space available on all hosts. Generally, a system needs to have enough space equivalent to twice the size of the update in the root directory. If the system does not have enough disk space to install the update, the appliance is bypassed and a summary detail, which managed hosts were installed successfully and which were unsuccessful.
  1. SSH in to the QRadar Console as the root user.
  2. Verify that none of your appliances are out of disk space before you move a large file. This command exports a list of partitions and the free space to diskchecks.txt 
    /opt/qradar/support/all_servers.sh -k "df -h /tmp /store/tmp /store/transient" | tee diskchecks.txt
  3. Enter the following command to copy the patch file (SFS) to all managed hosts. The -r flag is required for QRadar 7.3.3 and later systems. As by default all_servers.sh puts the SFS file in the /tmp directory. 
    /opt/qradar/support/all_servers.sh -p /store/tmp/<QRadar file name>.sfs -r /store/tmp
  4. Enter the following command to ensure the target directory exists on all managed hosts: 
    /opt/qradar/support/all_servers.sh "mkdir -p /media/updates && umount /media/updates"
  5. Mount all of the files on managed hosts by using all_servers. 
    /opt/qradar/support/all_servers.sh "mount -o loop -t squashfs /tmp/<QRadar file name>.sfs /media/updates"
  6. SSH to each managed host individually and enter: 
    /media/updates/installer
    Important: Do not launch the installer by using the all_servers command. Attempting to run the installer, by using all_servers.sh, prevents errors from being displayed to the user and can prevent pretests from running properly.
  7. The installer command starts a screen session and starts the installer menu. Select Yes to update the managed host.
  8. Run the installer command on each managed host in the deployment.
    NOTE: If the SSH session is disconnected, reconnect to the host you are upgrading and rerun the installer. The system takes you back to the current step in the update process.
  9. Wait for each appliance to finish as they all update in parallel. You can use the following command to monitor the installation on all the managed hosts: 
    /opt/qradar/support/all_servers.sh -R

    Results
    A summary message is displayed as each managed host completes the update or any errors are displayed. If you experience an upgrade error, contact QRadar Support.

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwsyAAA","label":"Admin Tasks"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.3.3;7.4.3;7.5.0"}]

Document Information

Modified date:
08 June 2023

UID

swg21998517

Page Feedback