IBM Support

QRadar: Software update checklist for administrators

Question/Answer


Question

What steps can administrators review before they attempt to update their QRadar deployment?

Answer


Important notes before you update to QRadar 7.3.2

Patches versus upgrades

There are two different checklists in this article, each having a slightly different process:

  • The most common software update scenario for users is a QRadar patch update. A patch is delivered in the form of an SFS file and is used to update QRadar software to the same software stream (V.R.M). For example, updating software from QRadar 7.3.1 to QRadar 7.3.2. Click the Patching tab above for the checklist details on this process.
     
  • Upgrades are major release updates, delivered as an ISO file. An upgrade generally increments the R bit in the V.R.M.F format that is referenced earlier. Click the Upgrading tab above for the checklist details on this process. For example, users who are on QRadar 7.2.8 must use the upgrade ISO file to update to the QRadar 7.3.1 software stream. Upgrades are major updates and it is common that the installer will update the operating system (OS) as part of the installation.

    - For information on software releases and were to find SFS or ISO release notes, see the QRadar Master Software List.
    - For more information on versions, see Technote 7008656: V.R.M.F Maintenance Stream Delivery Vehicle terminology explanation.








     

 

Patching

 

Administrators who are planning an software update to QRadar 7.3.2 should read the Overview tab for special installation/issue notes before they begin.
 

Checklist for teams and administrators

  1. Image result for check box Notify users of scheduled maintenance.
  2. Image result for check box Verify that running scans and reports are complete.
  3. Image result for check box Request that users close all open QRadar sessions to prevent error logs messages.
  4. Image result for check box Advise users to close any active 'screen' sessions that are open to QRadar appliances.
  5. Image result for check box Download your update to your local workstation. A link is provided in every QRadar Release Note.
  6. Image result for check box If you are unsure of the IP addresses or hostnames for all appliances in the deployment, run the /opt/qradar/support/deployment_info.sh utility to get a CSV file that contains a list of IP addresses for each appliance in the deployment.

Hardware and data reviews

  1. Image result for check box Administrators should always consider updating to the latest QRadar version available.
  2. Image result for check box Verify the checksum of the QRadar software downloaded from IBM Fix Central. A checksum file is provided for all downloads.
  3. Image result for check box HA clusters must have primary appliances in the online 'Active' state and the secondary appliance should have a status of 'Standby' before you update. If any of the HA appliances are Unknown, administrators must contact support before you start your update.
  4. Image result for check box Verify that IMM configured and functional on all appliances. For more information, see: Integrated Management Module II: User's Guide.
  5. Image result for check box Verify the firmware on the appliance is at the latest version. For more information, see the QRadar Master Firmware List.
  6. Image result for check box Confirm all appliances are at the same software version: /opt/qradar/support/all_servers.sh -C -k /opt/qradar/bin/myver -v > myver_output.txt
  7. Image result for check box Verify that all previous updates are unmounted: /opt/qradar/support/all_servers.sh -C -k “umount /media/updates"
  8. Image result for check box Verify disk space for the deployment: /opt/qradar/support/all_servers.sh -C -k df -h /root /var/log | tee diskchecks.txt
  9. Image result for check box Verify that the following directories are mounted and available (for HA pairs, see Step 10):
    • /store - Stores event and flow data on each appliance.
    • /storetmp - Stores configuration information on each appliance in QRadar 7.3.0 and later.
    • /store/tmp - Stores configuration information on each appliance in QRadar 7.2.8 and earlier.
    • /transient - Stored saved searches and index information.
  10. Image result for check box In an HA pair, the following partitions should be reviewed before you start a software update (Example):
    • /store should only be mounted on the ACTIVE appliance, and NOT the STANDBY.
    • /transient should be mounted on both appliances (ACTIVE & STANDBY) as this directory is used for replication.

QRadar software review

  1. Image result for check box Review system notifications for errors and warnings for the following messages before you attempt to update. These error and warning system notifications should be resolved before you attempt to update.
    • Image result for check box Performance or event pipeline degradation notifications
    • Image result for check box Memory notifications
    • Image result for check box TX sentry messages or process stopped notifications
    • Image result for check box HA active or HA standby failure system notifications
    • Image result for check box Disk failure system notifications
    • Image result for check box Disk Sentry noticed one or more storage partitions are unavailable notifications
    • Image result for check box Time synchronization system notifications
    • Image result for check box Unable to execute a backup request notifications
    • Image result for check box Data replication experiencing difficulty notifications
    • Image result for check box RAID controller misconfiguration notifications
  2. Image result for check box Manually complete a deploy to verify it completes successfully in the user interface: Admin > Deploy Changes.
  3. Image result for check box Verify the latest configuration backup completed successfully and download the file to a safe location.
  4. Image result for check box Are any applications in the error state or not displaying properly? This can be blank tabs, error messages or errors in the user interface. If yes, these issues should be resolved before the update is started.

Post installation

  1. Image result for check box Clear your browser cache and alert all users to clear their browser cache. Optionally, you can use Private Browsing mode to ensure that pages are not cached when using QRadar.
  2. Image result for check box Complete an auto update from the QRadar admin interface (Auto Updates > Get Manual Updates).
  3. Image result for check box Unmount the /media/updates directory on all hosts, type: /opt/qradar/support/all_servers.sh -C -k “umount /media/updates"
  4. Image result for check box Delete the sfs file from all appliances.
  5. Image result for check box Review any iptables rules that are configured to see if the interface names that have changed in QRadar 7.3.1 due to the Red Hat Enterprise 7 operating system updates.

You've experienced an update issue, what should you do?

  1. Never reboot or stop an update in progress unless advised by QRadar Support. Updates in progress will display "Patch in progress -- DO NOT REBOOT" messages.
  2. If the update displays an error message, you can note the error message for your case with QRadar Support.
  3. If the command-line is available, SSH to the appliance and type: /opt/qradar/support/get_logs.sh -s
  4. The output of the get_logs.sh utility will be required to review your case. After the get_logs.sh utility completes, download the .tgz file to your local workstation. The get_logs.sh utility will indicate location and file name of the tgz file to provide to support.
  5. Open a case with QRadar Support and describe the error.
  6. If your appliances are unavailable or not functional, you can indicate that you have a 'System down' issue.
  7. A QRadar Support representative will contact you using your preferred method of communication.

 

Upgrading

 

Checklist for teams and administrators

  1. Image result for check box Notify users of scheduled maintenance.
  2. Image result for check box Verify that running scans and reports are complete.
  3. Image result for check box Request that users close all open QRadar sessions to prevent error logs messages.
  4. Image result for check box Advise users to close any active 'screen' sessions that are open to QRadar appliances.
  5. Image result for check box Download your update to your local workstation. A link is provided in every QRadar Release Note.
  6. Image result for check box If you are unsure of the IP addresses or hostnames for all appliances in the deployment, run the /opt/qradar/support/deployment_info.sh utility to get a CSV file that contains a list of IP addresses for each appliance in the deployment.
  7. Image result for check box Verify that teams have moved their personal files to a safe location, which can include:
    • Scripts
    • Personal utilities
    • Important files or exports
    • Jar files or hotfixes that were provided by QRadar support
    • Verify no important files exist in temporary directories (/tmp or /storetmp)

Hardware and data reviews

  1. Image result for check box Administrators should always consider updating to the latest QRadar version available.
  2. Image result for check box Verify the checksum of the QRadar software downloaded from IBM Fix Central. A checksum file is provided for all downloads.
  3. Image result for check box HA clusters must have primary appliances in the online 'Active' state and the secondary appliance should have a status of 'Standby' before you update. If any of the HA appliances are 'Unknown' administrators must contact support before you start your update.
  4. Image result for check box Verify that IMM is configured and functional on all appliances. For more information, see: Integrated Management Module II: User's Guide.
  5. Image result for check box Verify the firmware on the appliance is at the latest version. For more information, see the QRadar Master Firmware List.
  6. Image result for check box Administrators can also take an additional backup step by completing a CMT export of their dashboards, rules, etc. For more information, see: Exporting QRadar Content with CMT.
  7. Image result for check box Confirm all appliances are at the same software version: /opt/qradar/support/all_servers.sh -C -k /opt/qradar/bin/myver -v > myver_output.txt
  8. Image result for check box If you have off-board storage configured (/store mounted off the appliance), see the QRadar Upgrade Guide.
  9. Image result for check box Verify that all previous updates or patches are unmounted: /opt/qradar/support/all_servers.sh -C -k “umount -v /media/*"
  10. Image result for check box Verify disk space for the deployment: /opt/qradar/support/all_servers.sh -C -k df -h /root /var/log | tee diskchecks.txt
  11. Image result for check box Administrators should always complete a pretest after mounting an ISO file using /media/cdrom/setup -t on important appliances in their deployment. This process typically takes between 2-5 minutes and does not restart services.
  12. Image result for check box Verify that the following directories are mounted and available (for HA pairs, see Step 13):
    • /store - Stores event and flow data on each appliance.
    • /storetmp - Stores configuration information on each appliance in QRadar 7.3.0 and later.
    • /store/tmp - Stores configuration information on each appliance in QRadar 7.2.8 and earlier.
    • /transient - Stored saved searches and index information.
  13. Image result for check box In an HA pair, the following partitions should be reviewed before you start a software update (Example):
    • /store should only be mounted on the ACTIVE appliance, and NOT the STANDBY.
    • /transient should be mounted on both appliances (ACTIVE & STANDBY) as this directory is used for replication.

QRadar software review

  1. Image result for check box Review system notifications for errors and warnings for the following messages before you attempt to update. These error and warning system notifications should be resolved before you attempt to update.
    • Image result for check box Performance or event pipeline degradation notifications
    • Image result for check box Memory notifications
    • Image result for check box TX sentry messages or process stopped notifications
    • Image result for check box HA active or HA standby failure system notifications
    • Image result for check box Disk failure system notifications
    • Image result for check box Disk Sentry noticed one or more storage partitions are unavailable notifications
    • Image result for check box Time synchronization system notifications
    • Image result for check box Unable to execute a backup request notifications
    • Image result for check box Data replication experiencing difficulty notifications
    • Image result for check box RAID controller misconfiguration notifications
  2. Image result for check box Manually complete a deploy to verify it completes successfully in the user interface: Admin > Deploy Changes.
  3. Image result for check box Verify the latest configuration backup completed successfully and download the file to a safe location.
  4. Image result for check box Verify the installed version of WinCollect. QRadar Support recommends you update to WinCollect v7.2.5 or later.
  5. Image result for check box Are any applications in the error state or not displaying properly? This can be blank tabs, error messages or errors in the user interface. If yes, these issues should be resolved before the update is started.

Post installation

  1. Image result for check box Clear your browser cache and alert all users to clear their browser cache. Optionally, you can use Private Browsing mode to ensure that pages are not cached when using QRadar.
  2. Image result for check box Complete an auto update from the QRadar admin interface (Auto Updates > Get Manual Updates).
  3. Image result for check box Unmount the /media/cdrom directory on all hosts, type: /opt/qradar/support/all_servers.sh -C -k “umount /media/cdrom"
  4. Image result for check box Delete the ISO from all appliances.
  5. Image result for check box Review any static routes or customized routing. As an upgrade can remove static routes that will need to be recreated by the administrator after the upgrade completes.
  6. Image result for check box Review any iptables rules that are configured to see if the interface names that have changed in QRadar 7.3.1 due to the Red Hat Enterprise 7 operating system updates.
  7. Image result for check box Verify that all apps are working as expected. If your app connects to outside servers, you have a QRadar proxy configured, and the app not functioning as expected after you update to QRadar 7.3.2, see: QRadar 7.3.2: How to tune proxy configurations for app containers.

You've experienced an update issue, what should you do?

  1. Never reboot or stop an update in progress unless advised by QRadar Support. Updates in progress will display "Patch in progress -- DO NOT REBOOT" messages.
  2. If the update displays an error message, you can note the error message for your case with QRadar Support.
  3. If the command-line is available, SSH to the appliance and type: /opt/qradar/support/get_logs.sh -s
  4. The output of the get_logs.sh utility will be required to review your case. After the get_logs.sh utility completes, download the .tgz file to your local workstation. The get_logs.sh utility will indicate location and file name of the tgz file to provide to support.
  5. Open a case with QRadar Support and describe the error.
  6. If your appliances are unavailable or not functional, you can indicate that you have a 'System down' issue.
  7. A QRadar Support representative will contact you using your preferred method of communication.

Where do you find more information?


 

Document information

More support for: IBM QRadar SIEM

Software version: All Versions

Operating system(s): Linux

Reference #: 0738599

Modified date: 01 April 2019