IBM Support

QRadar: Replacing a Console appliance in a deployment using the same IP address or hostname (Updated)

Troubleshooting


Problem

This tech note describes the process that can be used to migrate data from an older QRadar Console to a new Console appliance that uses the existing IP address or hostname. All managed host appliances stay as-is. This instruction is intended for non-HA appliances.

Environment

QRadar deployments where administrators are replacing a Console with new hardware while keeping managed hosts as-is. The new Console uses the same IP address as the Console being replaced.

Resolving The Problem

The procedure outline is intended to get the Console appliance replaced in the network quickly with minimum downtime by using the same IP address that was used by the old hardware. This procedure allows managed hosts in the deployment to continue to receive events while the Console is offline. There is no need for the administrator to remove managed host from the old Console as this procedure allows the new Console to takeover any existing hosts in the deployment.

 

Overview and check list

Step 1: Preparing your new Console hardware
Step 2: Preparing your old Console appliance
Step 3: Reassigning IP addresses on the old Console
Step 4: Setting IP addresses on the new Console appliance
Step 5: Moving certificates and any custom generated private and public key pairs
Step 6: Restoring the configuration backup to the new Console appliance
Step 7: How to transfer event and flow data to the new hardware
Step 8: Migration complete

 

Before you begin

  • Write down the network information for the old Console appliance as the administrator needs to manually type this information into the network configuration for the new appliance. 
    Note: Though we recommend hostnames to be in lower case, the new appliance hostname must match the value of the old console appliance you are replacing, including capitalization. If the hostname differs when you install new appliances, you might experience issues with Deploy Changes after you perform the configuration restore.

  • The administrator must have a recent configuration backup from the old Console appliance. The configuration backup is used to restore settings, users, rules, log sources, and more to the new Console.
  • The software version of the new Console appliance must match the software version of the old Console appliance. QRadar does not allow appliances at different software versions in the deployment. Administrators might be required to reinstall an ISO for the appliance to downgrade or use a Fix Pack (SFS) to upgrade on the new appliance. The paperwork that came with your appliance lists the installed software version.
  • Refer to a list of what is backed up as part of the configuration and data backups here: Backup and recovery.
     
Note: The following items are not backed-up as part of the configuration or data backup and needs to be manually backed up: 
/etc/httpd/conf/certs
/opt/qradar/conf/trusted_certificates/
/opt/qradar/conf/syslog-tls.keystore
/etc/ssh/*
/root/.ssh/*

Custom scripts, hot fixes, other customization
Note: New console installations have a 35-day temporary license. After the temporary license expires, you can lose access to the QRadar user interface (UI) and be unable to initiate the configuration restore. If your temporary license expires before you migrate, you can upload your purchased console license key (if you have a copy) to regain access to the UI, or you might need to reinstall the new console. Though there is no way to export one or more license keys from your current production system, the license key is migrated once a full configuration restore is applied (Step 6).

Important: App data is separated from the configuration backup and restores. To backup and restore app data, see Backing up and restoring app data.


 

Step 1: Preparing your new hardware

During this step, the administrator needs to first verify the software version from the deployment. If required, complete a QRadar installation on the new hardware. The installation of the new appliance uses a temporary IP address until the old hardware is removed from the deployment later on in this document.
 

Notice: Administrators migrating appliances on 7.4.3 versions should review the following issue before you begin a hardware migration. IJ37604: FAILURE TO DECRYPT A CONFIG RESTORE IN 7.4.3 FIX PACK 4 CAN CAUSE USER INTERFACE ISSUES.

 
Procedure
  1. Rack the appliance.
  2. Connect network connections.
  3. To determine the QRadar version installed from the factory. To verify the version:
    1. Power ON the appliance and login as root.
    2. When the system displays the license agreement (EULA), press and hold the Ctrl + c keys. This returns you to a command prompt.
    3. To view the installed software version, type: /opt/qradar/bin/myver
    4. If the new hardware's software version is older than the software running in production, logout, then log in again as root and complete the installation. After the installation completes, download the proper Fix Pack to bring the Console to the same version as the deployment.
    5. If the new hardware's software version is newer than the software running in production, you can either choose to upgrade your production system to match the new appliance or downgrade the software by installing an older release of QRadar from IBM’s Fix Central site. If you decide to reinstall the new system with an older release, complete that first, then begin this procedure again.

      Note: For a list of software versions, see: QRadar Forums, Software Version List .
  4. Select the desired installation type.
  5. Select the desired appliance type and continue through the configuration wizard.
  6. Type a temporary IP address and network information from the old Console when you configure the new hardware.
  7. Type a root password for the appliance.
  8. Follow the installation wizard to complete the installation.
  9. If required from Step 3e, upgrade the new hardware to the same patch level as the old Console appliance.

    Results
    You are now ready to prepare files on your old Console and ensure you have a configuration backup.
     

Step 2: Preparing your old Console appliance

During this step, the administrator needs to verify they have a recent configuration backup file from the old Console. If a recent configuration backup does not exist, then administrators can use the procedure to create an On-Demand Backup. By default, configuration backups are stored in /store/backup on the QRadar Console. It is recommended to copy a recent backup to a safe location, such as the administrator's workstation or another QRadar system in the deployment.

Important: Configuration backups can be restored only to the same version of QRadar that they were created with. If the administrator plans to change the overall QRadar version in the deployment, make sure you create a new configuration backup after any software change. Keep these files in a safe place for your hardware migration. Moving from a smaller Console to a larger or newer appliance is supported by the migration or backup process. For example, a 3105 Console's configuration backup can be applied to a 3128 or a 3148 appliance.
 

Notice: Administrators migrating appliances on 7.4.3 versions should review the following issue before you begin a hardware migration. IJ37604: FAILURE TO DECRYPT A CONFIG RESTORE IN 7.4.3 FIX PACK 4 CAN CAUSE USER INTERFACE ISSUES.

 
Procedure
  1. Log in to the old Console appliance.
  2. Click the Admin tab.
  3. Click Backup and Recovery.
  4. From the navigation menu, click On-Demand Backup.
  5. Type a name and a description for the new configuration backup.
  6. Click Run Backup.
  7. Wait for the configuration backup to complete.
  8. Click the name of the On-Demand Backup to download the file.
  9. Copy the configuration backup off of the old QRadar Console to a safe location.

    Results
    A configuration backup file is created for the new Console to use. This file is required later on in the procedure to restore users, rules, log sources, offenses, reports, admin configurations, and other system settings to the new hardware.
     

Step 3: Reassigning IP addresses on the old Console appliance

The administrator is now ready to change the IP address of the old Console to an unused or decommissioned IP range. This is done -manually- by adjusting the network configuration file directly, not by using the common “qchange_netsetup” command. Using this method, allows for the changing of the physical IP address of the system to avoid conflicts. It also allows for quick reversion to the old address, in the event, the backup restore does not complete successfully on the new system. After the IP address is changed on the existing console, it cannot reach or effect any changes to the other hosts in the deployment until or unless the IP address is reverted, if required.


Note: This task needs to be completed by using IMM or from a physical keyboard to prevent connection and lockout issues. For users who are accustomed to making changes of network configuration files in Linux“, it could be performed by using SSH and by using ‘screen’.

 
How to assign the IP address of the old console appliance to a decommissioned range
This procedure reassigns the IP address of the old appliance to a new address space or a decommissioned address space to free up the existing IP address for the new hardware.

Procedure
  1. Using IMM for remote access or the local Console keyboard, log in to the command line of the old appliance as the root user.
  2. To reassign the IP address of the old appliance:
  3. Verify which network interface is the management interface by typing: cat /etc/management_interface
    For example,

    The interface listed in this file is the QRadar management interface. The image is an example and administrators always need to verify the management interface.
  4. Change directory to /etc/sysconfig/network-scripts/
  5. Open the file ifcfg-{name}, as listed in the /etc/management_interface file.
    For example,
  6. Change the IP address to an unused or decommissioned range by editing the IPADDR= line.
    For example,
  7. Save the changes to the file.
  8. Shut down the hostcontext service by using the command:
    systemctl stop hostcontext
  9. Depending on whether or not you need to migrate data from the existing console, you need to power off or reboot the old console in order for the new console to deploy correctly.

    Important: If you still have data to migrate, the safest option for administrators is to reboot the appliance, especially if you do not have access to a crash cart for the appliance or IMM access. 

    Results
    After the old system is rebooted, the IP address switch is complete and the IP address change is complete, freeing up the old IP address to use on the new console. QRadar processes on the system can display errors, which are OK. If you switch the IP address back later, QRadar operates normally.

    The new appliance can now be set up and use the old Console's IP address and hostname. Administrators should not unrack the old hardware yet, as data still exists on the old appliance that needs to be transferred to the new appliance.

    Note: The hostnames that do not confirm to standards are no longer supported in QRadar 7.3.X. A valid hostname is a string up to 24 characters that include only [a-z][A-Z][0-9], minus sign (-), and period (.).
 

Step 4: Setting IP addresses on the new Console appliance

The new console hardware now assumes the IP address of the old console. To make this change, use the qchange_netsetup command.
Note: QRadar® development identified a defect in the product where running the qchange_netsetup utility can cause critical configuration issues on appliances. Administrators with QRadar® V7.4.1 or earlier are instructed to confirm information in qradar_netsetup.log before you complete any network changes that use the /opt/qradar/bin/qchange_netsetup utility. Refer to this page for more information: 
Important: A critical issue has been identified in /opt/qradar/bin/qchange_netsetup (IJ31239)
Procedure
  1. Using IMM for remote access or the local Console keyboard, log in to the command line of the new appliance as the root user.
  2. To change the IP address, type /opt/qradar/bin/qchange_netsetup
  3. Use the Configuration Wizard to change the IP address of the system to the old Console's IP address.
  4. Save and exit the Wizard to complete the address change.
Reminder: Though we recommend hostnames to be in lower case, the new appliance hostname must match the value of the old console appliance you are replacing, including capitalization. If the hostname differs when you install new appliances, you might experience issues with Deploy Changes after you perform the configuration restore.

Results
The new Console is installed and uses the old Console's IP address. The next step is to restore the configuration backup from the old Console.


 

Step 5: Moving certificates and custom generated private and public key pairs (as required)

Appliances that manage scanners and log sources that authenticate need the certificates copied  from the old appliance to the new appliance to ensure that log sources and scanners can connect to remote sources. If the administrator created custom generated private keys, these also need to be migrated by transferring /etc/ssh and /root/.ssh directories.
 
Procedure
  1. Log in to the QRadar old Console as the root user.
  2. To copy certificates from the old hardware to the new appliance (targetserver), type the following command:

    For certificates:
    Example 1: rsync -avz /opt/qradar/conf/trusted_certificates/ root@targetserver:/opt/qradar/conf
    Example 2: scp -pr /opt/qradar/conf/trusted_certificates/ root@targetserver:/opt/qradar/conf
  3. Configure the correct SSL certificate for tomcat: 

    Scenario 1: Use the old Console's certificates:
    1. Copy the old Console's certificate files to the new Console's /etc/httpd/conf/certs/ directory.

    scp /etc/httpd/conf/certs/cert.* root@targetserver:/etc/httpd/conf/certs/

    2. Install the old certificates.
    Important: Administrator must know the absolute path of the public and private keys.

    /opt/qradar/bin/install-ssl-cert.sh -i

    Follow the on-screen instructions.

    For more information about installing SSL certificates, see Installing a new SSL certificate.

    Scenario 2: Use the newly created self-signed certificates:
    1. Copy the public key from the new console to all the managed hosts.

    /opt/qradar/support/all_servers.sh -p /etc/httpd/conf/certs/cert.cert

    2. Move the public key to the right directory.

    /opt/qradar/support/all_servers.sh "mv -v /storetmp/cert.cert /etc/httpd/conf/certs/"

    3. Restart hostcontext in the managed hosts to apply the new certificates.

    /opt/qradar/support/all_servers.sh "systemctl restart hostcontext"
  4. Validate if the managed hosts are able to connect to tomcat.

    /opt/qradar/support/all_servers.sh "/opt/qradar/bin/test_tomcat_connection.sh"
    If administrator notices the following message "Unable to connect to tomcat", contact QRadar Support for assistance.
Results
The required certificate and ssh key files are transferred to the new Console. The administrator is now ready for the final step, which is to migrate event and flow data off the old appliance. Administrators can complete a dry run of the data transfer by using the -d parameter in the syncAriel utility.
 
 

Step 6: Restoring the configuration backup to the new Console appliance

The configuration backup from the old Console can now be applied to the new hardware. In most cases, it is recommended that the administrator uses secure copy (SCP) to move the file to the QRadar Console. The backup import facility in the User Interface (UI) has a size limit of 512 MBs. If your environment has a large configuration, the configuration backup size might grow beyond what the UI supports. If the file you are attempting to import is larger than 512 MB, this error is displayed:
http://www-01.ibm.com/support/docview.wss?uid=swg21991697&aid=1

Note: If using a backup where managed hosts were removed and/or added to the configuration since the backup was taken, a deployment problem might occur after migration. For more information, see QRadar: Deploy times out due to missing or mismatched tokens

To restore your backup

  1. Using SCP, copy the configuration backup file to /store/backupHost/inbound/ of the new Console.
  2. Log in to the QRadar Console as an administrator.
  3. Click the Admin tab and select the Backup and Recovery icon.
  4. Select the configuration backup you copied to the Console and click Restore.
  5. From the restore options list, select the Select All Configuration Items check box.
  6. From the restore options list, select the Select All Data Items check box.
  7. Click Restore to start the configuration restore process.
  8. From the Admin tab, Advanced, click the Deploy Full Configuration icon.
    Note: QRadar continues to collect events when you deploy the full configuration. When the event collection service must restart, QRadar does not restart it automatically. A message displays that gives you the option to cancel the deployment and restart the service at a more convenient time.
  9. Verify that event or flow sources that were reporting to the original host are being processed in the QRadar user interface.

    Results
    After the host is added back to the QRadar deployment, the deploy process ensures that required configuration is regenerated on the new appliance. The administrator can verify that log source data is being pulled and that flow data is being received by the new hardware. Any log sources that are not collecting data might require certificates to be moved to the new host.
 

Step 7: How to transfer event and flow data to the new hardware

The attached utility was designed by QRadar L3 engineering to facilitate moving data from /store/ariel of an old appliance to a new appliance. Data is moved in one month intervals to keep performance impact at a minimum. This utility does not move certificates or configurations, only data is /store/ariel/; however, it does leverage rsync, so SSH traffic must be allowed to migrate the data. The administrator might be required to accept SSH keys and provide root password for the target server to stat the transfer. Administrators need to also be aware that if they do not transfer private keys between hosts as outlined in a previous step, they are prompted to type a password each time the syncAriel.sh utility.

Note: The data transfer can be a lengthy process. Appliances can use cross-over cables if located in the same data center to expedite the transfer of event and flow information.

Procedure

  1. To copy data from the old Console to the new appliance, download the syncAriel.sh utility: syncAriel.sh
  2. Use SSH to log in to the old QRadar Console as the root user.
  3. Copy or SCP the file to the old Console, for example the /tmp directory.
  4. Navigate to the directory with the syncAriel utility and type: chmod +x syncAriel.sh.
  5. Type screen.
    Note: For data transfers, it is recommended that the administrator starts a screen session to reestablish the connection in case of a minor network outage. To detach the session so you can log out, type Ctrl-a and press d or use Ctrl+a, then Ctrl+d and use screen -r to reattach to the screen session.
  6. To run the utility, type:
    sh syncAriel.sh -i IP_address
    Where IP_address is the new Console's IP address. For a list of other commands, run the script without any parameters.
  7. Wait for the transfer to complete.
  8. Close the screen session.

    Results
    Data is migrated from /store/ariel of the old Console to the new Console appliance. Depending on how much data needs to be transferred this can be a lengthy process, so plan accordingly.

    Disconnects / Troubleshooting
    If your connection dropped or a network outage occurred, administrators can run the syncAriel.sh utility again to migrate data. The syncAriel.sh utility keeps track of files that have been rsync'd to the new appliance and data that has already been transferred is not copied a second time. Using screen helps to reconnect to an appliance for minor network issues.

Optional configurations
Appliances on a slower network connection can expand on the rsync examples to limit the transfer rate between appliances.
 
Procedure
  1. Log in to the old QRadar Console as the root user.
  2. To copy the data from the old hardware to the new appliance (targetserver), type the following command:

    Example 1: rsync -avz /store/ariel/ root@targetserver:/store/ariel
    Example 2: scp -pr /store/ariel root@targetserver:/store/

     
  3. Wait for the transfer to complete.
  

Step 8: Migration complete

The migration is complete. You may want to keep the old console on hand for a few days to ensure there are no other issues might arise that requires you to revert to the old appliance. Otherwise, after a week or two the old Console is no longer required and can be decommissioned or repurposed for non-QRadar uses.

 

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000cwtNAAQ","label":"Deployment"}],"ARM Case Number":"","Platform":[{"code":"PF016","label":"Linux"}],"Version":"7.5.0"}]

Document Information

Modified date:
22 August 2023

UID

swg21984607