Troubleshooting
Problem
The IBM Spectrum Protect backup-archive client, Space Management or API processes 7.1.8 or 8.1.2 or higher level, can fail when used in combination with other products, like IBM Spectrum Scale or IBM Db2, by using SSL/TLS communication to connect the IBM Spectrum Protect Server.
Symptom
The IBM Spectrum Protect processes reports an error message to the command-line output, the dsierror.log, or the dsmerror.log.
Possible errors reported are: ANS1579E, ANS9020E, ANS5283E, ANS2050E, ANS1025E, or undefined symbol error for symbol GSKKM_OpenKeyDbData.
An example for an error message issued is the following:
<process name>: relocation error: <process name>: symbol GSKKM_OpenKeyDbData, version GSKIKM not defined in file libgsk8km_64.so with link-time reference
Process name can be dsmc, dsmmigrate or dsmrecall
See also related Spectrum Scale APAR IJ03770
https://www-01.ibm.com/support/entdocview.wss?uid=isg1IJ03770
Cause
The problem occurs if the version of the GSKit library, included in IBM Spectrum Scale or IBM Db2, is installed at a lower version, compared to the GSKit version that is part of the IBM Spectrum Protect backup-archive client, Space Management, or API packages. The problem occurs only by using SSL/TLS communication with the IBM Spectrum Protect Server.
The IBM Spectrum Scale programs uses the environment variable LD_LIBRARY_PATH (Linux) and LIBPATH (AIX) to link the GSKit version that is included in the IBM Spectrum Scale package.
IBM Spectrum Protect uses the GSKit library installed to a global library path.
If IBM Spectrum Protect backup-archive client, Space Management or API programs are called by IBM Spectrum Scale programs they use the wrong library path for the environment.
If the GSKit library used by IBM Spectrum Scale is not compatible with the library required by IBM Spectrum Protect the processes fail.
Environment
Integrated environments between IBM Spectrum Protect backup-archive client, Space Management, API and IBM Spectrum Scale or IBM Db2. Especially dsmc, mmbackup and mmapplypolicy commands are affected.
Diagnosing The Problem
The backup or the HSM processing fails with the error message described in section "Symptom".
Resolving The Problem
For IBM Db2 one of the following workarounds can be applied to resolve the problem:
IBM Db2 workaround example:
LD_LIBRARY_PATH=/usr/local/ibm/gsk8_64/lib64/:$LD_LIBRARY_PATH
For IBM Spectrum Scale (GPFS) one of the following workarounds can be applied to resolve the problem:
IBM Spectrum Scale workaround example:
Backup:
Modify the following files where new code is marked in bold and code to remove isstrikethrough.
1. /usr/lpp/mmfs/bin/tsbackup33
...
$backupProgram = $tsmInstallDir . "/dsmc";
}
$backupProgram = "LD_LIBRARY_PATH= LIBPATH= ".$backupProgram;
$archiveProgram = $backupProgram;
...
2. /usr/lpp/mmfs/bin/mmbackup
The following changes apply to GPFS 5.x only:
...
backupBinary=${backupProgram}
backupProgram="env LD_LIBRARY_PATH= LIBPATH= ${backupProgram}"
archiveProgram=${backupProgram}
...
"TSM dsmc client is not found at:\n"\
"$backupProgram$backupBinary \n"\
...
The following changes apply to GPFS 4.x only:
...
backupProgram="${tsmInstallDir}/dsmc"
fi
backupBinary=${backupProgram}
backupProgram="env LD_LIBRARY_PATH= LIBPATH= ${backupProgram}"
archiveProgram=${backupProgram}
...
# Ensure this is a valid TSM BA client.
if [[ ! -x$backupProgram $backupBinary && $useTSM = true ]]
...
"TSM dsmc client is not found at:\n"\
"$backupProgram$backupBinary \n"\
...
# Check if the node we are on is has tsm client installed.
if [[ ! -x$backupProgram $backupBinary ]]
then
print -u2 -- ::POLICY_TEST_ERROR::\$serverName::$backupProgram $backupBinary could not be found.
...
# Check if the node we are on has tsm client installed.
if [[ ! -x$backupProgram $backupBinary ]]
then
print -u2 -- ::POLICY_TEST_ERROR::\$serverName::$backupProgram $backupBinary could not be found.
...
3. /usr/lpp/mmfs/bin/mmimgbackup
...
# TSM config files and parameters
backupProgram=${defBackupProgram}
backupBinary=${backupProgram}
backupProgram="env LD_LIBRARY_PATH= LIBPATH= ${backupProgram}"
configFile=${defaultTSMInstall}dsm.sys
...
# Ensure the backup program exists and can be run
######################################################
if [[ ! -x$backupProgram $backupBinary && -z $noTSMOpt && $outputMode = "file" ]]
...
Space Management:
Modify the file /usr/lpp/mmfs/samples/ilm/mmpolicyExec-hsm.sample (or the local copy you use). Add the following lines (marked with bold characters):
# Set binding to HSM commands
$MigrateCommand = "/usr/bin/dsmmigrate";
$PremigrateCommand = "/usr/bin/dsmmigrate";
$RecallCommand = "/usr/bin/dsmrecall";
$MigrateCommand = "LD_LIBRARY_PATH=\"\" ".$MigrateCommand;
$PremigrateCommand = $MigrateCommand;
$RecallCommand = "LD_LIBRARY_PATH=\"\" ".$RecallCommand;
Replace the following lines (marked with bold characters and strikethrough) with a single line (marked with characters in Italic) :
# Handle TEST command by verifying that HSM commands are installed
if ($command eq "TEST") {
# Verify HSM commands are installed on this node
print "$command -x $MigrateCommand\n" if ($Verbose > 0);
exit 1 if !(-x $MigrateCommand);
if ($PremigrateCommand ne $MigrateCommand) {
print "$command -x $PremigrateCommand\n" if ($Verbose > 0);
exit 1 if !(-x $PremigrateCommand);
}
if (($RecallCommand ne $MigrateCommand) &&
($RecallCommand ne $PremigrateCommand)) {
print "$command -x $RecallCommand\n" if ($Verbose > 0);
exit 1 if !(-x $RecallCommand);
}
exit 1 if !(-x "/usr/bin/dsmmigrate"); # Single line to add
print "$0: $command Ok\n" if ($Verbose > 0);
exit 0;
}
...
- Upgrade to new IBM Db2 version containing at least the GSKit version 8.0-50.78
- Set the LD_LIBRARY_PATH to point first to use the GSKit path installed by IBM Spectrum Protect
IBM Db2 workaround example:
LD_LIBRARY_PATH=/usr/local/ibm/gsk8_64/lib64/:$LD_LIBRARY_PATH
For IBM Spectrum Scale (GPFS) one of the following workarounds can be applied to resolve the problem:
- Upgrade to new GPFS version containing at least the GSKit version 8.0-50.78
- Unset the LD_LIBRARY_PATH in the GPFS scripts (see example) before using the dsmc, dsmmigrate or dsmrecall commands.
IBM Spectrum Scale workaround example:
Backup:
Modify the following files where new code is marked in bold and code to remove is
1. /usr/lpp/mmfs/bin/tsbackup33
...
$backupProgram = $tsmInstallDir . "/dsmc";
}
$backupProgram = "LD_LIBRARY_PATH= LIBPATH= ".$backupProgram;
$archiveProgram = $backupProgram;
...
2. /usr/lpp/mmfs/bin/mmbackup
The following changes apply to GPFS 5.x only:
...
backupBinary=${backupProgram}
backupProgram="env LD_LIBRARY_PATH= LIBPATH= ${backupProgram}"
archiveProgram=${backupProgram}
...
"TSM dsmc client is not found at:\n"\
"
...
The following changes apply to GPFS 4.x only:
...
backupProgram="${tsmInstallDir}/dsmc"
fi
backupBinary=${backupProgram}
backupProgram="env LD_LIBRARY_PATH= LIBPATH= ${backupProgram}"
archiveProgram=${backupProgram}
...
# Ensure this is a valid TSM BA client.
if [[ ! -x
...
"TSM dsmc client is not found at:\n"\
"
...
# Check if the node we are on is has tsm client installed.
if [[ ! -x
then
print -u2 -- ::POLICY_TEST_ERROR::\$serverName::
...
# Check if the node we are on has tsm client installed.
if [[ ! -x
then
print -u2 -- ::POLICY_TEST_ERROR::\$serverName::
...
3. /usr/lpp/mmfs/bin/mmimgbackup
...
# TSM config files and parameters
backupProgram=${defBackupProgram}
backupBinary=${backupProgram}
backupProgram="env LD_LIBRARY_PATH= LIBPATH= ${backupProgram}"
configFile=${defaultTSMInstall}dsm.sys
...
# Ensure the backup program exists and can be run
######################################################
if [[ ! -x
...
Space Management:
Modify the file /usr/lpp/mmfs/samples/ilm/mmpolicyExec-hsm.sample (or the local copy you use). Add the following lines (marked with bold characters):
# Set binding to HSM commands
$MigrateCommand = "/usr/bin/dsmmigrate";
$PremigrateCommand = "/usr/bin/dsmmigrate";
$RecallCommand = "/usr/bin/dsmrecall";
$MigrateCommand = "LD_LIBRARY_PATH=\"\" ".$MigrateCommand;
$PremigrateCommand = $MigrateCommand;
$RecallCommand = "LD_LIBRARY_PATH=\"\" ".$RecallCommand;
Replace the following lines (marked with bold characters and strikethrough) with a single line (marked with characters in Italic) :
# Handle TEST command by verifying that HSM commands are installed
if ($command eq "TEST") {
# Verify HSM commands are installed on this node
print "$command -x $MigrateCommand\n" if ($Verbose > 0);
exit 1 if !(-x "/usr/bin/dsmmigrate"); # Single line to add
print "$0: $command Ok\n" if ($Verbose > 0);
exit 0;
}
...
Related Information
[{"Product":{"code":"SSEQVQ","label":"IBM Spectrum Protect"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":"Client","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"}],"Version":"8.1.2;7.1.8;8.1.4","Edition":"Enterprise","Line of Business":{"code":"LOB26","label":"Storage"}}]
Was this topic helpful?
Document Information
Modified date:
10 September 2019
UID
swg22011742