IC99980: FMM0769E RECREATING THE VOLUME GROUP FROM THE HDISKS FAILED

Download Information: Version 4.1.0.2 Interim Fix: Tivoli Storage FlashCopy Manager
Download Information: Version 4.1.1 Tivoli Storage FlashCopy Manager

APAR status

• Closed as program error.

Error description

• IBM Tivoli Storage Flashcopy Manager mount operations on AIX 6.1
  TL8 or 7.1 TL2 and later  release levels may fail to perform
  the "recreatevg" command on volume groups.
  
  Executing the command
  backint_volume -u SID -f backup -i
  /oracle/SID/sapbackup/.FILE.lst -t volume_online -p
  /oracle/SID/112_64/dbs/initSID_mirror.utl -n
  /oracle/SID/sapbackup/.FILE.nls -c'
  
  will return the following error:
  
  16:50:56 FMM0769E Recreating the volume group from the hdisks
  hdisk3 hdisk4 hdisk5  failed.
  16:50:56 FMM0189I Semaphore released.
  16:50:56 FMM6925E Function call 'acsSetResources' failed.
  16:50:56 FMM6925E Function call 'mount_force' failed.
  16:50:56 FMM1559E Failed to verify consistency of DataContainer
  16:50:56 FMM1560E Not all file systems have been validated by
  the mount agents!
  16:50:56 16:50:56 FMM1554W The agent 'acsgen V. 3.2.0.4'
  terminated with exit code 1.
  16:50:57 FMM1233I Operation 'forced mount' completed.
  16:50:57 FMM6925E Function call 'mount_force' failed.
  
  Enabling a trace in the profile file will help to find that the
  command
  
  sh -x recreatevg -y SID___tdp1 -p -n -f -l /tmp/laa-ZagEl hdisk3
  hdisk4 hdisk5
  
  which tries to varyon VolumeGroups  despite Flashcopy Manager
  set -n flag.
  
  Trace file  acsgen.mount.<timestamp>.trace will contain the
  following information
  
  01/17/14   13:30:32.926 : hdwutil.cpp         (1120):
  HdwUtil::issueSystemCmd cat > /tmp/laa-ZagEl <<!eof
  orath1sapdat1lv:orath1sapdat1lv
  orath1sapdat2lv:orath1sapdat2lv
  orath1sapdat3lv:orath1sapdat3lv
  !eof
  sh -x recreatevg -y SID___tdp1 -p -n -f -l /tmp/laa-ZagEl hdisk3
  hdisk4 hdisk5  >>/tmp/eaa-4agEe 2>>/tmp/faa-4agEf
  echo hdwIntRC: $? >> /tmp/faa-4agEf
  01/17/14   13:30:39.548 : hdwutil.cpp         (1226):
  HdwUtil::issueSystemCmd + varyonvg -n SID___tdp1
  01/17/14   13:30:39.548 : hdwutil.cpp         (1226):
  HdwUtil::issueSystemCmd 0516-1971 varyonvg: The volume group is
  already varied on in other node. varyoff
  01/17/14   13:30:39.548 : hdwutil.cpp         (1226):
  HdwUtil::issueSystemCmd  the volume group in other node and
  retry the varyonvg, or use -O flag to
  01/17/14   13:30:39.548 : hdwutil.cpp         (1226):
  HdwUtil::issueSystemCmd  force varyon the volume group.
  01/17/14   13:30:39.548 : hdwutil.cpp         (1226):
  HdwUtil::issueSystemCmd + test_return 43
  01/17/14   13:30:39.548 : hdwutil.cpp         (1226):
  HdwUtil::issueSystemCmd 0516-1320 recreatevg: Unable to recreate
  volume group.
  01/17/14   13:30:39.548 : hdwutil.cpp         (1226):
  HdwUtil::issueSystemCmd + cleanup
  01/17/14   13:30:39.548 : hdwutil.cpp         (1226):
  HdwUtil::issueSystemCmd hdwIntRC: 1
  ...
  13:30:39.657 ffffffff (102) esd-defaultlog.cpp:129 FMM0769E
  Recreating the volume group from the hdisks hdisk3 hdisk4 hdisk5
  failed.
  
  
  Customer/L2 Diagnostics:
  
  It has been identified that the problem is caused by the
  different behavior that was introduced in
  the command recreatevg on AIX starting from version 6.1 TL8 and
  later
  
  Tivoli Storage Manager Versions Affected: All IBM Tivoli
  Flashcopy Managers on AIX platforms
  
  Initial Impact:  Medium
  
  Additional Keywords:
  recreatevg -n hdwIntRC  43  AIX 7.1 TL2 6.1 TL8 FMM0769E
  

Local fix

• Local Fix:
  Contact IBM Support
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * Users of IBM Tivoli Storage FlashCopy Manager for Unix v     *
  * 2.1, v 2.2, v 3.1, v3.2, v 4.1 using AIX 6.1 TL8 or AIX 7.1  *
  * TL2 or                                                       *
  * later with AIX LVM mirroring and enhanced concurrent capable *
  * volume groups.                                               *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available.                           *
  * This problem is targeted to be fixed in level 4.1.0.2        *
  ****************************************************************
  *
  

Problem conclusion

• IBM Tivoli Storage FlashCopy Manager for Unix will be updated
  with fix for the incorrect handling of the AIX recreatevg
  command.
  

Temporary fix

• When IBM Tivoli Storage Flashcopy Manager is used on AIX 6.1 TL8
  or 7.1 TL2 and later AIX release levels and
  AIX LVM mirroring is used in combination with enhanced
  concurrent capable volume groups then the following workaround
  can be used until a fix is available:
  
  1. Create a file (used as shell script) called "recreatevg" in
  the FlashCopy Manager instance directory on the backup system.
  This recreatevg script will have one line:
  /usr/sbin/recreatevg -O $*
  This file "recreatevg" must have the same owner and permissions
  as /usr/sbin/recreatevg.
  
  2. Modify PATH environment variable for the FlashCopy Manager
  acsgen process running on the backup system. To achieve this, do
  the following:
  - rename /<fcm_instance_path>/acsgen to
  /<fcm_instance_path>/acsgen_orig  (keep the original permissions
  and owner)
  - create new file with name 'acsgen' and exactly the same
  permissions as original acsgen (incl. s-bit and root:dba owner)
  - in the new file 'acsgen'
     a.  modify PATH  (e.g. export
  PATH=/<fcm_instance_path>/:$PATH ), so that the replaced
  recreatevg script will be called instead of original one
     b.  call the original acsgen_orig
  (/<fcm_installation_path>/acsgen_orig $*)
  This would result in a script like:
  export PATH=/<fcm_instance_path>/:$PATH
  /<fcm_instance_path>/acsgen_orig $*
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  FLSHCPY ORACLE

• Fixed component ID

  5608AC6OS

Applicable component levels

• R21A PSN

     UP

• R22A PSN

     UP

• R31A PSN

     UP

• R32A PSN

     UP

• R41A PSY

     UP