IV94530: AUTOMATIC FILE COLLECTION PROPAGATION DOES NOT WORK RELIABLE

PowerHA-7.2.0 SP5
PowerHA-7.2.0 SP3

APAR status

• Closed as program error.

Error description

• When the automatic file propagation is turned on
  the file propagation functionality does not work
  reliable. Files which are updated on a node in
  the time after the last local execution of the
  'automatic file propagation' and the next
  execution of the 'automatic file propagation'
  on a remote node, will not be propagated. This
  happens because each clfileprop execution is
  updating the timestamps of all files in
  HACMPfcmodtime ODM on all nodes with the
  current time.
  
  Example:
  --------
  
  root@ha1clC:/ # odmget HACMPfilecollection
  
  HACMPfilecollection:
          name = "Configuration_Files"
          description = "AIX and PowerHA
   SystemMirror configuration files"
          prop_sync = 1
          prop_auto = 1
  
  HACMPfilecollection:
          name = "HACMP_Files"
          description = "User-defined
   scripts configured to PowerHA SystemMirror"
          prop_sync = 1
          prop_auto = 1
  
  root@ha1clC:/ # grep -e "clfileprop: Automatic" \
   -e "clfileprop: Total" /var/hacmp/log/clutils.log \
   | tail -n 4
  Fri Jan 06 15:07:11 2017: clfileprop:
   Automatic file collection propagation called.
  Fri Jan 06 15:07:12 2017: clfileprop:
   Total number of files propagated to node ha2clC: 0
  Fri Jan 06 15:17:12 2017: clfileprop:
   Automatic file collection propagation called.
  Fri Jan 06 15:17:12 2017: clfileprop:
   Total number of files propagated to node ha2clC: 0
  
  root@ha2clC:/ # grep -e "clfileprop: Automatic" \
   -e "clfileprop: Total" /var/hacmp/log/clutils.log \
   | tail -n 4
  Fri Jan 06 15:10:53 2017: clfileprop:
   Automatic file collection propagation called.
  Fri Jan 06 15:10:53 2017: clfileprop:
   Total number of files propagated to node ha1clC: 0
  Fri Jan 06 15:20:53 2017: clfileprop:
   Automatic file collection propagation called.
  Fri Jan 06 15:20:53 2017: clfileprop:
   Total number of files propagated to node ha1clC: 0
  
  root@ha1clC:/ # odmget HACMPfcmodtime \
   | grep "modtime =" | sort -u
          modtime = 1483712455
  
  root@ha2clC:/ # odmget HACMPfcmodtime \
   | grep "modtime =" | sort -u
          modtime = 1483712455
  
  root@ha1clC:/ # perl -e 'print scalar
   localtime 1483712455;print "\n"'
  Fri Jan  6 15:20:55 2017
  -> timestamp of latest 'automatic file
     propagation', which happened on node ha2clC.
  
  Time of next 'automatic file propagation':
  - At node ha1clC: 15:27
  - At node ha2clC: 15:30
  
  A file, which will be updated on node ha2clC between
  15:21 and 15:27 will not be propagated, because the
  file propagation at 15:27 on node ha1clC will update
  timestamps in HACMPfcmodtime ODMs on both nodes with
  current time.
  
  So at 15:30 -when next automatic file propagation is
  executed on node ha2clC- the file itself will have
  an older timestamp than the timestamp in ODM:
  
  root@ha1clC:/ # odmget -q name=/etc/inetd.conf \
   HACMPfcfile
  
  HACMPfcfile:
          fc_name = "Configuration_Files"
          name = "/etc/inetd.conf"
  
  root@ha2clC:/ # touch /etc/inetd.conf
  
  root@ha2clC:/ # ls -l  /etc/inetd.conf
  -rw-rw-r--    1 root     system
   4822 Jan 06 15:25 /etc/inetd.conf
  
  root@ha1clC:/ # grep -e "clfileprop: Automatic" \
   -e "clfileprop: Total" /var/hacmp/log/clutils.log \
   | tail -n 4
  Fri Jan 06 15:17:12 2017: clfileprop:
   Automatic file collection propagation called.
  Fri Jan 06 15:17:12 2017: clfileprop:
   Total number of files propagated to node ha2clC: 0
  Fri Jan 06 15:27:12 2017: clfileprop:
   Automatic file collection propagation called.
  Fri Jan 06 15:27:12 2017: clfileprop:
   Total number of files propagated to node ha2clC: 0
  
  root@ha1clC:/ # odmget HACMPfcmodtime \
   | grep "modtime =" | sort -u
          modtime = 1483712834
  
  root@ha2clC:/ # odmget HACMPfcmodtime \
   | grep "modtime =" | sort -u
          modtime = 1483712834
  
  root@ha1clC:../hacmp/log # perl -e 'print
   scalar localtime 1483712834;print "\n"'
  Fri Jan  6 15:27:14 2017
  
  root@ha1clC:/ # grep -e "clfileprop: Automatic" \
   -e "clfileprop: Total" /var/hacmp/log/clutils.log \
   | tail -n 4
  Fri Jan 06 15:20:53 2017: clfileprop:
   Automatic file collection propagation called.
  Fri Jan 06 15:20:53 2017: clfileprop:
   Total number of files propagated to node ha1clC: 0
  Fri Jan 06 15:30:53 2017: clfileprop:
   Automatic file collection propagation called.
  Fri Jan 06 15:30:53 2017: clfileprop:
   Total number of files propagated to node ha1clC: 0
  
  root@ha1clC:/ # odmget HACMPfcmodtime \
   | grep "modtime =" | sort -u
          modtime = 1483713055
  
  root@ha2clC:/ # odmget HACMPfcmodtime \
   | grep "modtime =" | sort -u
          modtime = 1483713055
  
  root@ha1clC:/ # perl -e 'print scalar
   localtime 1483713055;print "\n"'
  Fri Jan  6 15:30:55 2017
  
  root@ha1clC:/ # ls -l /etc/inetd.conf
  -rw-rw-r--    1 root     system
   4822 Jan 03 16:43 /etc/inetd.conf
  
  root@ha2clC:/ # ls -l /etc/inetd.conf
  -rw-rw-r--    1 root     system
    4822 Jan 06 15:25 /etc/inetd.conf
  
  => Only timestmaps have been updated, but the
     /etc/inetd.conf file has not been propagated.
  

Local fix

• Copy the modified file manually to remote node(s).
  

Problem summary

• Files on remote nodes may never be propagated in various
  circumstances.
  

Problem conclusion

• Update HACMPfcmodtime entries properly. Use individual entries
  for files within directory.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

  IV92354

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  POWERHA SYSMIR

• Fixed component ID

  5765H3900

Applicable component levels