JR44008: Parallel Engine in IS 8.7 has poor performance creating lookup tables.

APAR status

• Closed as program error.

Error description

• This is an issue that has always existed in PX lookup.  The
  lookup create operator reads records from its input and writes
  those records to a temporary file.  Once all the records have
  been read the operator mmaps the file into memory and builds the
  hash lookup table in memory.  If APT_LUTCREATE_NO_MMAP is set
  malloc is used instead of mmap.  The very first piece, writing
  all the records to disk, is what takes a long time in this
  customer's case since it produces many small writes to the file
  system.
  
  The fix creates a new environment variable
  APT_LUTCREATE_FIXEDBLOCK that sets the size in MB of the memory
  block. The specified size of the memory block will be used to
  buffer records read by the Lookup create operator. The default
  value is 1MB when this APT environment is not set. Setting this
  environment variable to 0 will disable block buffering of input
  records causing each individual record to be written to disk
  upon being read. Buffering the records reduces the number of
  write operations. This improves performance on systems with
  contention in the file system.
  

Local fix

Problem summary

• Parallel Engine in IS 8.7 has poor performance creating  lookup
  tables.
  

Problem conclusion

• This is an issue that has always existed in PX lookup.  The
  lookup create operator reads records from its input and writes
  those records to a temporary file.  Once all the records have
  been read the operator mmaps the file into memory and builds the
  hash lookup table in memory.  If APT_LUTCREATE_NO_MMAP is set
  malloc is used instead of mmap.  The very first piece, writing
  all the records to disk, is what takes a long time in this
  customer's case since it produces many small writes to the file
  system.
  
  The fix creates  a new environment variable
  APT_LUTCREATE_INMEMORY that when set will try to read all
  records into memory, build the lookup table in memory, and then
  do 1 large write of the table to disk. This improves performance
  on systems with contention in the file system, by reducing the
  number of writes.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WIS DATASTAGE

• Fixed component ID

  5724Q36DS

Applicable component levels

• R870 PSY

     UP

• R910 PSY

     UP