JR44008: Parallel Engine in IS 8.7 has poor performance creating lookup tables.

Subscribe

You can track all active APARs for this component.

APAR status

  • Closed as program error.

Error description

  • This is an issue that has always existed in PX lookup.  The
    lookup create operator reads records from its input and writes
    those records to a temporary file.  Once all the records have
    been read the operator mmaps the file into memory and builds the
    hash lookup table in memory.  If APT_LUTCREATE_NO_MMAP is set
    malloc is used instead of mmap.  The very first piece, writing
    all the records to disk, is what takes a long time in this
    customer's case since it produces many small writes to the file
    system.
    
    The fix creates a new environment variable
    APT_LUTCREATE_FIXEDBLOCK that sets the size in MB of the memory
    block. The specified size of the memory block will be used to
    buffer records read by the Lookup create operator. The default
    value is 1MB when this APT environment is not set. Setting this
    environment variable to 0 will disable block buffering of input
    records causing each individual record to be written to disk
    upon being read. Buffering the records reduces the number of
    write operations. This improves performance on systems with
    contention in the file system.
    

Local fix

Problem summary

  • Parallel Engine in IS 8.7 has poor performance creating  lookup
    tables.
    

Problem conclusion

  •   This is an issue that has always existed in PX lookup.  The
    lookup create operator reads records from its input and writes
    those records to a temporary file.  Once all the records have
    been read the operator mmaps the file into memory and builds the
    hash lookup table in memory.  If APT_LUTCREATE_NO_MMAP is set
    malloc is used instead of mmap.  The very first piece, writing
    all the records to disk, is what takes a long time in this
    customer's case since it produces many small writes to the file
    system.
    
    The fix creates  a new environment variable
    APT_LUTCREATE_INMEMORY that when set will try to read all
    records into memory, build the lookup table in memory, and then
    do 1 large write of the table to disk. This improves performance
    on systems with contention in the file system, by reducing the
    number of writes.
    

Temporary fix

Comments

APAR Information

  • APAR number

    JR44008

  • Reported component name

    WIS DATASTAGE

  • Reported component ID

    5724Q36DS

  • Reported release

    870

  • Status

    CLOSED PER

  • PE

    NoPE

  • HIPER

    NoHIPER

  • Special Attention

    NoSpecatt

  • Submitted date

    2012-09-07

  • Closed date

    2012-10-15

  • Last modified date

    2013-09-17

  • APAR is sysrouted FROM one or more of the following:

  • APAR is sysrouted TO one or more of the following:

Fix information

  • Fixed component name

    WIS DATASTAGE

  • Fixed component ID

    5724Q36DS

Applicable component levels

  • R870 PSY

       UP

  • R910 PSY

       UP



Rate this page:

(0 users)Average rating

Document information


More support for:

InfoSphere DataStage

Software version:

8.7

Reference #:

JR44008

Modified date:

2013-09-17

Translate my page

Machine Translation

Content navigation