Job design tips

These topics give some hints and tips for the good design of parallel jobs.

InfoSphere DataStage Designer interface
The following are some tips for smooth use of the InfoSphere® DataStage® Designer when actually laying out your job on the canvas.
Processing large volumes of data
The ability to process large volumes of data in a short period of time depends on all aspects of the flow and the environment being optimized for maximum throughput and performance.
Modular development
You should aim to use modular development techniques in your job designs in order to maximize the reuse of parallel jobs and components and save yourself time.
Designing for good performance
Here are some tips for designing good performance into your parallel job from the outset.
Combining data
The two major ways of combining data in an InfoSphere DataStage job are by using a Lookup stage or a Join stage.
Sorting data
Look at job designs and try to reorder the job flow to combine operations around the same sort keys if possible, and coordinate your sorting strategy with your hashing strategy.
Default and explicit type conversions
When you are mapping data from source to target, you might need to perform data type conversions. Some conversions happen automatically, and can take place across the output mapping of any parallel job stage that has an input and an output link. Other conversions need a function to explicitly perform the conversion.
Using Transformer stages
In general, it is good practice not to use more Transformer stages than you have to. You should especially avoid using multiple Transformer stages where the logic can be combined into a single stage.
Using Sequential File stages
Certain considerations apply when reading and writing fixed-length fields using the Sequential File stage.
Using Database stages
The best choice is to use connector stages if available for your database. The next best choice is the Enterprise database stages as these give maximum parallel performance and features when compared to 'plug-in' stages

Parent topic: Parallel Job Reference