Maximum number of InfoSphere DataStage Jobs in a project

Technote (troubleshooting)


Problem(Abstract)

Filesystems can have a hard limit on number of subdirectories that can be contained in a directory, for example 32766 (32K -2) for JFS on AIX, EXT3 on Linux and vxfs/hfs on HP-UX.

For every job, DataStage will create hash files as subdirectories in the project directory. Here is a list of the subdirectories created for every job. In this list "nn" represents the number (or id) of the job:

DS_TEMPnn
RT_CONFIGnn
RT_STATUSnn
RT_LOGnn
RT_BPnn
RT_BPnn.O
RT_SCnn (only for parallel jobs)

As the number of subdirectories in a directory is limited to 32766, the maximum number of jobs for a project will be, in theory, approximately 4600 parallel jobs (32766/7 - 80 where 80 is for overhead of project default directory) or 5380 server jobs (32766/6 - 80). In practice, the number might be smaller due to subdirectories not counted above, such as directories created by QualityStage jobs.

Symptom

When the maximum number of subdirectories is reached in the project directory, an error similar to below can occur when trying to save a newly created job:

Error On CREATE.FILE command
Creating file "RT_CONFIGnn as type 30, mkdbfile: connot create file
RT_CONFIG4817. Unable to create operating system file "RT_CONFIG4817"

Where nn is the job number of the newly created job.


Resolving the problem

1. To identify if the limit of subdirectories in the project directory has been reached, run the command below. An error will be given in case maximum number has reached:

mkdir Test

2. If the maximum number of subdirectories is reached, consider:

Exporting jobs in the project category by category, reallocate jobs into new projects;

or


Use EXT4 if IS server is on Linux platform, however, from project management point of view, this needs to be considered carefully as it might be hard to manage a project that contains more than 4,000 jobs;

or


Move some subdirectories to another directory then use a softlink to link them back into the project directory. This approach requires lots of system management efforts and is recommended to engage with your systems administrator.


Rate this page:

(0 users)Average rating

Document information


More support for:

InfoSphere DataStage

Software version:

7.5.1, 7.5.2, 7.5.3, 8.0, 8.0.1, 8.0.1.1, 8.0.1.2, 8.0.1.3, 8.0.2, 8.1, 8.1.0.1, 8.1.0.2, 8.5, 8.5.0.1, 8.5.0.2, 8.7, 8.7.0.1

Operating system(s):

AIX, HP-UX, Linux

Reference #:

1611379

Modified date:

2012-11-08

Translate my page

Machine Translation

Content navigation