DataStage jobs hung in a project in IBM InfoSphere DataStage
All DataStage jobs are hung in a project; they are running, but hung.
The workaround is to reboot the server.
Possible causes include:
- Excessive amount of files in &PH& directory for the project
- Incorrect job status
- Internal locks
Resolving the problem
Check the following:
- There is a &PH& directory in each DataStage project directory, which contains information about active stages that is used for diagnostic purposes. The &PH& directory is added to every time a job (both types: server and parallel/PX) is run, and needs periodic cleaning out.
To clear the file from within DataStage:
- Ensure there are no DataStage jobs running anywhere on the system by running "ps -ef | grep DSD"
- From the DataStage Administrator, go to the Projects page, select the project whose file you want to clear and click the Command button. The Command Interface dialog box opens.
- Type the following into the command field: CLEAR.FILE &PH& (all uppercase)
- Click Execute to run the command and clear the file.
- If the job processes have aborted, but the stattus indicates it is running this can be resolved by clearing the status of the job, check for locks, and then try running the job again.
Clear Status file should be used only when the status of a job hangs in "running" status.
To Clear the Status File of the job, from the DataStage Director, select the job and then click menu Job -> Clear Status File).
If this menu is grayed out, you first need to "Enable job administration in Director" in your project from the DataStage Administrator, and restart your Director.
- If you cannot clear the status file, please try checking/removing locks
Here are two methods for checking/removing locks (you need only perform one method to remove the locks)
From the DataStage (DS) clients:
- Launch DS Director
- Select Job -> Cleanup Resources (if Cleanup Resources is disabled, go to step 10 below)
- In the Processes pane, click Show All
- In the Locks pane, click Show All
- In the Locks pane, scroll to the job name in the Item ID field
- Note the PID/User # associated with the job
- Click on the PID # in the upper pane (Processes)
- Click Show by process (Locks pane)
- Click Release All (Locks pane)
- Launch DS Administrator
- In the Projects tab, highlight the job
- Click Properties
- Check Enable job administration in Director
- Click OK
- Click Close
- Exit DS Director and relaunch
- Perform steps 3 - 9 above.
From Unix/Linux server:
- Log into the server using the dsadm user
- cd to the DSEngine directory
- Enter . ./dsenv to source the dsenv file
- Enter ./bin/uvsh to get into DataStage prompt
- Run LIST.READU EVERY to list all the locks
- Check active record locks under "Item Id" column for job name or RT_CONFIG# or RT_LOG# (# matches the job description number)
- Write down the Inode numbers and user numbers associated with these locks
- Enter UNLOCK INODE inode# USER user# ALL
- You can use Q to get out of DataStage command line, e.g. ">" prompt
More support for:
Software version: 7.5, 8.0.1, 8.1, 8.5, 8.7, 9.1
Operating system(s): AIX, HP-UX, Linux, Solaris, Windows
Reference #: 1390366
Modified date: 14 November 2013