User clicks "Transfer - Log Reports". User clicks tab "External Data Tracking". User selects a recent import, and clicks "Preview". The Controller client hangs. User waits a long time (e.g. 4 hours) and yet still the log report never appears.
During the hanging:
- the words "Not Responding" appear on the Windows title bar
- The CPU of the client goes to 100%
- The process that is running at 100% is CCR.EXE
Too many rows of data have been generated (via "track data") that it has exceeded the ability of Controller to handle the amount of data it is being asked to display.
When defining an import specification, the user has the choice of enabling "Track Data"
- If this is selected then it allows the use of the "External Data Tracking log reports"
- For more information, see Technote #1376909.
If "Track data" is enabled then it is possible to select/enable " Include Transformation Information"
- The user can select this option if you want to include the transformation information when using he Track Data option.
IMPORTANT: IBM recommends that:
- The "track data" option should only be used/enabled during the development phase of Controller
- In other words, it should only be used when testing the import specification, and should be disabled when the system is in production
- If using the "track data" option, then Controller should be limited to tracking no more than 30,000 rows.
- TIP: You can use the Server Preference "TRACKDATAROWS" to specify which rows should include transformation information when importing external files using track data. Rows not specified will still be imported but they will exclude transformation information.
Resolving the problem
For all production (non-development) import specifications, it should be unnecessary to use the "External Data Tracking" feature. Using the "track data" feature will cause strain and slow down the system
- Therefore it is recommended to disable 'Track Data' (inside the relevant 'production' import specification definition(s)):
- Click "Transfer - External Data - Define Import Specifications"
- Inside "Specification" box, select the relevant specification
- Open the 'General' tab
- Inside the "Options" section, untick the box "Track Data"
- Save changes.
- From now on, that import specification will no longer track data.
If there is a genuine reason why you need to use 'track data' for very large imports (e.g. in production) then there are several workarounds:
- Method #1 - Split the import into smaller chunks
- For example, instead of importing in one large chunk (e.g. one large CSV file) import in 2 or 3 chunks (2 or 3 smaller CSV files).
- Method #2 - If it is required that the system always uses 'track data', then limit the number of rows in the log report by using the server preference "TRACKDATAROWS".
- The value should be expressed in the form of a 'range' (1-x) where x is the maximum value
- The maximum recommended value of x is 30000
- Therefore, the maximum recommended range would be 1-30000
Steps for Method #2:
NOTE: This will only solve the problem for the *next* time that you perform a log report.
- In other words, the "External Data Tracking" function will work OK for all future imports, not for previous imports
- Logon to Controller using an administrative user (e.g. ADM)
- Click "Maintain - Configuration - General"
- Click "Server Preferences"
- Enter a new 'Variable Name' called: TRACKDATAROWS
- Enter a new 'Variable Value' called: 1-30000
- Click "Save"
- Close Controller
- Re-launch Controller
- Perform a data import
- Test that the log report works OK now.