Intermittent "Error in AFC:!java.lang.OutOfMemoryError!" when running consolidation jobs that include AFCs
User runs a consolidation (e.g. Consolidate with Status) which includes AFC calculations. After this has finished running, user opens the 'batch queue'. Intermittently, inside the 'batch queue' there is an error message (in the job details).
<time> Advanced Formula Calculation is started.
Error in AFC:!java.lang.OutOfMemoryError!
Execution of AFC not OK for: xxxxx
<time + x> Advanced Formula Calculation is finished
NOTE: The error does appear all the time (intermittent issue).
There are several different possible causes for the generic "Error in AFC:!java.lang.OutOfMemoryError!" message.
- For more examples, see separate IBM Technote #1636718.
This IBM Technote shall specifically *only* relate to the scenario where the JAVA JRE environment (which hosts the AFC calculation engine) has run out of memory.
The default memory limitations of the JRE depends on which version of Controller you are using:
10.1.1308 (and earlier)
- Initial RAM = 256 Mb
- Maximum RAM = 512 Mb
- In other words, the JRE will attempt to allocate as much RAM as it needs (using its own algorithm) up to the 32-bit maximum limit (2Gb).
- However, this memory limit can be manually increased to match the needs of the customer (see later).
This has the following memory limitations:
10.1.1691 and 10.1.2681 (inclusive) 32-bit version (which is default for these versions)
By default, the 32-bit version of these version's JRE has its RAM settings 'dynamically' configured.
10.2.0 onwards (plus also 10.1.1691 and 10.1.2681 inclusive 64-bit version, which is optional)
The default (out of the box) maximum RAM value = 512 Mb
Problem could potentially affect any environment, but it is much more likely for customers who are:
(1) Using versions of Controller *before* 10.1.1 FP1 (10.1.1671)
(2) Running multiple batch process servers.
- For example, in the following real-life example the customer is running 3:
- The intermittent nature of the problem is explained by the fact that (randomly) there may have multiple concurrent AFC jobs running that swamp the available memory for the JAVA JRE environment.
Diagnosing the problem
Launch Task Manager on the controller application server, and search for the process 'CCRProxy.exe'.
- You should notice that its memory used increases (over time)
By default (for most Controller versions) CCRProxy.exe will reach a limit of approximately 650Mb before the problem occurs. This limit is because:
- By default, most Controller version's JRE has a maximum limit of 512Mb
- However, there will also be a small amount of extra RAM utilised by 'CCRProxy.exe' and 'jvm.dll', which will then take the total up to around 650Mb or so.
Resolving the problem
If necessary, upgrade to the latest version of Controller (ideally 10.2 or later).
- Afterwards, increase the JAVA maximum memory limit (by creating/modifying the file 'CCRProxy.options').
1. If necessary, upgrade to the latest version of Controller (ideally 10.2 or later).
- TIP: Controller 10.2 automatically comes with a 64-bit JAVA proxy
2. If upgrading to a version between 10.1.1691 and 10.1.2681 (inclusive) then modify the JAVA proxy from 32-bit to 64-bit
- For instructions, see separate IBM Technote #1618609.
3. If necessary, increase the amount of RAM on the Controller application server. In other words, please ensure that your Controller application server has enough free RAM (physically installed in the server) to cope with reserving this amount for your JAVA service.
- As a general guide, make sure that the Controller application server has at least twice the RAM that you specify inside "CCRProxy.options" (see later).
4. Logon to the Controller application server as a Windows administrator
5. Browse to the folder ...\server
- TIP: By default, this will be in a location similar to: C:\Program Files\IBM\cognos\ccr_64\server
6. Check to see if the file CCRProxy.options exists.
- By default this file will not exist. If it does exist then (as a precaution) create a backup of the file (for example, copy it to CCRProxy.options.backup)
7. Save the relevant file CCRProxy.options (attached to this Technote) inside that folder
TIP: Several example files are attached to this Technote. Download the appropriate ZIP file (for your needs) and extract the file:
- 1Gb_maximum.zip = suitable for basic needs only (typically for low-powered server with 32-bit JAVA)
- 2Gb_maximum.zip = suitable for standard 32-bit JAVA needs
- 4Gb_maximum.zip = suitable for standard 64-bit JAVA needs
- 6Gb_maximum.zip = suitable for advanced 64-bit JAVA needs
More Information: The file '
CCRProxy.options' should contain two parameters, which contains the memory sizes for initial and maximum heap size, for example:
8. Right-click on "My Computer" and choose "Manage"
9. Open the section "Services"
10. Restart the Windows service " IBM Cognos Controller Java Proxy"
(1) Reduce the number of batch services running to 1
or (2) Use less-complicated JAVA-related functions (e.g. less complicated AFCs).