When building in replicated VOBs using IBM Rational ClearCase build tools (clearmake, omake, clearaudit), the build tools may crash with a "module not found" (0xc06d007e) exception if IBM Rational Common Licensing is used.
The following error may be present in the system event log:
Log Name: Application
Source: Application Error
Date: 5/31/2013 11:29:46 AM
Event ID: 1000
Task Category: Application Crashing Events
Faulting application name: clearmake.exe, version: 7.1209.0.138, time stamp: 0x50a30f58
Faulting module name: KERNELBASE.dll, version: 6.1.7601.17965, time stamp: 0x506dbe50
Exception code: 0xc06d007e
Fault offset: 0x0000c41f
Faulting process id: 0x1188
Faulting application start time: 0x01ce5e0b4c414cda
Faulting application path: C:\Program Files (X86)\IBM\RationalSDLC\ClearCase\bin\clearmake.exe
Faulting module path: C:\Windows\syswow64\KERNELBASE.dll
Report Id: 8a86517f-c9fe-11e2-805b-8851fb3f6283
Additionally, the following message may be present one or more places in the build logs:
clearmake: Leaving directory `N:\test\dir1\make'
*** Error code -1066598274
clearmake: Error: Build script failed for "longtarget"
This is the decimal representation of the hexadecimal exception code in the system event log.
These errors will generally occur no more that 15-20 minutes into the build.
Analysis of a provided memory dump file showed that the PATH environment variable was missing the ...\RationalSDLC\Common directory. This prevented loading of the Rational Common License DLL's and clearmake failed with the "Module Not Found" exception.
A deeper look into the build environment showed that the PATH environment variable was being changed during the build with the following construct:
override Path = $(TOOLROOT)/bin;$(CLEARCASE_DIR)/bin;$(SystemRoot)/SYSTEM32;$(SystemRoot)
Diagnosing the problem
If external tools can be installed in the build environment, the simplest way to determine if this issue is the cause of the crash is to collect a full process dump of the crashed ClearCase process and check the process environment block in a debugger.
- If "Windows Error Reporting" is configured as the default debugger, refer to http://msdn.microsoft.com/en-us/library/windows/desktop/bb787181%28v=vs.85%29.aspx for information on configuring to collect full memory dumps of crashed processes.
- If no default debugger is set, or the process dumps are not being collected by the above process, the System Internals Procdump tool can be configured to collect the dumps as needed. See http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx for more information.
Once the dumps are collected. They can be analyzed using the "Debugging Tools for Windows" available as part of the Windows SDK. See: http://msdn.microsoft.com/en-us/windows/hardware/gg463009 for download and installation instructions.
Once the tools are installed:
- Run the "windbg" tool. This is the GUI interface to the Windows debugging tools.
- Select File/Open Crash Dump
- Locate the dump created by one of the above tools and open it.
- In the command pane, enter:
and press Enter.
- A large amount of data will appear, the most important section is the "DllPath:" line. This shows where Windows will look when it needs to load DLL's. It will generally include the contents of the PATH environment variable (which is also in this output) and the standard Windows directories.
Note: you can also enter "!envvar PATH" to display only the PATH environment variable.
If this option is not available, collect:
- clearmake debug output.
- Any scripts used to start the builds.
- Any scripts used within builds if they in turn call clearmake.
The clearmake output will contain a "Processing description file..." line for each makefile that is being read. All of those makefiles will need to be analyzed to determine if any of them are redefining the path. Any scripts that are used to start builds will need to be similarly analyzed.
Resolving the problem
Completely overwriting the path as shown above is not an optimal practice. If tools need to be added to the path for a build, the best option is to prepend the tool directory to the existing path by changing the "Override" construct to:
override Path = $(TOOLROOT)/bin;$(PATH)
This " override" method is significantly less likely to cause problems if and when additional third party libraries are used for ClearCase tools.
Locating the PATH override may prove to be challenging. See the "If this option is not available" section in "Diagnosing the problem" above for the initial information to collect to locate where the PATH environment variable is being changed.