WebSphere Application Server Update Installer fails to install a fix pack due to a "mkdirs failed for", "failed to delete", "text file busy", or "Error 79 -- Unable to open destination file" error on Windows

Resolving The Problem

Review the error symptoms
If the Update Installer reports errors updating "java.exe" or "WASServiceMsg.dll", then proceed to the error symptom list below. Those particular files are special cases which have specific solutions.

If the Update Installer is reporting an issue updating a file such as a ".jar" file or ".xml" file, then make another attempt to install the maintenance pack if there is sufficient time available in your maintenance window.

There are two reasons why it is recommend to attempt to install the fix pack again. One reason is that there are some kinds of file-access errors that are caused by a "race condition". There is a chance that a subsequent attempt to install the fix pack will be successful, which would allow a system administrator to complete the update and move to other projects. (Further below, the technote explains potential resolutions for these problems, so please be assured that there are more robust solutions available than simply "trying to install again".) The other reason to attempt to install the fix pack again is because it might reveal a clue which is very helpful to the troubleshooting process. When the Update Installer encounters these kinds of file access errors, it is extremely helpful to determine if the Update Installer fails on exactly the same file, or on different files, in subsequent attempts.

Examine the name of the files and the error messages which caused the Update Installer to fail during each maintenance pack attempt. Review the following list of symptoms to determine which action to take.

• Error message states "mkdirs failed for" or "failed to delete" a different file during each attempt
  If you make several attempts to install or uninstall the same fix pack, and the Update Installer reports errors while updating different files each time, then review the types of files that the Update Installer is having issues with.
  
  If the Update Installer is reporting issues with "text" file types, such as .txt, .xml, .jsp, or .html, then an indexing utility might be interfering with the process. Review the solution below, Shut down indexing services.
  
  If the Update Installer is reporting issues with "binary" or "code" file types, such as .dll, .jar, or .exe, then an antivirus or anti-malware utility might be interfering with the process. Review the solution below, Shut down real-time security scanners.
  
  
• Error message states "mkdirs for" or "failed to delete" the same file even after multiple attempts
  If you make several attempts to install or uninstall the same fix pack, and the Update Installer consistently reports errors while updating the same file, then there might be an application server running, or there is a problem with that particular file. First review the solution below, Shut down all processes associated with the product. If that does not work, then try the solution Verify the integrity of a particular file.
  
  
• Error message states "Text file busy" and Update Installer fails to update "java" or "java.exe" or any DLL file:
  This is most likely caused by a process that is using the product's Java files. An application server process might still be running, or one of the product's utilities (such as wsadmin) is running. It is also possible that a third-party product is attempting to use the product's Java files. In this case, refer to the solution below, Shut down all processes associated with the product.
  
  
• Error message states "Error 79" or "Failed to delete" on "WASServiceMsg.dll":
  This is caused by a scenario where the Windows operating system has locked the WASServiceMsg.dll file so that it can interpret messages logged in the Windows Event Log. The solution is described in technote 1230924. In addition, some alternative steps are outlined in the solution below, Disable automatic server startup and reboot the system.
  
  
• Other symptoms for failing to update a particular file
  If the Update Installer's failure symptoms do not closely match the symptoms listed above, or the symptoms seem to mix together several different symptoms, then you might need to try a combination of these solutions. It is recommended that you attempt the solutions in this order:
1. Shut down all processes associated with the product
2. Disable automatic server startup and reboot the system
3. Verify the integrity of a particular file (if Update Installer consistently fails on one file)
4. Shut down indexing services
5. Shut down real-time security scanners

Shut down all processes associated with the product
In order to apply maintenance packs to the product, all of its servers and utilities must be shut down during the update process. In addition, if any third-party products directly access the product's files (such as performance monitoring tools which read the product's Java DLL files), then those products must be shut down as well. This means that no application servers, node agents, or deployment managers can be running while the update process is underway. The Update Installer makes efforts to detect whether processes are running before starting to install or uninstall a maintenance pack; however, it is feasible for certain processes to be undetected.

If a process is running, then some of the product's files will be locked by the running process. It is impossible for the operating system to modify a file that is locked. If the Update Installer attempts to modify a locked file, the operating system will prevent it. This causes the Update Installer to encounter an error such as "Text file busy" or "Failed to delete" at some point while applying a maintenance pack.

Use the methods described below to determine if any of the product's files are locked. These methods are effective in situations where Update Installer is consistently unable to update a .exe, .dll, or .jar file.

• Check the list of Windows Services to ensure they are shut down
  Open the Windows Services panel, and shut down any services which have these names:
  
  IBM WebSphere Application Server
  WebSphere Application Server
  IBM HTTP Server
  
  Hints for opening the Windows Service panel:
  
  On Windows XP and Windows 2003, go to the Start Menu and choose the Run option, then run the program named "services.msc".
  
  On Windows Vista, Windows 7, Windows 2008, and any later Windows operating systems, it is necessary to run the Services utility as an admistrative user. To do this, open the Start Menu and type "services.msc" in the search box. Then, right-click the "services.msc" item which appears in the list, and choose "Run as Administrative user".
  
  
• Use the Task Manager to identify Java processes
1. Open the Windows Task Manager utility. On some versions of Windows, you can use the CTRL-SHIFT-ESC key combination to open the Task Manager. Or, use the CTRL-ALT-DELETE key combination to bring up an options panel which allows you to open the Task Manager. Or, use the Start > Run menu item and run the process called "taskmgr.exe".
   
   
2. Click the Processes tab in the Task Manager. If you have administrative access to the system, look for a button or checkbox named Show processes from all users, and select that option. (This allows you to see if any other users on the system are running product processes.)
   
   
3. Click the Image Name column header to list running processes in alphabetical order. Then, look through the process list for any process named "java.exe", "javaw.exe", or "WASService.exe". Most WebSphere Application Server processes use "java.exe" to run product utilities and servers, so pay close attention to these processes. (Also, the IBM HTTP Server uses processes named "apache.exe", so pay attention to those processes for updating that product.)
   
   If you are reasonably sure that no other Java processes should be running on the system, then terminate any Java processes that are found in the task list. If other application servers do need to be running, then use caution and determine which processes, if any, should be terminated.
   
   Newer editions of Windows allow you to right-click a process in the list and select Properties. This spawns a dialog box that shows the path to the process executable. This information can help determine whether a process is running using the product's Java executable process. Older editions of Windows do not have this capability, so you will need to use other clues like the Process ID to determine whether the java.exe process belongs to the WebSphere product.
   
   
4. Once the Java processes are shut down, attempt to apply the maintenance pack again.
   
   
• Use the tasklist utility to identify owners of DLL file locks
  Newer editions of Windows offer a command-line tool called "tasklist", which can list processes that are responsible for holding a lock on a particular DLL file. Use this method of identifying a running process if Update Installer consistently reports that it is unable to update a particular DLL file.
  
  If the Update Installer is consistently unable to update a particular DLL file, then make note of the file name. Then, open a command prompt and run the following command. Substitute the name of the DLL file for <DLL_name> below.
  
  

tasklist /m <DLL_name>

The command will list any processes which have loaded and locked a particular DLL file. Terminate these processes to release that lock, then try applying the maintenance pack again.

• Use the Process Explorer utility to identify owners of file locks
  A utility named Process Explorer is available from Microsoft. The utility is available on Process Server download page. Follow the instructions provided by Microsoft to download and run this tool. This Microsoft Windows tool can help identify a process which is consistently holding a lock on a file. Use this method of identifying a running process if Update Installer consistently reports that it is unable to update a particular file, regardless of the type of file.
  
  The latest instructions associated with Process Explorer are available on the page linked above. In general, it is recommend that you use the "Search" function and supplying the name of the file that the Update Installer is consistently unable to update. If Process Explorer detects any processes locking that file, it will list the processes. Those processes can be shut down (or forcibly terminated). Once the processes are shut down, attempt to apply the maintenance pack again.
  

Disable automatic server startup and reboot the system
This solution is strongly recommended if the Update Installer reports issues while updating .exe files or DLL files, especially if the file is WASServiceMsg.dll. This solution outlines the process of preventing processes from automatically starting up when the system is rebooted, so that the operating system does not attempt to load and lock files associated with the product after a reboot.

1. Prevent any WebSphere Application Server or IBM HTTP Server services from starting up automatically when the system starts. To do this, first open the Windows Services panel.
   
   
2. In the Windows Services panel, look for the IBM WebSphere and IBM HTTP Server processes associated with the products you will update.
   
   
3. For each IBM WebSphere and IBM HTTP Server service, go to the properties for the service and choose "Manual" start-up (instead of "Automatic") startup. Keep track of each service you change, because you will want to revert these changes later.
   
   
4. In case you need to make other configuration changes to the system, make them now, before proceeding to the next step.
   
   
5. Reboot the system. Make sure that none of the IBM WebSphere or IBM HTTP Server services have started when the system comes up.
   
   
6. Attempt to apply the maintenance packs. (You might need to uninstall the partially-installed failed maintenance pack first.)
   
   
7. Upon successful installation of the maintenance packs, go to the Windows Services panel once again. Change the start-up settings on the services back to their original settings.
   
   

Verify the integrity of a particular file
If a particular file consistently causes the Update Installer to fail, then review the file's permissions to determine if there are insufficient rights for the Update Installer to modify or delete the file.

Next, try to obtain a new copy of the maintenance pack. If a maintenance pack is somehow corrupt, it might cause the Update Installer to report that an existing file has a problem, when the problem is actually a corrupt maintenance pack file. Replacing the corrupt maintenance pack file with an intact one might resolve a problem with updating product files.

If the maintenance pack does not appear to be corrupt, then turn your attention back to the file reported in the Update Installer error. Check that file's size. If the file is a zip or jar file but its size is zero bytes, then this could indicate that the file has been corrupted or lost. In that case, determine if the file is available in a recent system backup, then recover the file. Or, attempt to use the Update Installer to uninstall the failed maintenance package.

If the file appears to have correct permissions and it is not zero bytes, then determine if there is a filesystem issue affecting that file. A simple test to determine this is to make a copy of the file, then delete the original file, and then rename the copy file to the original file name. This forces the operating system to duplicate the file and then discard the original copy. If there is an issue with the filesystem that affects this file, the copy operation will fail, which indicates a filesystem problem. Contact your system administrator to investigate the issue further, as it might be necessary to run health checks on the filesystem.


Shut down indexing services and other filesystem utilites
Indexing services, such as the Microsoft Indexing Service built-in to Windows or Google Desktop, can potentially interrupt a maintenance pack update. These programs read the contents of text files and archive data in order to maintain a searchable index of the files on the system. These programs can potentially open and read a file which the Update Installer is attempting to delete or modify, which can cause an update to fail.

Filesystem utilities which monitor and defragment the filesystem in real-time, such as Diskeeper 2010 and related products, can have a similar effect as indexing services. These programs monitor filesystem activity, and might relocate files on the disk for more efficient access. While doing this, these programs can potentially open and lock a file which the Update Installer is attempting to delete or modify.

General advice regarding indexing services
If an indexing service interferes with the Update Installer, then the Update Installer will report a failure to update different files for each attempt at applying a maintenance pack. The failures will appear to be "random", and the failures might occur for any file type. (It is more likely for failures to occur on "text" file types, such as .xml, .txt, .html, or .jsp files, since indexing services will attempt to index human-readable text.)

If the system is running a third-party indexing service, such as Google Desktop, then disable that utility while running the Update Installer. Alternatively, if the third-party indexing service allows you to add an exclusion rule which prevents the utility from reading files in the product's directory, that can help avoid issues as well.

Windows Indexing Service
If the system is running the Microsoft Windows Indexing Service or Windows Search service, then turn off that service to avoid issues with applying fix packs using Update Installer. The name of the service varies, depending on the edition of Windows.

The most straightforward way to disable the service is to open the Windows Services Panel and look for the "Indexing Service" or "Windows Search" service. Right-click the service and choose the "stop" option. Once the service is stopped, then proceed to use the Update Installer to apply the maintenance pack. The service can be turned back on after the Update Installer is complete.

Alternatively, you can exclude the product's files from being indexed. To do this, have the system administrator update the product's files' attributes. (This can be accomplished by right-clicking the product's main folder, selecting Properties, selecting Advanced... in the Attributes section, and then turning off the option labeled, "Allow files in this folder to have contents indexed in addition to file properties".) There are other alternative ways of modifying the Indexing Services; please refer to Microsoft or other documentation to learn how.


Shut down real-time security scanners
Real-time security scanner software can also cause issues, similar to the indexing services discussed above. "Real-time security scanner" software includes antivirus software which scans all files written to and read from the filesystem, as well as anti-malware software that monitors system behavior looking for malicious activity. These programs can potentially open and read a file which the Update Installer is attempting to delete or modify, which can cause an update to fail.

Antivirus software is more prone to scan "binaries" such as executables, DLL files, and .jar files. If the Update Installer fails to update these types of files, then antivirus software could be contributing to the problem.

If the security controls on the system permit you to temporarily disable the real-time virus scan, then please do so before running the Update Installer. Or, add a rule to the antivirus software to exclude the product files from being scanned while the Update Installer is running.

The suggestion to turn off antivirus software is taken very seriously by clients. If you require further information as to why this is necessary, read the section below. The section below does not have any further troubleshooting hints, so if you do not require further information, you might skip reading this section. However, if your local system administrators express concern, they might need to review the next section.


Explanation of issues caused by race conditions
The suggestion to turn off antivirus software is taken very seriously by clients running production systems. Often, IBM Support is asked to provide a statement in order to satisfy business justification for doing this. The statement below gives a high-level explanation as to why the issue can occur. This section of the technote does not have further troubleshooting advice, so you might skip this section if you are not interested in the high-level explanation of race conditions.

Typically, if an indexing service or antivirus software interferes with Update Installer operations, it is due to something known as a "race condition". Below, indexing services and antivirus software are collectively referred to as "scanner software".

This occurs because scanner software monitors the filesystem to detect modifications made to files. As the Update Installer modifies and deletes product files, the scanner software might read information from files or directories that the Update Installer is in the midst of updating. A file or directory might be locked briefly while the scanner software reads it. If the file or directory is released by the time the Update Installer attempts to write or delete the file, then there is no problem. However, sometimes the indexing service does not release a file or directory from a lock quickly enough, which interferes with the Update Installer's next operation. (This is the "race condition".)

The problem usually occurs when the Update Installer attempts to delete a file or directory, then immediately create a replacement for that file or directory. This is the standard method the Update Installer uses to deliver file replacements. During this process, the Windows operating system will respect the Update Installer's command to delete the file. However, if scanner software has locked the file in that same brief period of time, the operating system will mark the file as "delete pending" instead of actually deleting the file. If the lock is released, then the file is deleted as soon as the lock is released. If the lock is not released, then the Update Installer will attempt to create the new version of the file it is replacing, and the new file will collide with the old file, which would have been deleted if it were not locked.

When a system is susceptible to this issue, there is usually a small chance per file updated that an error will occur. When applying a fix pack, thousands of files are updated, so although the chance for each individual update to fail is small, there is a much greater chance that the overall operation will fail at some point.

The problem occurs when scanning software reacts more slowly than Update Installer's delete-and-replace operation, which occurs in a window of hundredths of a millisecond. This makes it impossible to predict the problem in advance. Also, there are a number of uncontrollable factors which contribute to the issue (such as what the CPU is handling at that precise moment, the speed at which files are accessed in that instant, and whether or not filesystem data has a favorable cache hit at that moment). This makes it impossible to mitigate the problem when the scanner software is running. It means that the scanner software has to either be turned off, or given an exclusion rule so it does not monitor the product's files during an update.

The IBM WebSphere Application Server Support team is aware that suggestions to disable antivirus scanning functions are taken very seriously on corporate systems. IBM recommends reviewing other solutions, such as disabling Indexing Services, before resorting to disabling antivirus scanning functions. But, it has been confirmed through numerous reports that if the other solutions do not resolve the "file locking" problems, then disabling the antivirus real-time security scanner function resolved these issues.

IBM is investigating potential improvements to our utilities to avoid situations where antivirus software interferes with product updates. However, these improvements are not currently available. Our investigation of traces produced by tools such as Microsoft Process Monitor has led us to conclude that disabling the real-time scanner, or adding an exclusion rule, is currently the only way to avoid such issues.