Modified Instructions for Complete Restores of Windows Systems with the TSM Client: Bare Metal Restore (BMR), System State Restore, Windows System Object Restore

Resolving The Problem

Preparation:
In order to perform a complete system restore, a complete backup is needed. The backup must include a complete backup of the system drive, and a backup of the system state. You may also need to backup additional drives besides the system drive if they exist on your system. Generally, this should include any drive containing critical user data or application program files. The following points should be considered:

1. A scheduled backup of the default all-local domain will include both the system drive and system state.
2. Care must be taken not to exclude required application files from the system drive backup. The sample options file in the config directory lists suggested include/exclude rules which are known not to interfere with system recovery. The sample options files can be found in the normal installation path. For example, c:\program files\tivoli\tsm\config\dsm.smp.
3. A number of files on the system drive are automatically excluded from backup based on operating system controls. The "query inclexcl" command can be used to view the rules which will affect which files are automatically excluded during backup. Files which are under Windows system file protection can only be backed up as part of the system and boot files component of system state. They are automatically excluded from normal system drive backup processing.
4. System recovery time can be greatly reduced by maintaining a volume-level image of the base Windows installation.

Restore Procedure:
The following restore procedure is required when you are restoring your complete operating system over a running operating system. These procedures are not required if you are restoring with Automated System Recovery (ASR). See note #5 for additional details related to ASR.

Preparation steps:

1. It is strongly recommended that you restore to identical hardware to which the backup was taken from. In virtualized environments there often subtle differences between the original system and the restore target which can cause problems. Here are some common ones to avoid:
1. Different code levels for the hypervisor.
2. Different virtual hardware levels.
3. Different virtual devices assigned to the virtual machine. For example, a different type of virtual disk controller or network adapters, or changes in the number of these devices.
2. In order to restore, you need a base operating system running with the TSM client installed, and connectivity to the TSM server.
3. Windows must be installed in the same directory at which it was installed at the time of backup (c:\Windows for example.) The system drive file system must be formatted in the same file system type that existed at the time of backup. For systems previously upgraded from Windows NT4.0, you may have difficulty installing the base OS in the original directory. See notes below for more information.
4. The system name must be set to match the system name at the time of backup. Unless this is set, the system state component cannot be restored.
5. If you are recovering a domain controller, do not promote the base operating system to a domain controller prior to running the restore.
6. Apply the same Windows service pack to the base operating system which was installed at the time of backup. In addition, on Windows 2003 servers, the VSS hotfixes identified in KB940349 and KB934016 need to be installed following any service pack installations.
7. Partition and format any additional volumes required for recovering the systemstate or additional drives. For example, if you are restoring a domain controller with NTDS files stored on a drive other than C:, you will need to create the additional drives before proceeding with the restore. The new drives need to be of equal or greater capacity, formatted with the same file system type, and mapped to the same drive letter or directory that was used at the time of backup.
8. Ensure that the Windows default administrative shares exist for all of the drives to be restored. Issue the following Windows command to check this:
   > net share
9. For restores from a TSM server, you will need a working network connection. Even with local backupset restores, problems have been encountered resolving UNC names on systems where there are no active network connections, so you should have at least one active network connection before attempting the restore.
10. For Windows 2003 domain controller restores, the option FRSPRIMARYRESTORE can be set to YES prior to performing the systemstate restore. Recovering the first domain controller in a domain is one example where using the option to force an authoritative FRS restore is required. An authoritative SYSVOL restore is not possible when using ASR.
    Note: Performing an authoritative SYSVOL restore is not supported for Windows versions newer than Windows 2003. See APAR IC79919 for additional information.
11. For Windows 7 and Windows 2008 R2 systems, use regedit to confirm that the registry subkey HKEY_LOCAL_MACHINE->COMPONENTS exists prior to restoring the system state. See APAR IC65386 for more details.

Restore steps (see the additional information section for notes on using the Graphical Users Interfaces):

1. Restore the system drive. Ignore the request to reboot at the end of the restore and proceed to the next step! Repeat this restore step for all other drives that exist on your system.
   
   > dsmc restore c:\* -sub=yes -rep=all
   
2. Restore the system state. On Windows 2003, the option frsprimaryrestore yes can be added to the client option file prior to running the following commands if an authoritative sysvol restore for a domain controller is required. If you are restoring a Windows 2008 server running failover cluster services you may need to add the keyword bootable to the restore systemstate command. See below for more information on restoring clustered systems.
   
   > dsmc restore systemstate
   
3. Reboot

The following example shows a complete sequence of commands for restoring a Windows system using environment variables to avoid fixed drive letters and paths.

> dsmc restore %systemdrive%\* -sub=yes -rep=all
> dsmc restore systemstate


Additional Information
1. There are four files under system file protection which cannot be restored due to a Microsoft Windows limitation with replace on reboot. These files are: ntdll.dll, smss.exe, dtcsetup.exe, and ctl3dv2.dll. If you are restoring over a base Windows image which contains down-level versions of these files versus those contained in your backup image, you may experience system problems following the restore including:

• Failure of your system to boot due to a down-level version of ntdll.dll (see Microsoft article KB328035 for additional information.)
• One or more Windows critical updates may not be correctly restored. Following the restore, use Windows Update to reapply the critical updates which are not correctly restored.

2. When is it necessary to use "Directory Services Restore Mode" (DSRM)? The normal complete system recovery procedure for a domain controller does not require DSRM. The use of DSRM is only required when a server is already active as a domain controller, and recovery of NTDS is prevented when the Active directory services are active. When performing a complete system recovery, do not promote your newly installed base OS to a domain controller prior to performing the restore. The system should automatically become a functioning domain controller as a result of the restore. Alternatively, DSRM is required if your server is already operating as a domain controller, and restore of the active directory is needed to resolve an issue such as accidental deletion of active directory objects. In this scenario, booting to DSRM prevents the NTDS services from starting, allowing a TSM restore of system state including NTDS.

3. Following a domain controller recovery, you may see errors in the Windows event viewer for file replication services which indicate that the server is prevented from becoming a domain controller due to problems with the SYSVOL share. In normal cases, this message will eventually be followed by another message indicating that SYSVOL has been shared and is no longer preventing the server from becoming a domain controller. If this does not happen, manual repairs to the SYSVOL structure may be required. In many cases, the SYSVOL directory structure can be repaired, and FRS will replicate in the correct contents of the SYSVOL. Microsoft article KB315457 provides details on how to rebuild the SYSVOL tree including the junction points which are required before a system can be returned to a domain controller.

http://support.microsoft.com/kb/315457

4. In some situations, special steps are required to force Windows to install into a non-default directory when installing your base operating system. For example, a system which is upgraded from Windows 2000 to Windows 2003 will maintain the operating system files in C:\WINNT. When installing a base Windows 2003 operating system for recovery purposes, Windows will force the installation into the C:\WINDOWS directory by default. A successful restore will not be possible unless the base Windows 2003 operating is forced to install into the C:\WINNT directory instead of C:\WINDOWS. Microsoft article KB235478 provides instructions on how to force Windows to install into a non-default directory.

http://support.microsoft.com/kb/235478

5. An additional recovery feature known as Automated System Recovery (ASR) is available with the TSM client. With ASR you will not be required to perform many of the steps described in this document, including the pre-restore of the SFP catalogs. The TSM restores performed during ASR occur while Windows system file protection is not running. See the links at the beginning of this document for links to additional information on ASR.

6. The procedures listed in this article are not recommended for recovering an operating system to dissimilar hardware. However, in situations where there is no alternative, two methods have provided limited success repairing boot problems following restores to dissimilar hardware. The first technique involves retaining a copy of critical hardware specific files after installing the base operating system and copying these files back after performing the restore, but before rebooting the system. See the following Microsoft article for more discussion on this technique:

http://support.microsoft.com/kb/Q249694

The second technique uses the Windows installation CD repair facility to repair the restored operating system which is failing to boot. This procedure is some times referred to as "upgrade in-place" or "repair installation." See the following Microsoft articles for more information on this technique:

http://support.microsoft.com/kb/315341

http://support.microsoft.com/kb/816579

7. Tips for recovering NTDS or SYSVOL components to a new location without restoring the entire systemstate.

In certain circumstances, you may desire to restore NTDS or SYSVOL files for a domain controller to an alternate location, and then use Microsoft utilities to recover from these restored files. Tivoli does not provide support for these recovery techniques, but provides these commands for users who have a requirement for recovery of NTDS and/or SYSVOL without a complete systemstate recovery, and have knowledge of the procedures necessary for recovering from the files which have been restored to an alternate location.

WARNING: these restore commands may create junctions in the restore target directory that point back to critical directories on the running system. Do not delete the temporary restore destination using the Windows explorer interface since it is known to also delete the contents at the target of junctions. Instead, we recommend using the Windows rmdir command to remove the temporary directory.

rest "{SERV1\SystemState\NULL\System State\SystemState}\\serv1\e$|\ntds\*" e:\ntdsrest\ -sub=y

Your pathnames for where NTDS and SYSVOL may vary requiring adjustment to these commands. If you are having trouble locating the correct path, you can search for the files with a command like the following:

dsmc q b "{SERV1\SystemState\NULL\System State\SystemState}\ntds.dit" -sub=y

8. A systemstate restore may fail with the message ANS1056E Share/network path cannot be resolved. This failure may be caused by the absence of the default administrative shares for drives which need to be accessed during the systemstate restore. You can determine if you have this problem by issuing the Windows command net share to ensure the shares are listed for all drives. In most cases rebooting will restore the missing shares. Alternatively, stopping and starting the Windows server service may also restore these shares.

9. A systemstate restore may fail with the error message ANS1949E Microsoft volume shadow copy snapshot initialization failed. See the following technote for a possible solution to this problem:
http://www.ibm.com/support/docview.wss?&uid=swg21385864

10. Restores of Windows 2008 systems running failover cluster services require a modification to the restore commands described above. Without this modification, the restore systemstate command may fail with the
error:
ANS5211E The cluster service is offline. The cluster service must be online to perform an authoritative cluster database restore operation.

This error is encountered because the clusters services are not running as part of a base Windows operating system. In most recovery cases, other nodes in the cluster are still running, so there is no need to restore the cluster database since it will automatically copy back from the other nodes. In rarer cases, an authoritative restore of the cluster database is needed when other cluster nodes are not present or the cluster database has been corrupted on all nodes.

The non-authoritative restore sequence uses the bootable subkey to avoid ANS5211E:

dsmc restore c:\* -sub=yes -rep=all
dsmc restore systemstate bootable
reboot

Authoritative restore sequence :

dsmc restore c:\* -sub=yes -rep=all
dsmc restore systemstate bootable
reboot
dsmc restore systemstate clusterdb

11. The restore procedures described in this document can also be performed using the graphical user interface with the exception of the steps for performing a pre-restore of the Windows system file protection catalogs.

To restore the system drive:
a) Click "Restore"
b) Expand "File Level"
c) Select the entire drive representing the system drive, for example, \\tsmbldx861\c$ (C:)




d) Click the "Options" button, and select the options for files which already exist "Replace" and "Replace files even if read-only/locked", then click OK



e) Click "Restore", and select "Original location", then click "Restore" again



f) Ignore the request to reboot at the completion of this restore and proceed to the systemstate restore.

To restore the system state:
a) Click "Restore"
b) Select the object "SystemState", and click "Restore"



12. The following are links to additional information on TSM topics useful for Windows system recovery.

Best practices instructions for a new ASR system recovery approach:

• Best Practices for Recovering Windows Server 2008, Windows Server 2008 R2, Windows 7, and Windows Vista
• Best Practices for Recovering Windows Server 2012 and Windows 8

White paper titled "Tivoli Storage Manager Recovery Techniques Using Windows Preinstallation Environment (Windows PE)"
http://www.ibm.com/support/docview.wss?uid=swg27005028

Technote explaining restore location of the Windows event log files for Windows 2003
http://www.ibm.com/support/docview.wss?uid=swg21223228