Common error message returned by VMware vStorage API for Data Protection (VDAP) and reported in the Tivoli Storage Manager for Virtual Environments: Data Protection for VMware data mover

Resolving The Problem

VMware also has a “Best Practices” document that describes the use of the advanced transport mechanisms (SAN, HotAdd and NBD/NBDSSL). A link to that best practices document is supplied in each message explanation where it might be helpful to review the VMware best practices.

Message: VixDiskLib_Open => "You do not have access rights to this file"
Tivoli Storage Manager operations: Backup VM, Restore VM

Transport: ALL
Explanation and actions:
The data mover cannot resolve the fully qualified domain name (FQDN) of the ESXi host. Ensure that the data mover is configured to use the DNS or other services. Ping the ESXi, by name, from the data move and confirm it resolves the IP address.
1) Verify that the vCenter is licensed to use the "Storage APIs" feature.
2) To backup a target virtual machine, you must specify which VMware userid to use with the Tivoli Storage Manager data mover. This is done with the 'dsmc set password -type=VM' command. For more information, see https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.0/client/r_cmd_setpassword.html.

For advanced transport best practices, see https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vddk.pg.doc%2FvddkBkupVadp.9.5.html.

Transport: SAN
Explanation and actions:
1) The data mover must be installed on a physical system.
2) The Fibre Channel SAN is not accessible by the data mover node. Ensure that the LUN is mapped to the data mover.
3) The iSCSI SAN target is not connected to the data mover node. Ensure that the iSCSI target is connected to the data mover.
4) The SAN Policy must be set to OnLineAll using the Windows program called diskpart. For more information, see http://kb.vmware.com/kb/2000767.

For advanced transport best practices, see https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vddk.pg.doc%2FvddkBkupVadp.9.5.html.

Transport: SAN
Explanation and actions:
The guest to backup has an UUID that is not unique in the VMware environment.
The error will be seen only when using SAN transport and with client 7.1.2.0 to 7.1.3.1 because the guest UUID uniqueness check is enforced by VMware starting with VDDK 6.0.0 :

SAN mode VDDK 6.0 searches for virtual machines by BIOS UUID.
When using SAN transport, VDDK 6.0.0 tries to find requested virtual machines by looking up their BIOS UUID, instead of by the MoRef provided, as in previous VDDK releases. Because a BIOS UUID is not guaranteed unique, unlike a MoRef, this new behavior may cause problems after cloning, out of place VM restore, and so forth. The problem happens with both SAN backup and restore when two VMs managed by the same vCenter Server have identical BIOS UUID, whether the VMs are powered on or off. The error occurs whenVixDiskLib_Open(Ex) fetches the virtual disk's block map. A fix has been identified and will be available in a future release.

This is corrected with VDDK 6.0.1 which is included in Tivoli Storage Manager Client 7.1.3.2 and higher.

Transport: HOTADD
Explanation and actions:
The data mover must be installed on a virtual system in the same vCenter as the target virtual machine.

• The data mover does not have access to the target virtual machines datastore. Endure the data mover has access to the target virtual machines datastore.
• The SAN Policy must be set to OnLineAll using the Windows program called diskpart. For more information, see http://kb.vmware.com/kb/2000767.

For advanced transport best practices, see https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vddk.pg.doc%2FvddkBkupVadp.9.5.html.


Transport: NBD, NDBSSL
Explanation/Actions:
A network path is not available from the data mover to the ESXi system that is hosting the target virtual machine.

• ESXi host is not responding on port TCP 902, which is the port used by the NBD transport. Use Telnet to connect to the ESXi server to see if it accepts the connection (telnet <esxihost> 902). For more information, see http://kb.vmware.com/kb/1003487.

For advanced transport best practices, see https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vddk.pg.doc%2FvddkBkupVadp.9.5.html.


Message: VixDiskLib_Open => "Cannot use hotadd transport to mount virtual machines with non-SCSI disks"
Tivoli Storage Manager operations: Backup VM, Restore VM

Transport: SAN, NBD, NDBSSL
Explanation/Actions:

• It is unknown if this message occurs when you use SAN, NBD, or NBDSSL.

Transport: HOTADD
Explanation/Actions:

• The VDDK does not support the HotAdd transport with SATA virtual disks. For information, see http://www.vmware.com/support/developer/vddk/vddk-550-releasenotes.html.

Message: VixDiskLib_ConnectEx => "One of the parameters supplied is invalid"
Tivoli Storage Manager operations: Backup VM

Transport: SAN, NBD, NDBSSL
Explanation/Actions:

• It is unknown if this message occurs when you use SAN, NBD, or NBDSSL.

Transport: HOTADD
Explanation/Actions:

• HotAdd is the selected transport but the data mover is a physical system. HotAdd can only be used on a virtual machine.

Message: VixDiskLib_ConnectEx => "No transport modes available to access disks"
Tivoli Storage Manager operations: Backup VM, Restore VM also depends on Virtual Disk Development Kit (VDDK) version

Transport: SAN, NBD, NDBSSL
Explanation/Actions:

• It is unknown if this message occurs when you use SAN, NBD, or NBDSSL.

Transport: HOTADD
Explanation/Actions:

• HOTADD is the selected transport but the data mover is a physical system. HotAdd can only be used on a virtual machine.

Message: VixDiskLib_Write => "One of the parameters supplied is invalid"
Tivoli Storage Manager operations: Restore VM

Transport: SAN

Explanation/Actions:

• LUN or iSCSI target is marked "Read-only: Yes". Clear the attribute using "diskpart" (attribute disk clear readonly). For more information, see http://kb.vmware.com/kb/2000767.

Transport: HOTADD, NBD, NDBSSL
Explanation/Actions:

• It is unknown if this message occurs when you use HOTADD, NBD, or NBDSSL.

Message: VixDiskLib_Open => "The host is not licensed for this feature"
Tivoli Storage Manager operations: Backup VM, Restore VM

Transport: SAN, HOTADD, NBD, NDBSSL
Explanation/Actions:

• The Tivoli Storage Manager User Role for backups and restores is not applied at the vCenter level. With older vCenter versions the role could be applied at the ESXi host level, but this changed in vCenter 5.1. For more information, see http://kb.vmware.com/kb/2063054 and http://www-01.ibm.com/support/docview.wss?uid=swg27047438.

Message: VixDiskLib_Open => "Cannot hot-add the proxy's own disks to itself "
Tivoli Storage Manager operations: Backup VM

Transport: SAN, NBD, NDBSSL
Explanation/Actions:

• It is unknown if this message occurs when you use SAN, NBD, or NBDSSL.

Transport: HOTADD
Explanation/Actions:

• VMware's Virtual Disk Development Kit (VDDK) no longer supports HOTADDing the data mover system to itself for backup, NBD/NBDSSL must be used. See VMware Release Notes https://www.vmware.com/support/developer/vddk/vddk-550-releasenotes.html.

Message: Snapshot Clean-up => "Device or resource busy" or "The file is already in use"
Tivoli Storage Manager operations: Backup VM

Transport: SAN, NBD, NDBSSL
Explanation/Actions: See VMware's Virtual Disk Development Kit (VDDK) 5.1.1 Release Notes https://www.vmware.com/support/developer/vddk/vddk-511-releasenotes.html.

• Snapshot delta files are not consolidated after backup this is a VMware issue that is fixed in VDDK 5.1.2, 5.5 and in 5.0 Update 3. See the two VMware KBs http://kb.vmware.com/kb/2040846 and http://kb.vmware.com/kb/2050453.

Transport: HOTADD
Explanation/Actions:

• It is unknown if this message occurs when you use HOTADD.

Message: VixDiskLib_WriteMetadata => "The operation is not supported for the specified parameters"
Tivoli Storage Manager operations: Restore VM

Transport: SAN, HOTADD
Explanation/Actions:

• Metadata writes are not supported for HOTADD and SAN transports in all VDDK versions. For more information, see https://www.vmware.com/support/developer/vddk/vddk-550-releasenotes.html.

Transport: NBD/NDBSSL
Explanation/Actions:

• It is unknown if this message occurs when you use NBD or NBDSSL.

Message: VixMntApi_GetVolumeInfo => The problem here is that no data is returned and we get no error message from the VDDK SDK.
Tivoli Storage Manager operations: Backup VM, Restore VM

Transport: SAN, HOTADD, NBD, NDBSSL
Explanation/Actions:

• The boot disk must be scsi0:0 for use with VDDK mount API. For more information, see https://www.vmware.com/support/developer/vddk/vddk-550-releasenotes.html.

Message: "Volume/NTFS filesystem is corrupt (87)"
Tivoli Storage Manager operations: Backup VM

Transport: SAN, NBD, NDBSSL
Explanation/Actions:

• It is unknown if this message occurs when you use SAN, NBD, or NBDSSL.

Transport: HOTADD
Explanation/Actions:

• An NTFS virtual disk from a Window 2012 Server might produce Error 87 when the data mover mounts the disk in read-only mode that is on an earlier version of Windows Server. Use a newer version of Windows Server for the data mover. For more information, see http://www.vmware.com/support/developer/vddk/vddk-511-releasenotes.html.

Message: VixDiskLib_Write => "Unknown error" and the return code is -1 also a trace will show "The media is write protected"
Tivoli Storage Manager operations: Restore VM

Transport: SAN, HOTADD
Explanation/Actions

• The SAN Policy must be set to OnLineAll using the Windows program "diskpart". For more information, see VMware KB http://kb.vmware.com/kb/2000767.
• The LUN could be marked as offline or read only use diskpart tool as recommended by the VMware advance transport best practices document.

For advanced transport best practices, see https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vddk.pg.doc%2FvddkBkupVadp.9.5.html.


Transport: NBD, NDBSSL
Explanation/Actions:

• It is unknown if this message occurs when you use NBD or NBDSSL.

Message: VixMntApi VixDiskLib_Read => "Unknown error" and the return code is 1 also a trace will show "The media is write protected"
Tivoli Storage Manager operations: Backup VM

Transport: SAN, HOTADD
Explanation/Actions:

• The SAN Policy must be set to OnLineAll using the Windows program "diskpart". For more information, see http://kb.vmware.com/kb/2000767.
• The LUN could be marked as offline or read only use the Windows program "diskpart". For more information, see http://kb.vmware.com/kb/2000767.

For advanced transport best practices, see https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vddk.pg.doc%2FvddkBkupVadp.9.5.html.

Transport: NBD, NDBSSL
Explanation/Actions:

• It is unknown if this message occurs when you use NBD or NBDSSL.

Message: visdkCreateVmSnapshotMoRef => “Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine."
Tivoli Storage Manager operations: Backup VM

It is important to understand that the Tivoli Storage Manager data mover only makes a request for the vCenter to take a snapshot via the VMware Infrastructure (VI). VMware provides the following KB that discusses the snapshot process. See http://kb.vmware.com/kb/1015180.

Transport: ALL
Explanation/Actions:

• In all cases a manual snapshot should be attempted using VMware's vSphere Client. For information on how Tivoli Storage Manager requests virtual machine snapshots see http://www-01.ibm.com/support/docview.wss?uid=swg21585387.
• Often the main issue is that the Volume Shadow Copy Service (VSS) is unable to flush application data to disk in the virtual machine because the Input/Output was too high. The I/O must be reduced. VMware provides the VSS to the virtual machine via VMware Tools. You might need to upgrade to the latest version or remove VMware Tools from the virtual machine and then reinstall them. This is a complex problem; for more information, see http://kb.vmware.com/kb/1018194 and http://kb.vmware.com/kb/1007696.
• This is a common problem with Windows 2008 virtual machines and several VMware KB's discuss this. For information, see http://kb.vmware.com/kb/1031298, http://kb.vmware.com/kb/2034002 and http://kb.vmware.com/kb/2006849.

Transport: SAN
Explanation/Actions:

• This VMware KB covers quiesced snapshots on virtual machines using Microsoft iSCSI LUNs. http://kb.vmware.com/kb/1009073.

Message: visdkCreateVmSnapshotMoRef => “An error occurred while quiescing the virtual machine. See the virtual machine's event log for details."
Tivoli Storage Manager operations: Backup VM

Transport: SAN, HOTADD, NBD, NDBSSL
Explanation/Actions:

• The Volume Shadow Copy Service (VSS) was unable to flush application data to disk in the virtual machine because the Input/Output was too high. The I/O must be reduced. You might need to remove VMware Tools from the virtual machine and then reinstall them; or, quiescing must be disabled. This is a complex problem; for a detailed explanation, see http://kb.vmware.com/kb/1018194.

Message: visdkCreateVmSnapshotMoRef => “File is larger than the maximum size supported by datastore."
Tivoli Storage Manager operations: Backup VM

Transport: SAN, HOTADD, NBD, NDBSSL
Explanation/Actions:

• This can occur when the datastore cannot contain the snapshot files. For information, see http://kb.vmware.com/kb/1012384.

Message: VixDiskLib_Read => "One of the parameters supplied is invalid"
Tivoli Storage Manager operations: Backup VM

Transport: SAN
Explanation/Actions:

• This error can occurs in the Fibre Channel SAN transport when there is no available paths to the LUN, this could be a temporary SAN I/O failure.
• The disk lease time might have expired. To increase it, see http://kb.vmware.com/kb/1006711.
• The vStorage for Array Integration or VAAI is not enabled or the storage hardware does not fully support VAAI. For information, see http://kb.vmware.com/kb/1021976.
• Use NBD/NBDSSL until the Fibre Channel SAN issue can be corrected.

Transport: HOTADD, NBD, NDBSSL
Explanation/Actions:

• It is unknown if this message occurs when you use HOTADD, NBD, or NBDSSL.

Message: visdkWaitForTask => “A specified parameter was not correct.”
Tivoli Storage Manager operations: Restore VM

Transport: SAN, HOTADD, NBD, NDBSSL
Explanation/Actions:

• In some cases restoring a virtual machine directly to an ESXi host can cause problems if the virtual machine has attributes that are vCenter-specific. The restore process attempts to remove these attributes, but it is a complex process that is not always successful. A VMware KB article might help to determine which attribute is causing the problem (see http://kb.vmware.com/kb/2012122). The vmbackup_save_local and vmrestore_ovf_override test flags can be used to work-around this issue. For information, see http://www.ibm.com/support/docview.wss?uid=swg21636933.

Message: visdkWaitForTask => “A general system error occurred: Protocol error from VMX.”
Tivoli Storage Manager operations: Backup VM

Transport: SAN, HOTADD, NBD, NDBSSL
Explanation/Actions:

• Application consistent quiescing is only supported for virtual machine that use SCSI disks and the target virtual machine must have enough free SCSI slots as the number of disks that are contained in the snapshot. Application consistent quiescing is not supported for virtual machines with IDE or SATA disks. For information see, http://www.ibm.com/support/docview.wss?uid=swg21459852 and http://kb.vmware.com/kb/1037071.

Message: visdkWaitForTask => “Virtual machine is configured to use a device that prevents the snapshot operation: Device '' is a SCSI controller engaged in bus-sharing”
Tivoli Storage Manager operations: Backup VM

Transport: SAN, HOTADD, NBD, NDBSSL
Explanation/Actions:

• Backup cannot be performed on target virtual machines that are configured with bus-sharing controllers. For more information see, http://kb.vmware.com/kb/1006392 and http://www.ibm.com/support/docview.wss?uid=swg21600617.

Message: visdkCreateVM => “The operation is not supported on the object.”
Tivoli Storage Manager operations: Restore VM

Transport: SAN, HOTADD, NBD, NDBSSL
Explanation/Actions:

• In some cases, restoring a virtual machine might be difficult because of settings that are placed in the virtual machine's extraConfig attribute. The restore process attempts to remove these settings, but it is a complex process that is not always successful. A VMware KB article (http://kb.vmware.com/kb/2012122) might help to determine which attribute is causing the problem. The vmbackup_save_local and vmrestore_ovf_override test flags can be used to work around this issue. For more information, see http://www.ibm.com/support/docview.wss?uid=swg1IC87154 and http://www.ibm.com/support/docview.wss?uid=swg21636933.

Message: VixDiskLib_Read => “The server refused connection”
Tivoli Storage Manager operations: Backup VM

Transport: NBD, NDBSSL
Explanation/Actions:

• This message occurs intermittently when you are using older (before Tivoli Storage Manager 7.1) Tivoli Storage Manager clients when the virtual machine has large vmdk files and there is a firewall timeout setting for the communications between the ESXi hosts and the data mover host. The problem occurs because at the start of the backup session, connections are opened for all VMDKs. If one of the VMDK files is large or simply takes too long to be processed, the firewall can cut the idle connections of the remaining VMDKs in the sequential processing list. When the next VMDK is to be processed, the connection is refused and the backup fails. Upgrading to Tivoli Storage Manager 7.1, or newer, or using the SAN or HOTADD transport should fix this problem.

Message: VixDiskLib_Read => “SOAP 1.1 fault: SOAP-ENV:Client[no subcode] "No Data" Detail: get host by name failed in tcp_connect()”
Tivoli Storage Manager operations: Backup VM, Restore VM

Transport: SAN, HOTADD, NBD, NDBSSL
Explanation/Actions:

• This happens intermittently when the communication between the data mover and the vCenter fails, either because the vCenter is too busy to respond in time, or there is a firewall or “intelligent” network appliance that blocks the traffic.