Troubleshooting
Problem
This technote outlines the troubleshooting steps necessary to recover from the IBM® Rational® ClearCase® error []vobsvr_contact call failed: RPC: Program unavailable[].
Symptom
The following error appears as a result of cleartool or multitool commands run in a specific VOB:
vobsvr_contact call failed: RPC: Program unavailable
Example:
CLEARTOOL DESCRIBE
%>cleartool describe vob:\DemoVob
versioned object base "\DemoVob"
created 30-Oct-03.16:32:27 by user.group@host
VOB family feature level: 3
VOB storage host:pathname "host:C:\Vobstore\DemoVob.vbs"
VOB storage global pathname "\\host\Vobstore\DemoVob.vbs"
database schema version: 53
vobsvr_contact call failed: RPC: Program unavailable
Attributes:
FeatureLevel = 3
MULTITOOL CHMASTER
%> multitool chmaster -all original replica1
Chmaster -all will transfer mastership of all objects
currently mastered by original to replica1.
Do you want to proceed? [no] yes
vobsvr_contact call failed: RPC: Program unavailable
Other symptoms:
- Unable to unmount the VOB (same error)
- Unable to synchronize the VOB replica (same error)
- The Windows® Event Viewer logs will display an entry similar to the following:
vob_server.exe(1236): Error: Already running as pid 65535
There are other commands that fail in a similar manner.
Cause
The problem is there are multiple vob_server processes running on the VOB server for a single VOB object.
TROUBLESHOOTING:
1. Run rgy_check to see if there are any VOB entries that have been registered more than once to the same VOB storage path.
Example output from rgy_check:
C:\>rgy_check -vobs -storage
rgy_check: Error: Two VOB objects have the same pathname on a server.
First object:
-hostname = "host1"
-local_path = "e:\ClearCase_Storage\VOBs\PVOB.vbs"
-vob_replica = "eb5a90e9.60a74a34.9dc1.65:94:f9:8e:11:9a"
-vob_family = "58336078.7b114616.91be.17:4d:13:19:8b:92"
-attributes = "sumvob"
This object:
-hostname = "host1"
-local_path = "e:\ClearCase_Storage\VOBs\PVOB.vbs"
-vob_replica = "86412787.6f7841c4.ad7a.f0:c1:85:01:e2:fb"
-vob_family = "989f178c.1dad46c1.b822.12:97:6a:85:e9:9f"
-attributes = "sumvob"
rgy_check: Error: This VOB object is stranded (it has no tags in any region):
-hostname = "host1"
-local_path = "e:\ClearCase_Storage\VOBs\PVOB.vbs"
-vob_replica = "eb5a90e9.60a74a34.9dc1.65:94:f9:8e:11:9a"
-vob_family = "58336078.7b114616.91be.17:4d:13:19:8b:92"
-attributes = "sumvob"
- Review the ClearCase Reference Guide on the topic of rgy_check (cleartool man rgy_check) for more information.
2. Run albd_list and look for multiple vob_server entries (PIDs) pointing to the same VOB:
- %>albd_list
PID 1096:
vob_server, udp socket 1264: version 3; BUSY
Storage path C:\Vobstore\DemoVob.vbs
UUID 81ac617d.41124a89.bc71.65:05:b2:94:27:f3
PID 1660:
vob_server, udp socket 1557: version 3; BUSY
Storage path C:\Vobstore\DemoVob.vbs
UUID 81ac617d.41124a89.bc71.65:05:b2:94:27:f3
PID 1236:
vob_server, udp socket 1656: version 3; BUSY
Storage path C:\Vobstore\DemoVob.vbs
UUID 81ac617d.41124a89.bc71.65:05:b2:94:27:f3
Review technote 1148639 for more information about the albd_list utility.
Diagnosing The Problem
If the VOB has recently been moved and the above errors are accompanied by errors reported in the logs, refer to technote 1170877 for information about another issue that may be the cause.
Error: The vob_object registry for "/vobstorage/vob.vbs" specifies incorrect hostname for server; expected hostname: host1 actual hostname: myhost
Resolving The Problem
1. Fix all rgy_check issues based on the problem reported. The above example has two different problems reported; however, they are related (in this case) to the same VOB and thus related to the same problem.
- rgy_check: Error: Two VOB objects have the same pathname on a server.
- rgy_check: Error: This VOB object is stranded (it has no tags in any region)
- SOLUTION: Unregister all entries and re-register the VOB.
- Example:
%>cleartool unregister -vob -uuid eb5a90e9.60a74a34.9dc1.65:94:f9:8e:11:9a
%>cleartool unregister -vob -uuid 86412787.6f7841c4.ad7a.f0:c1:85:01:e2:fb
Note: Running rgy_check at this point should yield the following error:
%>rgy_check -vobs -storage
rgy_check: Error: This VOB tag has no VOB object with the same UUID:
-tag = "\PVOB"
-title = ""
-global_path = "\\host1\ccstg_e\VOBs\PVOB.vbs"
-hostname = "host1"
-mount_access = "private"
-mount_options = ""
-region = "windows"
-vob_replica = "86412787.6f7841c4.ad7a.f0:c1:85:01:e2:fb"
Using the information from the rgy_check output, register the VOB.
Note: If the VOB is a UCM PVOB, be sure to use the -ucmproject option.
%>cleartool register -vob -ucmproject \\host1\ccstg_e\VOBs\PVOB.vbs
Running rgy_check should now report No registry errors/inconsistencies detected.
%>rgy_check -vobs -storage
No registry errors/inconsistencies detected.
2. The vob_server processes need to be identified and then stopped.
- On Windows use the Task Manager and end each of the vob_server processes by PID.
- Example:
- Open Task Manager (Click Start > Run and type taskmgr).
- Find the vob_server process with the matching PIDs from the albd_list output, rick click the process and click End Process.
- On UNIX run: ps -ef | grep vob_server then issue the kill command
Once the vob_server processes are stopped, mount the VOB and continue with the operation previously started before the error occurred.
Was this topic helpful?
Document Information
Modified date:
16 June 2018
UID
swg21135500