1.     Meta callback failures
2.     Defect in the Base ORB marshaling code

CORBA.MARSHAL due to meta callback failures

NOTE:  Callbacks can be initiated on either a server or client, but most of the time the server is making the callback to the client.  The following discussion assumes the server is making the callback.

Callbacks can be necessary when either a client or server ORB reads a request or reply message and attempts to demarshal the data into meaningful data structures.  A callback to the remote ORB occurs when the version of a class being demarshaled is not exactly the same on both the client and server.  Typically, this class version mismatch occurs when the client and server have different JDK versions because many times request/reply messages include basic JDK classes.  Mismatches can also occur with business application classes.  The callback process rectifies the differences in the structure of a particular class(es) when unmarshaling data.

Sample Server Stack Trace snipit:

org.omg.CORBA.MARSHAL: Unable to read value from underlying bridge  
…
at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:1121)
at com.ibm.CORBA.iiop.ClientDelegate.createRequest(ClientDelegate.java:1253)
at com.ibm.rmi.corba.ClientDelegate.request(ClientDelegate.java:1841)
at com.ibm.CORBA.iiop.ClientDelegate.request(ClientDelegate.java:1209)
at org.omg.CORBA.portable.ObjectImpl._request(ObjectImpl.java:460)
at com.ibm.org.omg.SendingContext._CodeBaseStub.meta(_CodeBaseStub.java:115)   <<<  callback methods
at com.ibm.rmi.iiop.FVDCodeBaseDelegate.meta(FVDCodeBaseDelegate.java:145)        <<<  callback methods
at com.ibm.rmi.io.IIOPInputStream.inputObjectFVD(IIOPInputStream.java:1407)
at com.ibm.rmi.io.IIOPInputStream.readSerializable(IIOPInputStream.java:1032)
at com.ibm.rmi.io.IIOPInputStream.simpleReadObjectInternal(IIOPInputStream.java:317)
at com.ibm.rmi.io.IIOPInputStream.simpleReadObject(IIOPInputStream.java:347)
at com.ibm.rmi.io.ValueHandlerImpl.readValue(ValueHandlerImpl.java:200)
at com.ibm.rmi.iiop.CDRInputStream.read_value(CDRInputStream.java:1961)
at com.ibm.engine._EJSRemoteStatelessConnEngine_da67e2e1_Tie.getEngineInfo(_EJSRemoteStatelessConnEngine_da67e2e1_Tie.java:157)
at com.ibm.engine._EJSRemoteStatelessConnEngine_da67e2e1_Tie._invoke(_EJSRemoteStatelessConnEngine_da67e2e1_Tie.java:89)
at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:610)
…

Sample Client Side Stack:

Trace from server: 1198777258 at host myhost.com
>> org.omg.CORBA.MARSHAL: Unable to read value from underlying
bridge : initial and forwarded IOR inaccessible  vmcid: IBM
minor code: 89A  completed: No
at com.ibm.rmi.iiop.CDRInputStream.read_value(CDRInputStream.java:1968)
at com.ibm.tws.conn.engine._EJSRemoteStatelessConnEngine_da67e2e1_Tie.getEngineInfo(_EJSRemoteStatelessConnEngine_da67e2e1_Tie.java:157)
at com.ibm.tws.conn.engine._EJSRemoteStatelessConnEngine_da67e2e1_Tie._invoke(_EJSRemoteStatelessConnEngine_da67e2e1_Tie.java:89)
at com.ibm.CORBA.iiop.ServerDelegate.dispatchInvokeHandler(ServerDelegate.java:610)
at com.ibm.CORBA.iiop.ServerDelegate.dispatch(ServerDelegate.java:463)
at com.ibm.rmi.iiop.ORB.process(ORB.java:439)
at com.ibm.CORBA.iiop.ORB.process(ORB.java:1737)
at com.ibm.rmi.iiop.Connection.respondTo(Connection.java:2350)
at com.ibm.rmi.iiop.Connection.doWork(Connection.java:2195)
at com.ibm.rmi.iiop.WorkUnitImpl.doWork(WorkUnitImpl.java:65)
at com.ibm.ejs.oa.pool.PooledThread.run(ThreadPool.java:95)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled Code))
<<  END server: 1198777258 at host myhost.com

ORB trace showing meta/FVD trace:

The presence of inputObjectFVD() trace indicates the ORB is attempting to reconcile a mismatched class.  Just before that trace line will be the readSerializable trace line indicating the mismatched class.  Since the ORB caches the results of meta callbacks, it's possible to see this trace with no subsequent meta callback (the assumption being that the cache held info on the class in question).  This trace will be seen soon after reading either an incoming request or reply message.

[5/31/18 1:45:30:344 CDT] 00000202 ORBRas        3 com.ibm.rmi.io.IIOPInputStream readSerializable:1531 MessageListenerThreadPool : 46 clz = class com.cust.class  << mismatched class
[5/31/18 1:45:30:345 CDT] 00000202 ORBRas        3com.ibm.rmi.io.IIOPInputStream inputObjectFVD ORB.thread.pool : 1 Entry
[5/31/17 1:45:30:345 CDT] 00000202 ORBRas        3com.ibm.rmi.io.IIOPInputStream inputObjectFVD:2120 ORB.thread.pool : 1

Possible causes:

Root Cause 1:  Firewall is blocking the meta callback

Symptoms:       On the server side, the callback fails with a "java.net.ConnectException: Connection refused" but sometimes the failure can also be a connect timeout.  In either case, the firewall has blocked the server's attempt to create a new connection to the client over which the meta callback is to be made.

Solution:        Set the ORB property com.ibm.CORBA.enableClientCallbacks=true on the client side.  This causes the server to send the meta callback request to the client using the already-existing connection from that client.   

Root Cause 2:   Client is configured with a 127.0.0.1 or "localhost" address.

Symptoms:    Usually this problem occurs with thin clients.  If the client's ORB property com.ibm.CORBA.LocalHost is set to either "127.0.0.1" or "localhost" (which resolves to 127.0.0.1), AND if the client and server are on separate machines, then a "java.net.ConnectException: Connection refused"  occurs on the server.  This is because the server is trying to connect to the client via the loopback IP which will fail.  The server needs the "real" IP or hostname of the client.

Solution:        Set the client's com.ibm.CORBA.LocalHost property to a "real" hostname or IP address (i.e. NOT 127.0.0.1 or "localhost").

Root Cause 3:   Since a meta callback is like any other ORB request, it can fail for all sorts of other reasons that a typical request can (NO_RESPONSE, network issues, etc)

Solution:        Matching client/server ORB trace is usually needed to further debug the root cause.

CORBA.MARSHAL (non-meta callback cases)

If meta callback failures have been ruled out, then matching client/server ORB trace will be needed.  In some scenarios, a small testcase program will be needed to further debug the issue.