Memory leak seen when processing Scoring requests. The java heap will fill up and the server will crash if not caught in time.
java heap continually climbing and not being cleaned up by garbage collection.
The configuration consisted of a WebLogic cluster with a load balancer and two nodes. The load balancer would send requests to the nodes of the WebLogic cluster to see if they were active. The load balancer has various settings that can be configured to see which nodes of the cluster are still active. It can be configured to send a request to each node of the cluster - if the node does not respond, it knows the node is down and will not route requests to that node until it comes up and starts responding again.
In this instance the load balancer was configured to send a "get scoring service wsdl" request. It is not something that would be recommended to do, because its a heavy request. The response is fairly large, but it should be perfectly valid. Note, SPSS code does not process this request. It's something that the application server code does.
Something in this environment (load balancer - linux OS - Oracle WebLogic) was causing a TCP/IP socket to be left open. As a result, because the socket was left open, WebLogic was still hanging onto the request response.
Collaboration and Deployment Services has a scoring service PING request that can be used to see if the scoring service is still running. It's a small, light weight request that oddly enough WILL get routed through our code. Its well suited for this purpose.
Linux SuSE 11
WebLogic application server - clustered environment
Oracle 11g data base
Collaboration and Deployment Services 5 fix pack 1
Modeler 15 Server fix pack 1
Decision Management 7 fix pack 1
Diagnosing the problem
Using netstat to display all open ports and taking a java heap dump at the same time will show all sockets still left open
Resolving the problem
Change the node status request from the get wsdl request to the PING request for the scoring service