IBM Support

Intermittent "CCL-BIT-0005 A socket reported a communication error" caused by a lack of free ephemeral TCP/IP ports

Troubleshooting


Problem

Controller system works OK for many weeks/months. At some seemingly-random point in time, errors appear.

Symptom

A range of errors may appear, but here are a selection of some that one real-life customer saw:

Screen:

    Message:
    RSV-SRV-0063 An error occurred while executing the 'asynchRunSpecification_Request' command.
    CCL-BIT-0005 A socket reported a communication error.
    CAM_Connect=0xfffffff4 <errorDetail><errorCode>-12</errorCode><errorMessage>CAM-CRP-0026 The underlying socket: '192.168.71.9:9300' returned an error. </errorMessage><errorStack><errorCode>10048</errorCode><errorMessage>Could not connect the socket, errno: 0x2740(10048)
    </errorMessage></errorStack></errorDetail>

Cogserver.log:
192.168.71.9:9300 2372 2011-11-09 15:24:07.012 +0 630EAB2BBEAF70DE40EE79152D8A7CF0216024E0 hy2lhGwvq8yMMjlyw4sl8qC4swMdMMwqd9MGd8Ml 1 http-9300-5 DISP 4707 1 Audit.Other.dispatcher.DISP.pogo pogo com.cognos.pogo.impl.PogoSOAPMessageResponder Failure <messages><message><messageString>DPR-DPR-1035 Dispatcher detected an error.</messageString></message></messages> No response was generated.
192.168.71.9:9300 7104 2011-11-09 15:24:07.184 +0 630EAB2BBEAF70DE40EE79152D8A7CF0216024E0 sljMj9vG8dCCjjv2Chlv9wGwwdhs8yy2yylh4qwh sljMj9vG8dCCjjv2Chlv9wGwwdhs8yy2yylh4qwh 9424 RSVP 4803 1 Audit.RTUsage.RSVP Response Report ReportService Failure RSV-SRV-0063 An error occurred while executing the 'asynchRunSpecification_Request' command. CCL-BIT-0005 A socket reported a communication error. CAM_Connect=0xfffffff4 <errorDetail><errorCode>-12</errorCode><errorMessage>CAM-CRP-0026 The underlying socket: '192.168.71.9:9300' returned an error.</errorMessage><errorStack><errorCode>10048</errorCode><errorMessage>Could not connect the socket, errno: 0x2740(10048)</errorMessage></errorStack></errorDetail>
192.168.71.9:9300 7104 2011-11-09 15:24:07.199 +0 630EAB2BBEAF70DE40EE79152D8A7CF0216024E0 sljMj9vG8dCCjjv2Chlv9wGwwdhs8yy2yylh4qwh sljMj9vG8dCCjjv2Chlv9wGwwdhs8yy2yylh4qwh 8180 RSVP 4803 1 Audit.RTUsage.RSVP Response Report ReportService Failure CCL-BIT-0005 A socket reported a communication error.CAM_Connect=0xfffffff4 <errorDetail><errorCode>-12</errorCode><errorMessage>CAM-CRP-0026 The underlying socket: '192.168.71.9:9300' returned an error.</errorMessage><errorStack><errorCode>10048</errorCode><errorMessage>Could not connect the socket, errno: 0x2740(10048)</errorMessage></errorStack></errorDetail>
192.168.71.9:9300 7104 2011-11-09 15:25:51.247 +0 61044C36AA6BF53E7DBD1001B232A6FF3E003BF4 hq8Mqld84Cjlwhh8sdwMhMl222C84jqh2hhGsd4s hq8Mqld84Cjlwhh8sdwMhMl222C84jqh2hhGsd4s 8180 RSVP 4803 1 Audit.RTUsage.RSVP Response Report ReportService Failure CCL-BIT-0005 A socket reported a communication error.CAM_Connect=0xfffffff4 <errorDetail><errorCode>-12</errorCode><errorMessage>CAM-CRP-0026 The underlying socket: '192.168.71.9:9300' returned an error.</errorMessage><errorStack><errorCode>10048</errorCode><errorMessage>Could not connect the socket, errno: 0x2740(10048)</errorMessage></errorStack></errorDetail>
192.168.71.9:9300 7104 2011-11-09 15:25:51.247 +0 61044C36AA6BF53E7DBD1001B232A6FF3E003BF4 hq8Mqld84Cjlwhh8sdwMhMl222C84jqh2hhGsd4s hq8Mqld84Cjlwhh8sdwMhMl222C84jqh2hhGsd4s 9616 RSVP 4803 1 Audit.RTUsage.RSVP Response Report ReportService Failure RSV-SRV-0063 An error occurred while executing the 'asynchRunSpecification_Request' command. CCL-BIT-0005 A socket reported a communication error. CAM_Connect=0xfffffff4 <errorDetail><errorCode>-12</errorCode><errorMessage>CAM-CRP-0026 The underlying socket: '192.168.71.9:9300' returned an error.</errorMessage><errorStack><errorCode>10048</errorCode><errorMessage>Could not connect the socket, errno: 0x2740(10048)</errorMessage></errorStack></errorDetail>

Cause

The Controller application server (specifically, the BI report server component) has run out of spare TCP/IP ephemeral (temporary / briefly used) ports.

More Information:
The Windows TCP/IP protocol uses multiple TCP/IP port numbers (typically in the range 1025-5000) for outbound connections. After being used, they will take some time (typically approximately 4 minutes) to be recycled so that they can be re-used. Under exceptional circumstances, it is possible that the Windows server has no more free ports available for outbound connections.

Diagnosing The Problem

If you have Sophos Endpoint Security and Control installed on the Controller application server, then you may find an entry inside the Event Viewer (application log) similar to the following:

    Sophos Message Router
    Sophos Enterprise Console has detected that the number of ephemeral ports being used on this computer is approaching the maximum permitted. You may need to make changes to the computer's TCP/IP configuration to prevent network problems.
    There are 3431 ephemeral ports in use. The maximum number of ephemeral ports available on this machine is 3976.
    See Sophos Knowledge Base article 4243 at
    http://www.sophos.com/support/knowledgebase

Resolving The Problem

Either:

(a) Increase the number of TCP ports available for outgoing connections

or (b) Decrease the recycle time for old/used ports.

Steps:

IMPORTANT: These changes are controlled by registry keys. Before continuing, please make sure that you know the risks of changing registry key settings. Refer to official third-party (Microsoft) documentation/recommendations (for example see links below) before continuing:

1. Obtain a short period of downtime (no users on the system)

2. Logon to the Controller application server (and/or reporting server) as a Windows administrator

3. Launch REGEDIT

4. Navigate to the following key: HKLM\System\CurrentControlSet\Services\Tcpip\Parameters

5. Create new (or modify existing) DWORD entries:


(a) "MaxUserPort" - this controls the number of TCP ports

TIP: Below is a suggested value:

  • Name: MaxUserPort
  • Type: DWORD
  • Recommended value: 15000 (decimal)

(b) "TcpTimedWaitDelay" - this controls the recycle time for old/used ports

TIP: Below are some suggested values:

  • Name: TcpTimedWaitDelay
  • Type: DWORD
  • Recommended value: 200 (decimal)

6. Reboot application server.

[{"Product":{"code":"SS9S6B","label":"IBM Cognos Controller"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Controller","Platform":[{"code":"PF033","label":"Windows"}],"Version":"10.1","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
15 June 2018

UID

swg21575283

Page Feedback