IBM Support

Configuring TLS encryption in IBM FileNet Content Foundation for email notification results in error javax.net.ssl.SSLException: Unrecognized SSL message error

Troubleshooting


Problem

When configuring TLS encryption, as part of the configuration for email notification encryption (SSL/TLS) in IBM FileNet Content Foundation V5.2.1 and using the target SMTP servers TLS port, it fails with the error : Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

Symptom

The following error stack is found in the pesvr_trace.log file on the PE server when an email notification is attempted with an SMTP server using the TLS port:

Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

Cause

This has been identified as a product defect under APAR PJ43435 and has been fixed in CPE 5.2.1 Fix Pack 3.

Environment

SSL encryption is supported only with P8CPE 5.2.1 or later, TLS encryption is supported with 5.2.1.3-P8CPE-FP003.

Diagnosing The Problem

After enabling SSL encryption for email notification on a CPE 5.2.1 server, and configuring the TLS port, all email notifications fail with the error, which can be seen in the pesvr_trace.log file on the CPE server with "email notification" and "exceptions" tracing options enabled.

Resolving The Problem

1. Upgrade the CPE 5.2.1 server to 5.2.1 FP003.

2. Activate the new STARTTLS feature as follows:

    1. Add the prefix "STARTTLS:" to the SMTP host field value in the ACCE SMTP Subsystem section. For example, to correctly connect to Gmail's STARTTLS service, the following SMTP host field value is used: "STARTTLS:smtp.gmail.com".
    2. The SMTP port number and login ID must be set according to the SMTP server requirements.
    3. When the STARTTLS feature is activated, the "Enable SSL" field in ACCE will be ignored.
    4. STARTTLS protocol requires an initial SMTP connection without SSL, then part way through the STARTTLS protocol handshake, TLS/SSL will be enabled and used by both. Therefore, it may be necessary to import the SMTP server's trust certificate into the customer's application server's keystore in order for TLS/SSL to work correctly.

NOTE:

The target SMTP server will dictate the correct port for SSL configuration or or TLS configuration. For example, this gmail link** provides the valid ports:


https://support.google.com/mail/troubleshooter/1668960?hl=en#ts=1665119,1665160,2769079


Outgoing Mail (SMTP) Server - requires TLS or SSL: smtp.gmail.com
Use Authentication: Yes
Port for TLS: 587
Port for SSL: 465

MS Office 365 SMTP server supports only TLS encryption (no SSL)**:


https://support.office.com/en-sg/article/Outlook-settings-for-POP-and-IMAP-access-for-Office-365-for-business-or-Microsoft-Exchange-accounts-7fc677eb-2491-4cbc-8153-8e7113525f6c

Encryption SMTP Server Port Version
SMTP smtp.office365.com 587 explicit TLS v1-1.2

**: The links external to IBM were valid at the time of the publication of this Technote.

Setting the TLS protocol:
In 5.5.8-IF6, 5.5.10-IF3, 5.5.11-IF1, 5.5.12, and newer releases, a new parameter will allow the customer to specify the mail smtp protocols.
-Dcom.filenet.mail.smtp.protocols=[TLS_version] as a JVM argument to the CPE server instances and will require the CPE restart.
in example
-Dcom.filenet.mail.smtp.protocols=TLSv1.2

Email clients that use implicit TLS over SSL port (465) are required to set the TLS protocol flag.
Optionally - Email clients that use TLS protocol over TLS port (587) optionally need to set the TLS protocol flag if the version of TLS needs to be set.

[{"Product":{"code":"SSTHRT","label":"IBM Case Foundation"},"Business Unit":{"code":"BU053","label":"Cloud \u0026 Data Platform"},"Component":"Process Engine","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"5.2.1","Edition":"All Editions","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
31 January 2024

UID

swg21963707