Certificate expiration monitoring in SSL

SSL Certificates are important to a server to maintain the security of data that is passed to and from other servers. With the use of SSL certificate's comes the responsibility to keeping them up to date. The certificate expiration monitor administrative task is a scheduled task that cycles through all the keystores in the security configuration and reports on any certificates that are expired, certificates that fall within the expiration threshold, and certificates that fall within the pre-notification period.

Important: The certificate expiration monitor covers keystores under the server configuration. A client keystore, such as (profile)/etc/key.p12 is not monitored.
Configuring the certificate monitor
Certificate monitor configuration settings can be configured on the administrative console by selecting Security > SSL certificate and key management > Manage certificate expiration.. On the Manage certificate expiration panel, you can perform the following functions.
  • Enable or disable certificate monitor checking on the Enable Checking check box. Certificate monitor checking is enabled by default.
  • Set the expiration threshold in the box labeled Expiration replacement threshold. The expiration replacement threshold is the number of days before a certificate expiration that a certificate can be replaced and has a default value of 60 days.
  • Set the pre-notification threshold in the box labeled Certificate pre-notification threshold. The certificate pre-notification threshold is the number of days before the expiration threshold where the certificate monitor reports the date that a certificate can start getting replaced.
  • Replace expired certificates or certificates in the expiration threshold with a new certificate if the Automatically replace expiring self-signed and chained certificates box is selected. This box is selected by default. If this option is not selected, then the certificate expiration monitor only reports that a certificate is expired or is within the expiration threshold and does not replace the certificate.
  • Delete the certificates after they are replaced by selecting the Delete expiring certificates and signers after replacement box. This box is selected by default.
  • The schedule that the certificate expiration monitor is run can be set in the Expiration checking field. As a default, the monitor is set to run every four weeks on a Sunday at 9:30 PM.
  • How to receive the certificate monitor report can be set in the Expiration check notification field. As a default, the report is written to the System out log.
The following security.xml configuration object specifies when the monitor task starts, determines the certificate expiration threshold, and indicates whether you are notified in an email by using Simple Mail Transfer Protocol (SMTP) or in a message log. The pre-notification log is set in a custom property.
<wsCertificateExpirationMonitor xmi:id="WSCertificateExpirationMonitor_1"
name="Certificate Expiration Monitor" daysBeforeNotification="30"
isEnabled="true" autoReplace="true" deleteOld="true"
wsNotification="WSNotification_1" wsSchedule="WSSchedule_2"
nextStartDate="1134358204849"/>
The pre-notification period is set by a custom property in the security.xml file. 
<properties xmi:id="Property_1422758742456"
name="com.ibm.ws.security.expirationMonitorNotificationPeriod" value="90"/>
The following sample the schedule object shows the frequency attribute as the number of days between each run of the certificate monitor. 
<wsSchedules xmi:id="WSSchedule_2" name="ExpirationMonitorSchedule" 
frequency="30" dayOfWeek="1" hour="21" minute="30"/>

The dayOfWeek attribute adjusts the schedule to run on a specified day of the week, which is always the same day regardless of whether the frequency is set to 30 or 31 days. Based on 24-hour clock, the hour and minute attributes determine when the expiration monitor is started on the specified day.

The following sample code of the notification object shows the notification configuration, which notifies you after the expiration monitor runs. 
<>wsNotifications xmi:id="WSNotification_1" name="MessageLog" logToSystemOut="true" emailList=""/

For expiration monitor notifications, you can select message log, email by using SMTP server, or both methods of notification. When you configure the email option, use the format user@domain@smtpserver. If you do not specify an SMTP server, WebSphere® Application Server defaults to the same domain as the email address. For example, if you configure joeuser@ibm.com, WebSphere Application Server attempts to call smtp-server.ibm.com. To specify multiple email addresses by using scripting, you must add a pipe (|) character between entries. When you specify the logToSystemOut attribute, the expiration monitor results are sent to the message log for the environment, which is typically the SystemOut.log file.

The following functions can be performed by using the certificate expiration monitor
  1. Clears out the NodeDefaultDeletedStore or DmgrDefaultDeletedStore. This operation is performed silently without reporting that the certificates are deleted.
  2. Checks the root keystores, DmgrDefaultRootStore, or NodeDefaultRootStore and the DmgrRSATokenRootStore or NodeRSATokenRootStore. If any root certificates are expired, fall within the threshold period, or the pre-notification period, then the certificate is noted in the report.
  3. Any root certificates that are expired or fall in the threshold period are re-created using the same information that is used to create the original one.. Any signer certificates from the original root certificate are replaced with the signers from the new root certificate.
  4. If a root certificate is replaced, then all the keystores are checked to see any chained certificates that are signed with the original root certificate. If there are, then the chain certificate is renewed (re-created with the new root certificate). Any signer certificate from the original certificate is replaced with the signer from the re-created certificate.
  5. After all root keystores are processed, the rest of the keystores are checked for expired certificates, certificates in the expiration threshold, or certificates in the pre-notification period. Any certificate that falls in any one of these categories is noted in the report.
  6. If there are any expired certificates or certificates in the expiration threshold period and these certificates are self-signed certificates or chained certificates that are created by WebSphere, then they are replaced. If the chained certificates root is not in the root keystore, then, it is re-created as a default root certificate. Any signer certificates from the original certificate are replaced with the signer from the new certificate."
  7. Depending on the configuration settings, a report is generated and either returned, written to a log file, or mailed.
The type of certificates replaced

The expiration monitor replaces self-signed certificates and chained personal certificates that are signed by a root certificate in DmgrDefaultRootStore or NodeDefaultRootStore. Self-signed certificates are renewed by using all the information that was used to create the original self-signed certificate. A chained certificate is renewed by using the same root certificate that was used to sign the original certificate. Certificates that are created by a certificate authority (CA) cannot be replaced by the certificate monitor and needs to be replaced by the administrator.

Signer certificates are the public part of a personal certificate. So they are not created on their own but extracted from a personal certificate. If the certificate monitor replaces a personal certificate, it checks all keystores to see if there are any signer certificates extracted from the expired personal certificate and replaces it with a signer certificated extracted from the newly created personal certificate. If a signer certificate was added to a keystore from a certificate file or added by using the retrieve from port option, then, it is likely to be replaced. If it is expired or in the expiration threshold, then users are notified that the signer certificate is not replaced. Delete and replace the certificate with a valid one.

The certificate expiration report

When the certificate expiration monitor runs, a report is generated reporting what was done. This report is written to the SystemOut.log by default. Users can configure to have the certificate monitor report that is sent to them in email. If a user runs the certificate expiration monitor from the wsadmin AdminTask startCertificateExpMonitor, the report is returned to the user and the location that the Certificate expiration notification setting says to send the report.

If the certificate monitor runs and there are no expired certificate to report the output, it returns an informational message that indicates that the certificates were checked and no expiration issues were found.. For example,
**** Subject:  Expiration Monitor ****;

Hostname: myHost
Profile UUID:AppSrv01-BASE-8da8e11e-70c6-451f-862b-8a08e408110b
Process type: UnManagedProcess
Checking for expired certificate and certificates in the 60 days threshold period.
CWPKI0735I: All certificates were searched and no expiration issues were found.

The certificate expiration monitor reports information about certificates in sections. The pre-notification section, the expired certificate section, the replaced certificate section, and the certificates that cannot be replaced section. Certificates in the report are identified by whether they are a personal certificate or a signer certificate, its alias name, and the keystore the certificate is in.

The pre-notification section lists all certificate in the pre-notification threshold. It serves as a warning period to tell users what date a certificate will be replaced if the certificate expiration monitor runs on or after that date. The section also provides a warning about why users might need to be concerned about the upcoming certificate replacement. If there are no certificates in the pre-notification period, then it is not part of the report. For example,
*** CERTIFICATES WITHIN THE 90 DAYS OF THE CERTIFICATE EXPIRATION THRESHOLD (MAY BE REPLACED WITHIN 90 DAYS)  ***;

CWPKI0714I: The certificate expiration monitor has recently run and discovered that the certificates, which are listed in associated messages, will be replaced within the next 90 days. This replacement is based on the configured policy
to automatically replace expiring self-signed certificates 60 days prior to expiration. This notification informs you that problems might arise when the certificates are automatically replaced.

CWPKI0715I: In some cases, automatically replacing certificates can cause outages for Web server plug-ins operating on unmanaged nodes. In such a situation, the plug-in will be unable to contact the application servers over HTTPS
because it will be using signers for certificates that have been replaced by the automatic replacement process.  To prevent what may be a serious outage you should act before the scheduled replacement date and replace the expiring certificates and update the plug-in kdb
to use the new signers.

CWPKI0719I: The testcert1 personal certificate in the "testKeyStore((cell):myHostNode01Cell:(node):myHostNode01)" keystore is due to expire on May 10, 2015 and might be replaced after the Mar 11, 2015 threshold date.
The expired certificates section identifies expired certificate or certificates in the expiration threshold. If the certificate is in the expiration threshold, it says when the certificate expires. For example, 
*** CERTIFICATES THAT ARE EXIPIRED OR IN THE EXPIRATION THRESHOLD ***;

CWPKI0642I: Signer certificate alias "oldcert" in KeyStore "NodeDefaultTrustStore((cell):myHostNode01Cell:(node):myHostNode01)" will expire on Feb 19, 2015.

CWPKI0680I: Personal certificate alias "selfcert" in KeyStore "testKeyStore((cell):myHostNode01Cell:(node):myHostNode01)" expired on Nov 9, 2014.

CWPKI0643I: Personal certificate alias "testcert" in KeyStore "testKeyStore((cell):myHostNode01Cell:(node):myHostNode01)" will expire on Feb 19, 2015.
The replaced certificate section list all certificates that were replaced. Self-signed personals certificate and chained personal certificates that are signed with a root certificate in the server's root keystore can be replaced. Signer certificates can be replaced by signer that is extracted from personal certificates that were replaced.
*** CERTIFICATES THAT ARE EXPIRED OR BEYOND THE EXPIRATION THRESHOLD AND HAVE BEEN REPLACED ***;

CWPKI0645I: Personal certificate alias "selfcert" in KeyStore "testKeyStore((cell):myHostNode01Cell:(node):myHostNode01)" was REPLACED.

CWPKI0644I: Signer certificate alias "selfcert" in KeyStore "NodeDefaultTrustStore((cell):myHostNode01Cell:(node):myHostNode01)" was REPLACED.

CWPKI0645I: Personal certificate alias "testcert" in KeyStore "testKeyStore((cell):myHostNode01Cell:(node):myHostNode01)" was REPLACED.
The certificates that cannot be replaced section list all certificates that are expired or in the expiration threshold that cannot be replaced. A personal certificate is not replaced if it is in a read only keystore or the certificate monitor does not have the information need to create a new one. If the certificate is signed by a certificate that is not in the server's root keystore it cannot be replaced; it is likely a CA certificate. A signer certificate cannot be replaced if the personal certificate it came from does not exist in the configuration. For example,
*** CERTIFICATES THAT ARE EXPIRED OR BEYOND THE EXPIRATION THRESHOLD THAT CANNOTBE REPLACED BY THE SERVER ***;

CWPKI0642I: Signer certificate alias "oldcert" in KeyStore "NodeDefaultTrustStore((cell):myHostNode01Cell:(node):myHostNode01)" will expire on Feb 19, 2015.
Clean out the deleted keystore file

The expiration monitor clears out the deleted certificate keystore file. The deleted keystore file, NodeDefaultDeletedStore, is where copies of certificates that were deleted from other keystores are put in case a user needs to restore it. The file needs to be cleaned up from time to time so when the certificate monitor runs the deleted keystore file is cleaned out. Due to the nature of the PKCS12 keystore, there must be at least one entry in the keystore. There is no reporting on the certificates that are deleted from the deleted keystore file.

Note: When the following configuration option is enabled, the run time can be dynamically affected when the expiration monitor replaces certificates.
Note: The following configuration option only takes affect when Global Security is enabled.

Security > SSL certificate and key management. In the configuration settings, check the check box for Dynamically update the run time when SSL configuration changes occur.

When enabled, any certificates that are replaced cause the client SSL run time to use the new certificates immediately, which in turn, flushes SSL and keystore caches and causes some ports that use SSLServerSockets (RMI/IIOP on distributed and Admin SOAP) to restart. Restarting ports breaks existing connections. These connections can be reconnected after the port restart is completed. Endpoints that use the channel framework (HTTP, BUS, RMI/IIOP on z/OS) leave existing connections unaffected but still use the new certificates for new connections.

When the dynamic change property is disabled and before the new certificates become effective, the administrator needs to recycle all processes in the entire cell after each node has the synchronized configuration.