Configuring WebSphere eXtreme Scale to use FIPS 140-2

Federal Information Processing Standard (FIPS) 140-2 specifies required levels of encryption for Transport Layer Security/Secure Sockets Layer (TLS/SSL). This standard ensures high protection of data as it is sent over the wire.

1. Edit the java.security file.
   The location of the java.security depends on your Java virtual machine (JVM) configuration:

   • If you are using the default JVM that ships with the product, the file is in the wxs_install_root/java/jre/lib/security directory.
   • If you are using a different JVM, edit the file in the java_home/jre/lib/security directory.
   • If you installed WebSphere eXtreme Scale on an HP-UX or Solaris operating system, WebSphere eXtreme Scale requires the IBM hybrid JDK. A hybrid platform is a platform where you get 32-bit and 64-bit instance support in the same installation. You must update the java.security file in the jre/lib/security directory to include the IBMJCEFIPS provider first, such as: security.provider.1=com.ibm.crypto.fips.provider.IBMJCEFIPS.

   The file must contain the following text:

   security.provider.1=com.ibm.crypto.fips.provider.IBMJCEFIPS
   security.provider.2=com.ibm.jsse2.IBMJSSEProvider2
   security.provider.3=com.ibm.crypto.provider.IBMJCE
   security.provider.4=com.ibm.security.jgss.IBMJGSSProvider
   security.provider.5=com.ibm.security.cert.IBMCertPath
   security.provider.6=com.ibm.security.sasl.IBMSASL
   security.provider.7=com.ibm.xml.crypto.IBMXMLCryptoProvider
   security.provider.8=com.ibm.xml.enc.IBMXMLEncProvider
   security.provider.9=org.apache.harmony.security.provider.PolicyProvider
   security.provider.10=com.ibm.security.jgss.mech.spnego.IBMSPNEGO

   Important: If you are running Solaris or HP-UX you need to include the following line:

   security.provider.11=sun.security.provider.Sun

2. Edit the Secure Sockets Layer (SSL) configuration in the server properties files for the catalog server and container servers to use the TLSv1 protocol. You must also configure any clients that access the catalog and container servers to use the TLSv1 protocol.
   These files must contain the following properties and values:

   contextProvider=IBMJSSE2
   transportType=SSL-Required
   protocol=TLSv1
   fips=true

   For more information about SSL properties, see Server properties file and Client properties file.
3. Create a keystore with a FIPS 140-2 compatible certificate, for use by the catalog and container servers. The certificate should not use an MD5 signature algorithm. FIPS does not dictate RSA or key length. For more information, see Enabling keystore authentication in eXtreme Scale container and catalog servers.
4. Restart your catalog and container servers.
   When you start the catalog servers, you must specify Java virtual machine (JVM) arguments. The arguments that you use depend on which version of Java SE you are using. If the FIPS property is set and the -Dcom.ibm.jsse2.usefipsprovider argument is set when you start the server, the JVM setting overrides the FIPS option that is configured in the security server properties.

   • For Java 6 SR 10 and later, or Java 7, specify the -Dcom.ibm.jsse2.usefipsprovider=true argument when you start the server.

   Restriction: When WebSphere eXtreme Scale is configured to run with the ORB transport, you cannot configure SSL to use both FIPS encryption and SP800-131a data protection. Running with both security standards is only allowed when eXtreme Scale is configured to run with the eXtremeIO (XIO) transport.

   For more information, see Starting and stopping secure servers.