IBM Support

IBM BigInsights: Big SQL commands fail with ConnectException error

Troubleshooting


Problem

IBM BigInsights:Bigsql commands fail with ConnectException

Cause

On HDFS encryption enabled clusters, KMS (Key management server) should be accessible and running.

Environment

Linux

Diagnosing The Problem

The following error trace is observed in bigsql log
2016-04-05 22:23:20,685 ERROR com.ibm.biginsights.ie.files.FileImporter [Thread-3] : Error during File import

java.io.IOException: java.net.ConnectException: Connection refused
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:88)
at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:86)
at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2233)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:121)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:142)

Resolving The Problem


Identify the host where KMS server is running , by logging into Ambari and reading any of the following configuration variables of HDFS service

"dfs.encryption.key.provider.uri or hadoop.security.key.provider.path "

For example, if dfs.encryption.key.provider.uri is set to kms://http@bivm.ibm.com:16000/kms , kms is running on bivm.ibm.com

Check if kms process is running and occupying port 16000.

Following is an example on how to perform the check on linux

1) netstat -anp|grep 16000

[root@bivm /]# netstat -anp|grep 16000
tcp 0 0 :::16000 :::* LISTEN 6090/java

2) [root@bdavm327 /]# ps -ef|grep 6090
root 6090 1 0 Feb10 ? 00:54:08 /usr/bin/java -Djava.util.logging.config.file=/usr/kms-demo/hadoop/share/hadoop/kms/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dkms.home.dir=/usr/kms-demo/hadoop -Dkms.config.dir=/usr/kms-demo/hadoop/etc/hadoop -Dkms.log.dir=/usr/kms-demo/hadoop/logs -Dkms.temp.dir=/usr/kms-demo/hadoop/temp -Dkms.admin.port=16001 -Dkms.http.port=16000 -Dkms.max.threads=1000 -Dkms.ssl.keystore.file=/root/.keystore -Djava.library.path=/usr/kms-demo/hadoop/libexec/../lib/native/ -Djava.endorsed.dirs=/usr/kms-demo/hadoop/share/hadoop/kms/tomcat/endorsed -classpath /usr/kms-demo/hadoop/share/hadoop/kms/tomcat/bin/bootstrap.jar -Dcatalina.base=/usr/kms-demo/hadoop/share/hadoop/kms/tomcat -Dcatalina.home=/usr/kms-demo/hadoop/share/hadoop/kms/tomcat -Djava.io.tmpdir=/usr/kms-demo/hadoop/share/hadoop/kms/tomcat/temp org.apache.catalina.startup.Bootstrap start

If the kms is not running start the kms server from where its deployed .

For example, kms is deployed in /usr/kms-demo/hadoop then start the kms server using following command.

/usr/kms-demo/hadoop/sbin/

./kms.sh start &

[{"Product":{"code":"SSCRJT","label":"IBM Db2 Big SQL"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Big SQL","Platform":[{"code":"PF016","label":"Linux"}],"Version":"4.0.0;4.1.0;4.2.0","Edition":"Enterprise Edition;Basic Edition;Community Edition;Quick Start Edition","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
08 April 2021

UID

swg21980695

Page Feedback