Configuring JDBC access to Spark data sources

Answer

Spark that are accessed through the JDBC API are referred to in this text as Spark data sources.

Before you begin

The spark JDBC driver must be installed and configured on the computer that acts as the federated server.

Spark provides an JDBC driver for the AIX, Linux operating systems.

For prerequisites and limitations that apply to all JDBC data sources, See Configuring access to JDBC data sources.

Restrictions

• IUD and Transparent IUD are not supported.
• Method 1 is not supported for NNSTAT.
• Spark type: Binary supports binary(256) for query and will be truncated when the lengh > 256.
• Spark type: Binay and String will be truncated when the lengh > 32767 in passthru-mode session.
• TINYINT, SMALLINT, INT, BIGINT, DECIMAL types return "-" when remote data overflow.
• COMMIT and ROLLBACK is not supported.

Procedure

1. Use the following method to prepare the federated server and federated database: Prepare the federated server to access data sources through JDBC.

Now we take spark1.3.1 + hive1.2.1 on linuxamd64 for example:

1. Setup spark and then start thriftserver using $spark_path/sbin/start-thriftserver.sh

2. Find Jdbc driver:hive-jdbc-0.14.0.2.2.4.2-2-standalone.jar and hadoop-common-2.6.0.2.2.4.2-2.jar, put them into the computer that acts as the federated server.

3. Set CLASSPATH for jdbc driver and then jdbc wrapper can load jdbc driver to connect to spark.