Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions docs/interpreter/jdbc.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,70 @@ The last step is **Dependency Setting**. Since Zeppelin only includes `PostgreSQ

That's it. You can find more JDBC connection setting examples([Mysql](#mysql), [MariaDB](#mariadb), [Redshift](#redshift), [Apache Hive](#apache-hive), [Apache Phoenix](#apache-phoenix), and [Apache Tajo](#apache-tajo)) in [this section](#examples).

## JDBC Interpreter Datasource Pool Configuration
The Jdbc interpreter uses the connection pool technology, and supports users to do some personal configuration of the connection pool. For example, we can configure `default.validationQuery='select 1'` and `default.testOnBorrow=true` in the Interpreter configuration to avoid the "Invalid SessionHandle" runtime error caused by Session timeout when connecting to HiveServer2 through JDBC interpreter.

The Jdbc Interpreter supports the following database connection pool configurations:

<table class="table-configuration">
<tr>
<th>Property Name</th>
<th>Default</th>
<th>Description</th>
</tr>
<tr>
<td>testOnBorrow</td>
<td>false</td>
<td>The indication of whether objects will be validated before being borrowed from the pool. If the object fails to validate, it will be dropped from the pool, and we will attempt to borrow another.</td>
</tr>
<tr>
<td>testOnCreate</td>
<td>false</td>
<td>The indication of whether objects will be validated after creation. If the object fails to validate, the borrow attempt that triggered the object creation will fail.</td>
</tr>
<tr>
<td>testOnReturn</td>
<td>false</td>
<td>The indication of whether objects will be validated before being returned to the pool.</td>
</tr>
<tr>
<td>testWhileIdle</td>
<td>false</td>
<td>The indication of whether objects will be validated by the idle object evictor (if any). If an object fails to validate, it will be dropped from the pool.</td>
</tr>
<tr>
<td>timeBetweenEvictionRunsMillis</td>
<td>-1L</td>
<td>The number of milliseconds to sleep between runs of the idle object evictor thread. When non-positive, no idle object evictor thread will be run.</td>
</tr>
<tr>
<td>maxWaitMillis</td>
<td>-1L</td>
<td>The maximum number of milliseconds that the pool will wait (when there are no available connections) for a connection to be returned before throwing an exception, or -1 to wait indefinitely.</td>
</tr>
<tr>
<td>maxIdle</td>
<td>8</td>
<td>The maximum number of connections that can remain idle in the pool, without extra ones being released, or negative for no limit.</td>
</tr>
<tr>
<td>minIdle</td>
<td>0</td>
<td>The minimum number of connections that can remain idle in the pool, without extra ones being created, or zero to create none.</td>
</tr>
<tr>
<td>maxTotal</td>
<td>-1</td>
<td>The maximum number of active connections that can be allocated from this pool at the same time, or negative for no limit.</td>
</tr>
<tr>
<td>validationQuery</td>
<td>show database</td>
<td>The SQL query that will be used to validate connections from this pool before returning them to the caller. If specified, this query MUST be an SQL SELECT statement that returns at least one row. If not specified, connections will be validation by calling the isValid() method.</td>
</tr>
</table>


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use html table to render this. Here's one example of spark interpreter.

https://github.com/apache/zeppelin/blob/master/docs/interpreter/spark.md

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use html table to render this. Here's one example of spark interpreter.

https://github.com/apache/zeppelin/blob/master/docs/interpreter/spark.md

@zjffdu OK, I have updated the datasource pool docs in the jdbc.md with the html table syntax.

## More properties
There are more JDBC interpreter properties you can specify like below.

Expand Down
27 changes: 26 additions & 1 deletion jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import org.apache.zeppelin.interpreter.util.SqlSplitter;
import org.apache.zeppelin.jdbc.hive.HiveUtils;
import org.apache.zeppelin.tabledata.TableDataUtils;
import org.apache.zeppelin.util.PropertiesUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Expand Down Expand Up @@ -419,6 +420,28 @@ private void setUserProperty(String dbPrefix, InterpreterContext context)
}
}

private void configConnectionPool(GenericObjectPool connectionPool, Properties properties) {
boolean testOnBorrow = "true".equalsIgnoreCase(properties.getProperty("testOnBorrow"));
boolean testOnCreate = "true".equalsIgnoreCase(properties.getProperty("testOnCreate"));
boolean testOnReturn = "true".equalsIgnoreCase(properties.getProperty("testOnReturn"));
boolean testWhileIdle = "true".equalsIgnoreCase(properties.getProperty("testWhileIdle"));
long timeBetweenEvictionRunsMillis = PropertiesUtil.getLong(properties, "timeBetweenEvictionRunsMillis", -1L);
long maxWaitMillis = PropertiesUtil.getLong(properties, "maxWaitMillis", -1L);
int maxIdle = PropertiesUtil.getInt(properties, "maxIdle", 8);
int minIdle = PropertiesUtil.getInt(properties, "minIdle", 0);
int maxTotal = PropertiesUtil.getInt(properties, "maxTotal", -1);

connectionPool.setTestOnBorrow(testOnBorrow);
connectionPool.setTestOnCreate(testOnCreate);
connectionPool.setTestOnReturn(testOnReturn);
connectionPool.setTestWhileIdle(testWhileIdle);
connectionPool.setTimeBetweenEvictionRunsMillis(timeBetweenEvictionRunsMillis);
connectionPool.setMaxIdle(maxIdle);
connectionPool.setMinIdle(minIdle);
connectionPool.setMaxTotal(maxTotal);
connectionPool.setMaxWaitMillis(maxWaitMillis);
}

private void createConnectionPool(String url, String user, String dbPrefix,
Properties properties) throws SQLException, ClassNotFoundException {

Expand All @@ -441,8 +464,10 @@ private void createConnectionPool(String url, String user, String dbPrefix,
final String maxConnectionLifetime =
StringUtils.defaultIfEmpty(getProperty("zeppelin.jdbc.maxConnLifetime"), "-1");
poolableConnectionFactory.setMaxConnLifetimeMillis(Long.parseLong(maxConnectionLifetime));
poolableConnectionFactory.setValidationQuery("show databases");
poolableConnectionFactory.setValidationQuery(
PropertiesUtil.getString(properties, "validationQuery", "show databases"));
ObjectPool connectionPool = new GenericObjectPool(poolableConnectionFactory);
this.configConnectionPool((GenericObjectPool)connectionPool, properties);

poolableConnectionFactory.setPool(connectionPool);
Class.forName(driverClass);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.zeppelin.util;

import java.util.Properties;

/**
* java.util.Properties read utils
*/
public class PropertiesUtil {

/**
* read the string property
* @param properties the properties object
* @param key the property key
* @param defaultValue the default value
* @return the string result
*/
public static String getString(Properties properties, String key, String defaultValue) {
return properties.getProperty(key) == null ? defaultValue : properties.getProperty(key);
}


/**
* read the long property
* @param properties the properties object
* @param key the property key
* @param defaultValue the default value
* @return the long result
*/
public static long getLong(Properties properties, String key, long defaultValue) {
String valueString = properties.getProperty(key);
if (valueString == null){
return defaultValue;
}
try {
return Long.parseLong(valueString);
}catch (Exception e){
return defaultValue;
}
}

/**
* read the long property
* @param properties the properties object
* @param key the property key
* @param defaultValue the default value
* @return the int result
*/
public static int getInt(Properties properties, String key, int defaultValue) {
String valueString = properties.getProperty(key);
if (valueString == null){
return defaultValue;
}
try {
return Integer.parseInt(valueString);
}catch (Exception e){
return defaultValue;
}
}
}