Hive and other dbs with non-standard DatabaseMetaData implementation support

[snuyanzin]

The issue is that while trying to connect to hive (from HDP 2.6.5)
there are exceptions (full traces are below) like method not supported for java.sql.DatabaseMetaData#supportsTransactionIsolationLevel and java.sql.DatabaseMetaData#storesUpperCaseIdentifiers. At the same time javadoc from java.sql.DatabaseMetaData says (for both)

• @return true if so; false otherwise

The proposal here is to wrap DatabaseMetaData and in case of SQLFeatureNotSupportedException or exception message Method not supported use default behavior defined by java.sql.DatabaseMetaData instead of failure. Both SQLFeatureNotSupportedException and exception message Method not supported should be used as before and after https://issues.apache.org/jira/browse/HIVE-15730 the behavior is different.

java.sql.SQLException: Method not supported                                                                                                                                                                        
        at org.apache.hive.jdbc.HiveDatabaseMetaData.storesUpperCaseIdentifiers(HiveDatabaseMetaData.java:838)                                                                                                     
        at sqlline.DatabaseConnection.setCompletions(DatabaseConnection.java:55)                                                                                                                                   
        at sqlline.SqlLine.setCompletions(SqlLine.java:1728)                                                                                                                                                       
        at sqlline.Commands.connect(Commands.java:1243)                                                                                                                                                            
        at sqlline.Commands.connect(Commands.java:1134)                                                                                                                                                            
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)                                                                                                                                             
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)                                                                                                                           
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)                                                                                                                   
        at java.lang.reflect.Method.invoke(Method.java:498)                                                                                                                                                        
        at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)                                                                                                                              
        at sqlline.SqlLine.dispatch(SqlLine.java:673)                                                                                                                                                              
        at sqlline.SqlLine.begin(SqlLine.java:537)                                                                                                                                                                 
        at sqlline.SqlLine.start(SqlLine.java:262)                                                                                                                                                                 
        at sqlline.SqlLine.main(SqlLine.java:193)     

and

java.sql.SQLException: Method not supported                                                                                                                                                                        
        at org.apache.hive.jdbc.HiveDatabaseMetaData.supportsTransactionIsolationLevel(HiveDatabaseMetaData.java:1102)                                                                                             
        at sqlline.Commands.isolation(Commands.java:816)                                                                                                                                                           
        at sqlline.DatabaseConnection.connect(DatabaseConnection.java:161)                                                                                                                                         
        at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:178)                                                                                                                                   
        at sqlline.Commands.connect(Commands.java:1242)                                                                                                                                                            
        at sqlline.Commands.connect(Commands.java:1134)                                                                                                                                                            
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)                                                                                                                                             
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)                                                                                                                           
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)                                                                                                                   
        at java.lang.reflect.Method.invoke(Method.java:498)                                                                                                                                                        
        at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)                                                                                                                              
        at sqlline.SqlLine.dispatch(SqlLine.java:673)                                                                                                                                                              
        at sqlline.SqlLine.begin(SqlLine.java:537)                                                                                                                                                                 
        at sqlline.SqlLine.start(SqlLine.java:262)                                                                                                                                                                 
        at sqlline.SqlLine.main(SqlLine.java:193)                                                                                                                                                                  

[julianhyde]

I like the idea of wrapping DatabaseMetaData to mask errors.

JDBC APIs tend to change from one version of Java to another, so a simple 'class DelegatingDatabaseMetaData implements DatabaseMetaData { ... }' may not work. I suggest that you write the wrapper using reflection and invocation handlers.

How about adding a mode, say strictJdbc=true, that would disable the wrappers? It would allow developers of JDBC drivers to see bugs in their drivers. If you don't think this mode would be useful or worth the effort I quite understand.

[snuyanzin]

Yes I think it makes sense to use reflection here, thank you.

I added property you mentioned strictJdbc with false default value. However I use the wrapper all the time but if strictJdbc=true then there will be performed the only requested method call with provided arguments and no other stuff.

Also there were added tests emulated hive's behavior and a separate test for wrapper.

Current working with hive version
https://github.com/snuyanzin/sqlline/tree/SQLLINE_183

[julianhyde]

I'm not totally comfortable with DatabaseMetaDataWrapper (1) being a class not an interface, (2) not being an implementation of DatabaseMetaData. However, I can swallow those objections - we will discover how well this code works as time progresses.

DatabaseMetaDataWrapper is currently public. Can I make it package-protected? Then we can change our minds about it later.

[snuyanzin]

Can I make it package-protected?

yes, I do not have objections.

(2) not being an implementation of DatabaseMetaData

do not understand you here as before you mentioned

JDBC APIs tend to change from one version of Java to another, so a simple 'class DelegatingDatabaseMetaData implements DatabaseMetaData { ... }' may not work. I suggest that you write the wrapper using reflection and invocation handlers.

[julianhyde]

If you use a reflection proxy, the Java runtime will generate an implementation of DatabaseMetaData on the fly, calling a proxy you supply.

If you want to expose additional functionality that is not in the DatabaseMetaData interface, users could call databaseMetaData.unwrap(YourHandler.class) to access an additional handler.

[julianhyde]

Fixed in e2c67fe, PR #196; thanks @snuyanzin!