[BUG] OpenSearch ODBC client will not connect to OpenSearch cluster

[dredwilliams]

Describe the bug
OpenSearch ODBC client will not connect to OpenSearch cluster.

Using the 'Test' button in the ODBC Data Sources Configuration dialog, I get the "Connection error: SQL plugin is not available, please install the SQL plugin to use this driver" error message. Using curl with the same parameters in a WSL window on the same system works as expected.

Tracing through the code, this message is generated in the CheckSQLPluginAvailability() method in opensearch_communication.cpp. Based on what I can see in the log on the Windows side (see below), it is being triggered by an exception thrown in the ParseErrorResponse() method -- which probably means it is getting a non-JSON response to parse. The problem is, I can't see what that response is -- I can only sometimes get the OpenSearch logs to show a connection (though communication is confirmed using tcpdump on the server side), much less tell me what it is sending back. The root log level of the node I'm trying to connect to is set to DEBUG ... I've tried to set to TRACE, but couldn't sift any additional useful information out -- but if y'all can tell me what to search for, I'll set it to TRACE again and send the results.

When using curl to test, I can see messages from o.o.s.l.p.RestSqlAction showing the query received, but not when I test the ODBC driver -- which makes me wonder if the driver is sending the appropriate API call at all? Again, I can't see enough in the logs to answer this question.

I see similar issues raised in several different threads in the forums, with the frequent cause being SSL configuration, possible certificate issues, but (as you see below) I've worked through those issues to make sure I've got those bases covered. I installed the cluster's Root CA in the windows trusted authorities keystore, and verified using a browser that the cluster's cert is accepted.

To Reproduce
Steps to reproduce the behavior:

1. Download ODBC driver (see issue Publish download links for ODBC and JDBC drivers sql#244 )
2. Install driver and open ODBC Data Sources (64 bit) tool
3. Enter parameters (see screenshot below) on main screen, selecting 'SSL' from advanced options sub-dialog (hostname validation is not set, but appears to make no difference)
4. Click the 'test' button
5. Error dialog pops up after timeout period (see below)
6. Logs from the Windows side are also below

Expected behavior
The ODBC driver should connect with the OpenSearch cluster

Plugins
These are the plugins currently installed:

opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-notebooks
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-sql

Screenshots
Configuration Dialog:

Error Dialog:

Windows Log:

opensearch[OpenSearchCommunication::LogMsg]832: Verifying connection options.
opensearch[OpenSearchCommunication::LogMsg]832: Required connection option are valid.
opensearch[OpenSearchCommunication::LogMsg]832: Starting DB connection.
opensearch[OpenSearchCommunication::LogMsg]832: Attempting to establish DB connection.
opensearch[OpenSearchCommunication::LogMsg]832: 
opensearch[OpenSearchCommunication::LogMsg]832: Checking for SQL plugin status.
opensearch[OpenSearchCommunication::LogMsg]832: Parsing error response (with schema validation)
opensearch[OpenSearchCommunication::LogMsg]832: Unexpected exception thrown from the server, the SQL plugin is not installed or in unhealthy status.
opensearch[OpenSearchCommunication::LogMsg]832: SQL plugin is not available, please install the SQL plugin to use this driver.
opensearch[OpenSearchCommunication::LogMsg]832: Dropping DB connection.
connection[CC_set_error_statements]564: entering self=0000025EB7BD36E0
connection[CC_log_error]652: CONN ERROR: func=LIBOPENSEARCH_connect, desc='', errnum=202, errmsg='Connection error: [OpenSearch][SQL ODBC Driver][SQL Plugin] Connection error: SQL plugin is not available, please install the SQL plugin to use this driver.'
connection[CC_log_error]655:             ------------------------------------------------------------
connection[CC_log_error]658:             henv=0000025EB7BAD060, conn=0000025EB7BD36E0, status=0, num_stmts=16
connection[CC_log_error]660:             opensearchconn=0000000000000000, stmts=0000025EB7BA2140, lobj_type=-999
connection[CC_log_error]652: CONN ERROR: func=OPENSEARCHAPI_DriverConnect, desc='Error from CC_Connect', errnum=202, errmsg='Connection error: [OpenSearch][SQL ODBC Driver][SQL Plugin] Connection error: SQL plugin is not available, please install the SQL plugin to use this driver.'
connection[CC_log_error]655:             ------------------------------------------------------------
connection[CC_log_error]658:             henv=0000025EB7BAD060, conn=0000025EB7BD36E0, status=0, num_stmts=16
connection[CC_log_error]660:             opensearchconn=0000000000000000, stmts=0000025EB7BA2140, lobj_type=-999
odbcapi30w[SQLGetDiagRecW]208: entering
opensearch[OPENSEARCHAPI_GetDiagRec]50: entering type=2 rec=1
 environ.c[OPENSEARCHAPI_ConnectError]239: entering hdbc=0000025EB7BD36E0 <512>
connection[CC_get_error]629: entering
connection[CC_get_error]641: leaving
 environ.c[OPENSEARCHAPI_ConnectError]256: CC_get_error: status = 202, msg = #Connection error: [OpenSearch][SQL ODBC Driver][SQL Plugin] Connection error: SQL plugin is not available, please install the SQL plugin to use this driver.#
 environ.c[OPENSEARCHAPI_ConnectError]343: 	     szSqlState = '08001',len=156, szError='Connection error: [OpenSearch][SQL ODBC Driver][SQL Plugin] Connection error: SQL plugin is not available, please install the SQL plugin to use this driver.'
opensearch[OPENSEARCHAPI_GetDiagRec]71: leaving 0
win_unicod[utf8_to_ucs2_lf]254: ilen=156 bufcount=512 ocount=156
odbcapi30.[SQLFreeHandle]230: entering
connection[OPENSEARCHAPI_FreeConnect]218: entering...hdbc=0000025EB7BD36E0
connection[CC_Destructor]325: entering self=0000025EB7BD36E0
connection[CC_cleanup]424: entering self=0000025EB7BD36E0
connection[CC_set_error_statements]564: entering self=0000025EB7BD36E0
connection[CC_log_error]652: CONN ERROR: func=CC_cleanup, desc='', errnum=1, errmsg='Connection not open'
connection[CC_log_error]655:             ------------------------------------------------------------
connection[CC_log_error]658:             henv=0000025EB7BAD060, conn=0000025EB7BD36E0, status=0, num_stmts=16
connection[CC_log_error]660:             opensearchconn=0000000000000000, stmts=0000025EB7BA2140, lobj_type=-999
connection[CC_cleanup]439: after LIBOPENSEARCH_disconnect
dlg_specif[CC_conninfo_init]463: entering opt=1
connection[CC_cleanup]505: leaving
connection[CC_Destructor]332: after CC_Cleanup
connection[CC_Destructor]343: after free statement holders
connection[CC_Destructor]354: leaving
connection[OPENSEARCHAPI_FreeConnect]236: leaving...
odbcapi30.[SQLFreeHandle]230: entering
 environ.c[OPENSEARCHAPI_FreeEnv]80: entering env=0000025EB7BAD060
 environ.c[EN_Destructor]453: entering self=0000025EB7BAD060
 environ.c[EN_Destructor]476: clearing conns count=128
 environ.c[EN_Destructor]488: leaving rv=1
 environ.c[OPENSEARCHAPI_FreeEnv]83:    ok

Host/Environment (please complete the following information):
Client - Windows 10 (latest updates)
ODBC Driver - tested with versions 1.1.0.1 and 1.2 - same behavior
Cluster - OpenSearch 1.1.0 (tarball) running on Debian 11 & Fedora 34 hosts (the host parameter was pointed to a Debian host)

Additional context
I'm curious if the 299 Deprecation warning is interfering with the exchange between the ODBC driver and the cluster, but I cannot find a way to shut it off to validate that hypothesis.

[chloe-zh]

We currently are working on the release of ODBC and JDBC drivers for OpenSearch. The release should be working fine with your opensearch cluster. Another workaround is to get the latest installer directly from the source code / GitHub actions cicd workflows, which is not a stable and officially released one though.

[dredwilliams]

@chloe-zh I'm glad it's being worked, but I pulled the installer that was created at 3pm EST yesterday (commit 65f449b) in the "OpenSearch ODBC Driver" workflow -- same result. The config dialog had a version 1.2.0.0. The logs on the Windows side show nothing different.

[chloe-zh]

Could you check the error log from opensearch side, that would be helpful! Thanks!

[dredwilliams]

I turned logging up to TRACE

rootLogger.level = trace

in log42j.properties ... and got these two messages in the cluster.log:

[2021-11-11T17:25:51,995][TRACE][o.o.h.AbstractHttpServerTransport] [poggin] Http channel accepted: Netty4HttpChannel{localAddress=/10.0.0.66:9200, remoteAddress=/10.0.0.7:62739}

[2021-11-11T17:25:52,035][TRACE][o.o.h.AbstractHttpServerTransport] [poggin] Http channel accepted: Netty4HttpChannel{localAddress=/10.0.0.66:9200, remoteAddress=/10.0.0.7:62740}

I can find no other trace of the attempt ... but again, I verified that there was data exchanged between the two systems using tcpdump.

Are there any other logging settings I should adjust, or additional places to look?

[dredwilliams]

UPDATE: I spun up a new cluster for testing purposes, and disabled SSL on the public API (port 9200) -- the ODBC driver connected just fine -- so it is an SSL issue. I turn on SSL and it behaves as described before.

I have loaded (and verified) the CA for the server certificate into the Windows trusted authorities keystore:

• I can pull up the cluster in a browser with no warnings about insecure certificates
• I can use 'curl' in a WSL window on the same system with no warnings

So -- it appears that the ODBC driver SSL routines are not respecting/using the system CA store? Am I missing something?

[adcalsie2]

I'm having the same problems in an open distro cluster. Waiting for solution

[diego2glez]

@chloe-zh

I'm been experiencing this SSL connection problem since to many OpenDistro versions before.
Seems that this never worked?

[dredwilliams]

@chloe-zh - Is there any more information I can provide that will help narrow this down?

[davidcui1225]

We have a release for the SQL Drivers including the ODBC client that is launching this week- that is expected to resolve this bug

[dredwilliams]

@davidcui1225 - if you want me to test, let me know when there is something I can download -- I'd love to get this issue fixed!

[dredwilliams]

@davidcui1225 - How far down the queue is this to work on? I just downloaded the latest artifact and get the same errors.

[joshuali925]

The new ODBC driver links are available on https://opensearch.org/artifacts, you can download it here:

• macos_x64: https://artifacts.opensearch.org/opensearch-clients/odbc/signed_opensearch-sql-odbc-mac-1.1.0.1.zip
• win32: https://artifacts.opensearch.org/opensearch-clients/odbc/signed_opensearch-sql-odbc-win32-1.1.0.1.msi
• win64: https://artifacts.opensearch.org/opensearch-clients/odbc/signed_opensearch-sql-odbc-win64-1.1.0.1.msi

The SSL issue is separate and possibly related to #19