SQL: add support for API key to JDBC and CLI

docs/changelog/142021.yaml

@@ -0,0 +1,5 @@
+area: SQL
+issues: []
+pr: 142021
+summary: Add support for API key to JDBC and CLI
+type: enhancement

docs/reference/query-languages/sql/sql-cli.md

@@ -28,6 +28,22 @@ If security is enabled on your cluster, you can pass the username and password i
 $ ./bin/elasticsearch-sql-cli https://sql_user:strongpassword@some.server:9200
 ```
 
+### API Key Authentication [sql-cli-apikey]
+
+As an alternative to basic authentication, you can use API key authentication with the `--apikey` option. API keys can be created using the [Create API key API](docs-content://deploy-manage/api-keys/elasticsearch-api-keys.md). The API key should be provided in its encoded form (the `encoded` value returned by the Create API key API):
+
+```bash
+$ ./bin/elasticsearch-sql-cli --apikey <encoded-api-key> https://some.server:9200
+```
+
+::::{note}
+When using API key authentication, do not include username and password in the URL. The CLI will return an error if both API key and basic authentication credentials are provided.
+::::
+
+::::{warning}
+Command line arguments are visible to other users on the system through process listing commands like `ps aux` or by inspecting `/proc/<pid>/cmdline`. Avoid using this method on shared systems where other users might be able to view your credentials.
+::::
+
 Once the CLI is running you can use any [query](elasticsearch://reference/query-languages/sql/sql-spec.md) that Elasticsearch supports:
 
 ```sql

docs/reference/query-languages/sql/sql-jdbc.md

@@ -120,6 +120,24 @@ $$$jdbc-cfg-timezone$$$
 :   Basic Authentication password
 
 
+### API Key Authentication [jdbc-cfg-auth-apikey]
+
+As an alternative to basic authentication, you can use API key authentication. API keys can be created using the [Create API key API](docs-content://deploy-manage/api-keys/elasticsearch-api-keys.md). The API key should be provided in its encoded form (the `encoded` value returned by the Create API key API).
+
+`apiKey`
+:   Encoded API key for authentication. Cannot be used together with `user`/`password` basic authentication.
+
+::::{note}
+When using API key authentication, do not specify `user` or `password`. The driver will return an error if both API key and basic authentication credentials are provided.
+::::
+
+Example connection URL using API key:
+
+```text
+jdbc:es://http://server:9200/?apiKey=<encoded-api-key>
+```
+
+
 ### SSL [jdbc-cfg-ssl]
 
 `ssl` (default `false`)

docs/reference/query-languages/sql/sql-security.md

@@ -20,11 +20,14 @@ In case of an encrypted transport, the SSL/TLS support needs to be enabled in El
 
 ## Authentication [_authentication]
 
-The authentication support in {{es}} SQL is of two types:
+The authentication support in {{es}} SQL is of three types:
 
 Username/Password
 :   Set these through `user` and `password` properties.
 
+API Key
+:   Use an API key for authentication by setting the `apiKey` property. API keys can be created using the [Create API key API](docs-content://deploy-manage/api-keys/elasticsearch-api-keys.md). The API key should be provided in its encoded form (the `encoded` value returned by the Create API key API). This is an alternative to username/password authentication and cannot be used together with it. For the CLI, use the `--apikey` command line option.
+
 PKI/X.509
 :   Use X.509 certificates to authenticate {{es}} SQL to {{es}}. For this, one would need to setup the `keystore` containing the private key and certificate to the appropriate user (configured in {{es}}) and the `truststore` with the CA certificate used to sign the SSL/TLS certificates in the {{es}} cluster. That is, one should setup the key to authenticate {{es}} SQL and also to verify that is the right one. To do so, one should set the `ssl.keystore.location` and `ssl.truststore.location` properties to indicate the `keystore` and `truststore` to use. It is recommended to have these secured through a password in which case `ssl.keystore.pass` and `ssl.truststore.pass` properties are required.
 

libs/cli/src/main/java/org/elasticsearch/cli/Command.java

@@ -89,13 +89,19 @@ protected void mainWithoutErrorHandling(String[] args, Terminal terminal, Proces
         LoggerFactory loggerFactory = LoggerFactory.provider();
         if (options.has(silentOption)) {
             terminal.setVerbosity(Terminal.Verbosity.SILENT);
-            loggerFactory.setRootLevel(Level.OFF);
+            if (loggerFactory != null) {
+                loggerFactory.setRootLevel(Level.OFF);
+            }
         } else if (options.has(verboseOption)) {
             terminal.setVerbosity(Terminal.Verbosity.VERBOSE);
-            loggerFactory.setRootLevel(Level.DEBUG);
+            if (loggerFactory != null) {
+                loggerFactory.setRootLevel(Level.DEBUG);
+            }
         } else {
             terminal.setVerbosity(Terminal.Verbosity.NORMAL);
-            loggerFactory.setRootLevel(Level.INFO);
+            if (loggerFactory != null) {
+                loggerFactory.setRootLevel(Level.INFO);
+            }
         }
 
         execute(terminal, options, processInfo);

x-pack/plugin/sql/jdbc/src/test/java/org/elasticsearch/xpack/sql/jdbc/JdbcConfigurationTests.java

@@ -25,6 +25,8 @@
 import java.util.Properties;
 import java.util.stream.Collectors;
 
+import static org.elasticsearch.xpack.sql.client.ConnectionConfiguration.AUTH_API_KEY;
+import static org.elasticsearch.xpack.sql.client.ConnectionConfiguration.AUTH_USER;
 import static org.elasticsearch.xpack.sql.client.ConnectionConfiguration.CONNECT_TIMEOUT;
 import static org.elasticsearch.xpack.sql.client.ConnectionConfiguration.NETWORK_TIMEOUT;
 import static org.elasticsearch.xpack.sql.client.ConnectionConfiguration.PAGE_SIZE;
@@ -33,6 +35,7 @@
 import static org.elasticsearch.xpack.sql.client.ConnectionConfiguration.QUERY_TIMEOUT;
 import static org.elasticsearch.xpack.sql.jdbc.JdbcConfiguration.URL_FULL_PREFIX;
 import static org.elasticsearch.xpack.sql.jdbc.JdbcConfiguration.URL_PREFIX;
+import static org.hamcrest.Matchers.containsString;
 import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.is;
 
@@ -399,4 +402,40 @@ private void assertJdbcSqlException(String wrongSetting, String correctSetting,
         JdbcSQLException ex = expectThrows(JdbcSQLException.class, () -> JdbcConfiguration.create(url, props, 0));
         assertEquals("Unknown parameter [" + wrongSetting + "]; did you mean [" + correctSetting + "]", ex.getMessage());
     }
+
+    public void testApiKeyInUrl() throws Exception {
+        String apiKey = "test_api_key_encoded";
+        JdbcConfiguration ci = ci(jdbcPrefix() + "test:9200?apiKey=" + apiKey);
+        assertThat(ci.apiKey(), is(apiKey));
+        assertNull(ci.authUser());
+        assertNull(ci.authPass());
+    }
+
+    public void testApiKeyInProperties() throws Exception {
+        String apiKey = "test_api_key_encoded";
+        Properties props = new Properties();
+        props.setProperty(AUTH_API_KEY, apiKey);
+        JdbcConfiguration ci = JdbcConfiguration.create(jdbcPrefix() + "test:9200", props, 0);
+        assertThat(ci.apiKey(), is(apiKey));
+        assertNull(ci.authUser());
+        assertNull(ci.authPass());
+    }
+
+    public void testApiKeyAndUserMutuallyExclusive() {
+        String apiKey = "test_api_key_encoded";
+        Properties props = new Properties();
+        props.setProperty(AUTH_API_KEY, apiKey);
+        props.setProperty(AUTH_USER, "user");
+        JdbcSQLException ex = expectThrows(JdbcSQLException.class, () -> JdbcConfiguration.create(jdbcPrefix() + "test:9200", props, 0));
+        assertThat(ex.getMessage(), containsString("Cannot use both API key and basic authentication"));
+    }
+
+    public void testApiKeyAndUserInUrlMutuallyExclusive() {
+        String apiKey = "test_api_key_encoded";
+        JdbcSQLException ex = expectThrows(
+            JdbcSQLException.class,
+            () -> ci(jdbcPrefix() + "test:9200?apiKey=" + apiKey + "&user=testuser")
+        );
+        assertThat(ex.getMessage(), containsString("Cannot use both API key and basic authentication"));
+    }
 }

x-pack/plugin/sql/qa/server/security/src/test/java/org/elasticsearch/xpack/sql/qa/security/CliApiKeyIT.java

@@ -0,0 +1,150 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+package org.elasticsearch.xpack.sql.qa.security;
+
+import org.elasticsearch.client.Request;
+import org.elasticsearch.xpack.sql.qa.cli.EmbeddedCli;
+import org.elasticsearch.xpack.sql.qa.cli.EmbeddedCli.ApiKeySecurityConfig;
+
+import static org.elasticsearch.xpack.sql.qa.security.RestSqlIT.SSL_ENABLED;
+import static org.hamcrest.Matchers.containsString;
+
+/**
+ * Integration tests for CLI connections using API key authentication.
+ */
+public class CliApiKeyIT extends SqlApiKeyTestCase {
+
+    public void testCliConnectionWithApiKey() throws Exception {
+        String encodedApiKey = createApiKey("""
+            {
+                "name": "cli_test_key",
+                "role_descriptors": {
+                    "role": {
+                        "cluster": ["monitor"],
+                        "indices": [
+                            {
+                                "names": ["*"],
+                                "privileges": ["all"]
+                            }
+                        ]
+                    }
+                }
+            }
+            """);
+
+        Request createIndex = new Request("PUT", "/test_cli_api_key");
+        createIndex.setJsonEntity("""
+            {
+                "mappings": {
+                    "properties": {
+                        "value": { "type": "integer" }
+                    }
+                }
+            }
+            """);
+        client().performRequest(createIndex);
+
+        Request indexDoc = new Request("PUT", "/test_cli_api_key/_doc/1");
+        indexDoc.addParameter("refresh", "true");
+        indexDoc.setJsonEntity("""
+            {
+                "value": 123
+            }
+            """);
+        client().performRequest(indexDoc);
+
+        ApiKeySecurityConfig apiKeyConfig = createApiKeySecurityConfig(encodedApiKey);
+
+        try (EmbeddedCli cli = new EmbeddedCli(elasticsearchAddress(), true, apiKeyConfig)) {
+            String result = cli.command("SELECT value FROM test_cli_api_key");
+            assertThat(result, containsString("value"));
+            cli.readLine(); // separator line
+            String valueLine = cli.readLine();
+            assertThat(valueLine, containsString("123"));
+        }
+    }
+
+    public void testCliConnectionWithInvalidApiKey() throws Exception {
+        ApiKeySecurityConfig apiKeyConfig = new ApiKeySecurityConfig(
+            SSL_ENABLED,
+            "invalid_api_key_value",
+            SSL_ENABLED ? SqlSecurityTestCluster.getKeystorePath() : null,
+            SSL_ENABLED ? SqlSecurityTestCluster.KEYSTORE_PASSWORD : null
+        );
+
+        try (EmbeddedCli cli = new EmbeddedCli(elasticsearchAddress(), false, apiKeyConfig)) {
+            String result = cli.command("SELECT 1");
+            StringBuilder fullError = new StringBuilder(result);
+            String line;
+            while ((line = cli.readLine()) != null && !line.isEmpty()) {
+                fullError.append(line);
+            }
+            String errorMessage = fullError.toString();
+            assertTrue(
+                "Expected authentication error but got: " + errorMessage,
+                errorMessage.contains("security_exception") || errorMessage.contains("Communication error")
+            );
+        }
+    }
+
+    public void testCliConnectionWithLimitedApiKey() throws Exception {
+        Request createRestrictedIndex = new Request("PUT", "/cli_restricted_index");
+        createRestrictedIndex.setJsonEntity("""
+            {
+                "mappings": {
+                    "properties": {
+                        "secret": { "type": "keyword" }
+                    }
+                }
+            }
+            """);
+        client().performRequest(createRestrictedIndex);
+
+        Request indexRestrictedDoc = new Request("PUT", "/cli_restricted_index/_doc/1");
+        indexRestrictedDoc.addParameter("refresh", "true");
+        indexRestrictedDoc.setJsonEntity("""
+            {
+                "secret": "confidential"
+            }
+            """);
+        client().performRequest(indexRestrictedDoc);
+
+        String encodedApiKey = createApiKey("""
+            {
+                "name": "cli_limited_key",
+                "role_descriptors": {
+                    "role": {
+                        "cluster": ["monitor"],
+                        "indices": [
+                            {
+                                "names": ["cli_allowed_*"],
+                                "privileges": ["read"]
+                            }
+                        ]
+                    }
+                }
+            }
+            """);
+
+        ApiKeySecurityConfig apiKeyConfig = createApiKeySecurityConfig(encodedApiKey);
+
+        try (EmbeddedCli cli = new EmbeddedCli(elasticsearchAddress(), true, apiKeyConfig)) {
+            String result = cli.command("SELECT * FROM cli_restricted_index");
+            String errorLine = cli.readLine();
+            assertThat(errorLine, containsString("Unknown index [cli_restricted_index]"));
+        }
+    }
+
+    private ApiKeySecurityConfig createApiKeySecurityConfig(String apiKey) {
+        return new ApiKeySecurityConfig(
+            SSL_ENABLED,
+            apiKey,
+            SSL_ENABLED ? SqlSecurityTestCluster.getKeystorePath() : null,
+            SSL_ENABLED ? SqlSecurityTestCluster.KEYSTORE_PASSWORD : null
+        );
+    }
+}