Support per-project client for GCS repository by ywangd · Pull Request #131899 · elastic/elasticsearch

ywangd · 2025-07-25T04:35:53Z

Similar to #127631, this PR adds per project client support for GCS repository.

…ect-clients

elasticsearchmachine · 2025-07-29T00:01:38Z

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

ywangd · 2025-07-29T00:04:43Z

...y-gcs/src/main/java/org/elasticsearch/repositories/gcs/GoogleCloudStorageClientsManager.java

+        for (var projectId : perProjectClientsHolders.keySet()) {
+            if (currentProjects.containsKey(projectId) == false) {
+                assert ProjectId.DEFAULT.equals(projectId) == false;
+                perProjectClientsHolders.remove(projectId);


The approach for managing per-project clients is similar to what we did for s3 but simpler. It is simpler because there is no ref-counting for the clients and we don't actively close it when it is not needed. Therefore, we simply remove and replace it when needed.

ywangd · 2025-07-29T00:05:27Z

...y-gcs/src/main/java/org/elasticsearch/repositories/gcs/GoogleCloudStorageClientsManager.java

+    void refreshAndClearCacheForClusterClients(Map<String, GoogleCloudStorageClientSettings> clientsSettings) {
+        clusterClientsHolder.refreshAndClearCache(clientsSettings);
+    }


This is used for stateful or cluster level clients to keep the behaviour identical to existing code.

pxsalehi · 2025-07-30T09:28:03Z

...pository-gcs/src/main/java/org/elasticsearch/repositories/gcs/GoogleCloudStorageService.java

-
-    public GoogleCloudStorageService(boolean isServerless) {
-        this.isServerless = isServerless;
+    @SuppressWarnings("this-escape")


I think we should try to avoid this. You can add a static method to create the instance and then create the client manager.

This is added because we pass GoogleCloudStorageService#createClient to the clients manager. It's hard to make this method static because it uses other methods that are overridden in tests. Not entirely sure whether this is your suggestion?

Alternatively, I can move the clientsManager inside GoogleCloudStorageService and this problem should go away since they are in one file. What do you think?

Moving the client manager inside GoogleCloudStorageService seems like a good solution!

Pushed ab073f8

pxsalehi · 2025-07-30T09:50:08Z

...y-gcs/src/main/java/org/elasticsearch/repositories/gcs/GoogleCloudStorageClientsManager.java

+            clientCache = clientCache.entrySet()
+                .stream()
+                .filter(entry -> entry.getKey().equals(repositoryName) == false)
+                .collect(Collectors.toUnmodifiableMap(Map.Entry::getKey, java.util.Map.Entry::getValue));


you can probably drop the java.util.

pxsalehi · 2025-07-30T12:14:52Z

...y-gcs/src/main/java/org/elasticsearch/repositories/gcs/GoogleCloudStorageClientsManager.java

+                continue;
+            }
+            final ProjectSecrets projectSecrets = project.custom(ProjectSecrets.TYPE);
+            // Project secrets can be null when node restarts. It may not have any s3 credentials if s3 is not in use.


Copy/pasted code 🤦 pushed fbdd313

pxsalehi · 2025-07-30T12:20:13Z

...y-gcs/src/main/java/org/elasticsearch/repositories/gcs/GoogleCloudStorageClientsManager.java

+                // Skip project clients that have no credentials configured. This should not happen in serverless.
+                // But it is safer to skip them and is also a more consistent behaviour with the cases when
+                // project secrets are not present.


is this supposed to be a transient state? Does it worth a warn log?

This will be a configuration problem on the CP side. Yes, we should log it. There is one for empty clients settings map after this. But I think we can combine them and continue the processing. Pushed 57de151 which also adds a test. I'll update it similarly for s3 in a separate PR.

…ect-clients

pxsalehi

LGTM, pending getting rid of the new @SuppressWarnings("this-escape").

…ect-clients

Log a message only when the client settings have actual changes. Relates: elastic#131899

Log a message only when the client settings have actual changes. Relates: #131899

This PR adds per project client support for Azure repository similar to #131899 (GCS) and #127631 (s3). Resolves: ES-11714

Add IT for GCS per-project clients added in elastic#131899

Create GCSService in createComponents

9b3c605

elasticsearchmachine added the v9.2.0 label Jul 25, 2025

ywangd added 11 commits July 25, 2025 14:57

more wiring changes

ff000bd

even more wiring

cca38e4

Move existing client management code across

c6c5f9f

all wired up

cb8f02f

fix assertion

ea91764

separate cluster and project holders

cba172b

add tests

6cd68ca

remove deprecated methods

d23285f

enhance GCS TestUtils

4127e57

tweak

3c03efb

Merge remote-tracking branch 'origin/main' into ES-11715-gcs-per-proj…

0006459

…ect-clients

ywangd added >non-issue :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs labels Jul 28, 2025

ywangd requested review from DaveCTurner and pxsalehi July 28, 2025 23:56

ywangd marked this pull request as ready for review July 29, 2025 00:01

elasticsearchmachine added the Team:Distributed Coordination (obsolete) Meta label for Distributed Coordination team. Obsolete. Please do not use. label Jul 29, 2025

elasticsearchmachine added the serverless-linked Added by automation, don't add manually label Jul 29, 2025

ywangd commented Jul 29, 2025

View reviewed changes

equals

e4957cb

pxsalehi reviewed Jul 30, 2025

View reviewed changes

ywangd added 4 commits July 31, 2025 11:34

drop java.util

ea420be

replace with GCS

fbdd313

better handling for clients with incorrect configured secrets

57de151

Merge remote-tracking branch 'origin/main' into ES-11715-gcs-per-proj…

54a7b38

…ect-clients

ywangd requested a review from pxsalehi July 31, 2025 02:30

pxsalehi approved these changes Jul 31, 2025

View reviewed changes

ywangd added 2 commits August 4, 2025 11:50

inline GoogleCloudStorageClientsManager

ab073f8

Merge remote-tracking branch 'origin/main' into ES-11715-gcs-per-proj…

acb5d05

…ect-clients

ywangd added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Aug 4, 2025

elasticsearchmachine merged commit 2559e28 into elastic:main Aug 4, 2025
33 checks passed

ywangd deleted the ES-11715-gcs-per-project-clients branch August 4, 2025 02:55

ywangd added a commit to ywangd/elasticsearch that referenced this pull request Aug 5, 2025

Reduce logging frequency for GCS per project clients

e966efe

Log a message only when the client settings have actual changes. Relates: elastic#131899

ywangd mentioned this pull request Aug 5, 2025

Reduce logging frequency for GCS per project clients #132429

Merged

elasticsearchmachine pushed a commit that referenced this pull request Aug 5, 2025

Reduce logging frequency for GCS per project clients (#132429)

a11556b

Log a message only when the client settings have actual changes. Relates: #131899

ywangd mentioned this pull request Aug 12, 2025

Support per-project client for Azure repository #132701

Merged

elasticsearchmachine pushed a commit that referenced this pull request Aug 13, 2025

Support per-project client for Azure repository (#132701)

19b61ef

This PR adds per project client support for Azure repository similar to #131899 (GCS) and #127631 (s3). Resolves: ES-11714

ywangd mentioned this pull request Aug 14, 2025

POC: Add support for per project repo client #126584

Closed

breskeby pushed a commit to breskeby/elasticsearch that referenced this pull request Feb 11, 2026

Test for per-project clients of GCS repository (elastic#4282)

03054a4

Add IT for GCS per-project clients added in elastic#131899

breskeby pushed a commit to breskeby/elasticsearch that referenced this pull request Feb 11, 2026

Test for per-project clients of GCS repository (elastic#4282)

7eaf7b1

Add IT for GCS per-project clients added in elastic#131899

Conversation

ywangd commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pxsalehi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

ywangd commented Jul 25, 2025 •

edited

Loading