Skip to content

Conversation

@arteam
Copy link
Contributor

@arteam arteam commented Jan 18, 2022

The change adds 3 new client properties for the GCS repository:

  • gcs.client.default.proxy.type
  • gcs.client.default.proxy.host
  • gcs.client.default.proxy.port

They allow to configure a java.net.Proxy for the GCS SDK to use when communicating with the GCS API.

Resolves #82444

@arteam arteam added >enhancement :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v8.0.0 auto-backport Automatically create backport pull requests when merged labels Jan 18, 2022
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Jan 18, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

The change adds 3 new client properties for the GCS repository:

* gcs.client.default.proxy.type
* gcs.client.default.proxy.host
* gcs.client.default.proxy.port

They allow to configure a [java.net.Proxy](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/net/Proxy.html)
for the GCS SDK to use when communicating with the GCS API.

Resolves elastic#82444
@arteam
Copy link
Contributor Author

arteam commented Jan 20, 2022

@elasticmachine update branch

@arteam arteam requested a review from tlrx January 20, 2022 11:36
Copy link
Member

@tlrx tlrx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@arteam arteam merged commit 1ddaf25 into elastic:master Jan 20, 2022
@arteam
Copy link
Contributor Author

arteam commented Jan 20, 2022

Thanks Tanguy!

arteam added a commit to arteam/elasticsearch that referenced this pull request Jan 20, 2022
* Add support for HTTP Proxies for the GCS repository

The change adds 3 new client properties for the GCS repository:

* gcs.client.default.proxy.type
* gcs.client.default.proxy.host
* gcs.client.default.proxy.port

They allow to configure a [java.net.Proxy](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/net/Proxy.html)
for the GCS SDK to use when communicating with the GCS API.

Resolves elastic#82444
@elasticsearchmachine
Copy link
Collaborator

💚 Backport successful

Status Branch Result
8.0

@arteam arteam deleted the gcs-proxy branch January 20, 2022 14:53
arteam added a commit that referenced this pull request Jan 20, 2022
The change adds 3 new client properties for the GCS repository:

* gcs.client.default.proxy.type
* gcs.client.default.proxy.host
* gcs.client.default.proxy.port

They allow to configure a [java.net.Proxy](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/net/Proxy.html)
for the GCS SDK to use when communicating with the GCS API.

Resolves #82444
@sangeethdba
Copy link

does these settings work with elastic 6.8.2 or 7.x

@sangeethdba
Copy link

path: /_snapshot/backup, params: {pretty=, repository=backup}
org.elasticsearch.repositories.RepositoryException: [backup] cannot create blob store
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:328) ~[elasticsearch-6.8.2.jar:6.8.2]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:625) ~[elasticsearch-6.8.2.jar:6.8.2]
at org.elasticsearch.repositories.RepositoriesService.lambda$verifyRepository$2(RepositoriesService.java:230) ~[elasticsearch-6.8.2.jar:6.8.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-6.8.2.jar:6.8.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
Caused by: org.elasticsearch.common.blobstore.BlobStoreException: Unable to check if bucket [com-dbre-backup-np] exists
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:109) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.(GoogleCloudStorageBlobStore.java:75) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:137) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:47) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:324) ~[elasticsearch-6.8.2.jar:6.8.2]
... 6 more
Caused by: com.google.cloud.storage.StorageException: Error getting access token for service account:
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:226) ~[?:?]
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:403) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:196) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:193) ~[?:?]
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89) ~[?:?]
at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:192) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$doesBucketExist$0(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_322]
at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.(GoogleCloudStorageBlobStore.java:75) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:137) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:47) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:324) ~[elasticsearch-6.8.2.jar:6.8.2]
... 6 more
Caused by: java.io.IOException: Error getting access token for service account:
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:395) ~[?:?]
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:179) ~[?:?]
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:165) ~[?:?]
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96) ~[?:?]
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159) ~[?:?]
at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109) ~[?:?]
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469) ~[?:?]
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:400) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:196) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:193) ~[?:?]
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89) ~[?:?]
at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:192) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$doesBucketExist$0(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_322]
at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.(GoogleCloudStorageBlobStore.java:75) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:137) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:47) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:324) ~[elasticsearch-6.8.2.jar:6.8.2]
... 6 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_322]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_322]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_322]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_322]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_322]
at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_322]
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:288) ~[?:?]
at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:?]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:264) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:203) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:189) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1342) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1317) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264) ~[?:?]
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77) ~[?:?]
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981) ~[?:?]
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:393) ~[?:?]
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:179) ~[?:?]
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:165) ~[?:?]
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96) ~[?:?]
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159) ~[?:?]
at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:109) ~[?:?]
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352) ~[?:?]
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469) ~[?:?]
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:400) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:196) ~[?:?]
at com.google.cloud.storage.StorageImpl$4.call(StorageImpl.java:193) ~[?:?]
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89) ~[?:?]
at com.google.cloud.RetryHelper.run(RetryHelper.java:76) ~[?:?]
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50) ~[?:?]
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:192) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.lambda$doesBucketExist$0(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_322]
at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.doesBucketExist(GoogleCloudStorageBlobStore.java:106) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.(GoogleCloudStorageBlobStore.java:75) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:137) ~[?:?]
at org.elasticsearch.repositories.gcs.GoogleCloudStorageRepository.createBlobStore(GoogleCloudStorageRepository.java:47) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:324) ~[elasticsearch-6.8.2.jar:6.8.2]

@sangeethdba
Copy link

@tlrx @arteam

elasticsearchmachine pushed a commit that referenced this pull request Feb 15, 2022
Re-adds HTTP proxy settings for the GCS repository type. These settings
were added with #82737.

The docs were accidentally removed as part of
#82996.
elasticsearchmachine pushed a commit that referenced this pull request Feb 15, 2022
Re-adds HTTP proxy settings for the GCS repository type. These settings
were added with #82737.

The docs were accidentally removed as part of
#82996.

(cherry picked from commit e8b34c7)
elasticsearchmachine pushed a commit that referenced this pull request Feb 15, 2022
Re-adds HTTP proxy settings for the GCS repository type. These settings
were added with #82737.

The docs were accidentally removed as part of
#82996.

(cherry picked from commit e8b34c7)
jrodewig added a commit that referenced this pull request Feb 16, 2022
We've had a few users ask about support for HTTP proxy connections to GCS snapshot repositories in 7.x and 6.x.
However, client settings for those connections are only supported in 8.0+.

This adds a note to the 7.17 and 6.8 docs to nudge users to upgrade.

Relates to #82737

Closes #83959
elasticsearchmachine pushed a commit that referenced this pull request Feb 16, 2022
)

We've had a few users ask about support for HTTP proxy connections to GCS snapshot repositories in 7.x and 6.x.
However, client settings for those connections are only supported in 8.0+.

This adds a note to the 7.17 and 6.8 docs to nudge users to upgrade.

Relates to #82737

Closes #83959

(cherry picked from commit 99de635)
weizijun added a commit to weizijun/elasticsearch that referenced this pull request Feb 16, 2022
…ijun/elasticsearch into fix-none-tsdb-index-dimension-tests

* 'fix-none-tsdb-index-dimension-tests' of github.com:weizijun/elasticsearch: (37 commits)
  [docs] Mention JDK 17 in the Contributing docs (elastic#84018)
  Fix GeoIpDownloader startup during rolling upgrade (elastic#84000)
  Script: Fields API for Dense Vector (elastic#83550)
  Move InferenceConfigUpdate under VersionedNamedWriteable (elastic#84022)
  [ML] Fix license feature test cleanup (elastic#84020)
  Replace deprecated api in artifact transforms (elastic#84015)
  QL: Add leniency option to SQL CLI (elastic#83795)
  [Stack Monitoring] add kibana_stats version alias to -mb template (elastic#83930)
  Optimize spliterator for ImmutableOpenMap (elastic#83899)
  Feature usage actions for archive (elastic#83931)
  Use latch to speedup multi feature migration test (elastic#84007)
  Make action names available in NodeClient (elastic#83919)
  [DOCS] Re-add HTTP proxy setings from elastic#82737 (elastic#84001)
  Add CI matrix configuration for snapshot BWC versions (elastic#83990)
  Update YAML Rest tests to check for product header on all responses (elastic#83290)
  TSDB: Add time series aggs cancellation (elastic#83492)
  [DOCS] Fix percolate query headings (elastic#83988)
  [DOCS] Move tip for percolate query example (elastic#83972)
  Simplify LocalExporter cleaner function to fix failing tests (elastic#83812)
  [GCE Discovery] Correcly handle large zones with 500 or more instances (elastic#83785)
  ...
arteam added a commit to arteam/elasticsearch that referenced this pull request Apr 11, 2022
They were added in elastic#82737, but not exposed via the GCSPlugin to end users.
arteam added a commit that referenced this pull request Apr 12, 2022
They were added in #82737, but not exposed via the GCSPlugin to end users.
arteam added a commit to arteam/elasticsearch that referenced this pull request Apr 12, 2022
They were added in elastic#82737, but not exposed via the GCSPlugin to end users.
arteam added a commit to arteam/elasticsearch that referenced this pull request Apr 12, 2022
They were added in elastic#82737, but not exposed via the GCSPlugin to end users.
arteam added a commit to arteam/elasticsearch that referenced this pull request Apr 12, 2022
They were added in elastic#82737, but not exposed via the GCSPlugin to end users.
arteam added a commit that referenced this pull request Apr 12, 2022
They were added in #82737, but not exposed via the GCSPlugin to end users.
arteam added a commit that referenced this pull request Apr 12, 2022
They were added in #82737, but not exposed via the GCSPlugin to end users.
arteam added a commit that referenced this pull request Apr 12, 2022
They were added in #82737, but not exposed via the GCSPlugin to end users.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-backport Automatically create backport pull requests when merged :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >enhancement Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. v8.0.0 v8.1.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for HTTP Proxies for the Google Cloud Storage repository plugin

5 participants