Skip to content

HDDS-3806. Support recognize aws v2 Authorization header. #1098

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ChenSammi merged 4 commits into from
Jul 20, 2020
Merged

HDDS-3806. Support recognize aws v2 Authorization header. #1098

ChenSammi merged 4 commits into from
Jul 20, 2020

Conversation

@ChenSammi
Copy link
Contributor

@ChenSammi ChenSammi commented Jun 19, 2020

@ChenSammi ChenSammi requested a review from bharatviswa504 June 22, 2020 07:41
@elek
Copy link
Member

elek commented Jun 24, 2020

Thanks to add this support @ChenSammi

Can you please share some details about the testing? Do you have any recommendation for the client? Couldn't see any option in aws cli but ready to install any other (open source ;-)) tools to test it and merge.

@ChenSammi
Copy link
Contributor Author

ChenSammi commented Jun 29, 2020

Thanks to add this support @ChenSammi

Can you please share some details about the testing? Do you have any recommendation for the client? Couldn't see any option in aws cli but ready to install any other (open source ;-)) tools to test it and merge.

Marton, aws sdk 1.10.* by default using V2 authrization header, and 1.11.* by default using V4 authrization header. You can reproduce the issue with a 1.10.* aws client.

elek
elek approved these changes Jul 16, 2020
View reviewed changes
Copy link
Member

@elek elek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks the information @ChenSammi

I tested it and worked well.

  1. I started a docker based Ozone cluster with and without the patch

  2. I modified the ozone freon s3kg to support 1.10 SDK:

a) changed the aws sdk version in main pom to 1.10.77

b) created AwsClient without using the builder:

      s3 = new AmazonS3Client();

      s3.setEndpoint(endpoint);
      s3.setS3ClientOptions(
          S3ClientOptions.builder()
              .setPathStyleAccess(true).build());

With the patch, it worked well. Without the patch I got exeption:

s3g_1       | org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke public void org.apache.hadoop.ozone.s3.AWSV4SignatureProcessor.init() throws java.lang.Exception on org.apache.hadoop.ozone.s3.AWSV4SignatureProcessor@2610a3c6

@github-actions
Copy link

github-actions bot commented Jul 20, 2020

To re-run CI checks, please follow these steps with the source branch checked out:

git commit --allow-empty -m 'trigger new CI check'
git push

@ChenSammi ChenSammi merged commit c64d226 into apache:master Jul 20, 2020
@ChenSammi
Copy link
Contributor Author

ChenSammi commented Jul 20, 2020

Thanks @elek for the review.

errose28 added a commit to errose28/ozone that referenced this pull request Jul 20, 2020
@errose28
Merge branch 'master' into HDDS-3869
af8b4d4 
* master:
  HDDS-3984. Support filter and search the columns in recon UI (apache#1218)
  HDDS-3806. Support recognize aws v2 Authorization header. (apache#1098)
  HDDS-3955. Unable to list intermediate paths on keys created using S3G. (apache#1196)
  HDDS-3741. Reload old OM state if Install Snapshot from Leader fails (apache#1129)
  HDDS-3965. SCM failed to start up for duplicated pipeline detected. (apache#1210)
  HDDS-3855. Add upgrade smoketest (apache#1142)
  HDDS-3964. Ratis config key mismatch (apache#1204)
  HDDS-3612. Allow mounting bucket under other volume (apache#1104)
  HDDS-3926. OM Token Identifier table should use in-house serialization. (apache#1182)
  HDDS-3824: OM read requests should make SCM#refreshPipeline outside BUCKET_LOCK (apache#1164)
  HDDS-3966. Disable flaky TestOMRatisSnapshots
errose28 added a commit to errose28/ozone that referenced this pull request Jul 20, 2020
@errose28
Merge branch 'HDDS-3869' into switch-to-new-interface
cf6e0c4 
* HDDS-3869:
  HDDS-3984. Support filter and search the columns in recon UI (apache#1218)
  HDDS-3806. Support recognize aws v2 Authorization header. (apache#1098)
  HDDS-3955. Unable to list intermediate paths on keys created using S3G. (apache#1196)
  HDDS-3741. Reload old OM state if Install Snapshot from Leader fails (apache#1129)
  HDDS-3965. SCM failed to start up for duplicated pipeline detected. (apache#1210)
errose28 added a commit to errose28/ozone that referenced this pull request Jul 20, 2020
@errose28
Merge branch 'master' into HDDS-3976
18f6a18 
* master:
  HDDS-3984. Support filter and search the columns in recon UI (apache#1218)
  HDDS-3806. Support recognize aws v2 Authorization header. (apache#1098)
  HDDS-3955. Unable to list intermediate paths on keys created using S3G. (apache#1196)
  HDDS-3741. Reload old OM state if Install Snapshot from Leader fails (apache#1129)
  HDDS-3965. SCM failed to start up for duplicated pipeline detected. (apache#1210)
errose28 pushed a commit to errose28/ozone that referenced this pull request Jul 21, 2020
@ChenSammi @errose28
HDDS-3806. Support recognize aws v2 Authorization header. (apache#1098)
81cdb94
errose28 added a commit to errose28/ozone that referenced this pull request Jul 21, 2020
@errose28
Merge branch 'add-deleted-block-table' into HDDS-3869
2fa068b 
* add-deleted-block-table: (63 commits)
  Make block iterator tests use deleted blocks table, and remove the now unused #deleted#
  Replace uses of #deleted# key prefix with access to new deleted blocks table
  Add deleted blocks table to base level DB wrappers
  Have block deleting service test look for #deleted# keys in metadata table
  Move block delete to correct table and remove debugging print statement
  Import schema version when importing container data from export
  HDDS-3984. Support filter and search the columns in recon UI (apache#1218)
  HDDS-3806. Support recognize aws v2 Authorization header. (apache#1098)
  HDDS-3955. Unable to list intermediate paths on keys created using S3G. (apache#1196)
  HDDS-3741. Reload old OM state if Install Snapshot from Leader fails (apache#1129)
  Move new key value block iterator implementation and tests to new interface
  Fix checkstyle violations
  HDDS-3965. SCM failed to start up for duplicated pipeline detected. (apache#1210)
  Update comments
  Add comments on added helper method
  Remove seekToLast() from iterator interface, implementation, and tests
  Add more robust unit test with alternating key matches
  All unit tests pass after allowing keys with deleted and deleting prefixes to be made
  HDDS-3855. Add upgrade smoketest (apache#1142)
  HDDS-3964. Ratis config key mismatch (apache#1204)
  ...
ChenSammi added a commit that referenced this pull request Jul 22, 2020
@ChenSammi
HDDS-3806. Support recognize aws v2 Authorization header. (#1098)
8b346ed 
(cherry picked from commit c64d226)
timmylicheng pushed a commit that referenced this pull request Aug 6, 2020
@ChenSammi @timmylicheng
HDDS-3806. Support recognize aws v2 Authorization header. (#1098)
5d6f832
rakeshadr pushed a commit to rakeshadr/hadoop-ozone that referenced this pull request Sep 3, 2020
@ChenSammi @rakeshadr
HDDS-3806. Support recognize aws v2 Authorization header. (apache#1098)
0305a77
@elek elek mentioned this pull request Mar 30, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@elek elek elek approved these changes

@bharatviswa504 bharatviswa504 Awaiting requested review from bharatviswa504

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ChenSammi @elek