From 348bd0d718f9e9ccdc2d5756b806e69d0117b60e Mon Sep 17 00:00:00 2001 From: Steve Loughran Date: Tue, 14 May 2019 12:51:26 +0100 Subject: [PATCH 1/3] HADOOP-116117 Update AWS SDK to 1.11.549. Contributed by Steve Loughran. Change-Id: Iff125bf3e2f409ae7fd0d05440f4f7ba38c24331 --- NOTICE.txt | 8 ++++---- hadoop-project/pom.xml | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/NOTICE.txt b/NOTICE.txt index b4329b8095a591..37e80d70d40031 100644 --- a/NOTICE.txt +++ b/NOTICE.txt @@ -8,10 +8,10 @@ following notices: * Copyright 2011 FuseSource Corp. http://fusesource.com The binary distribution of this product bundles binaries of -AWS SDK for Java - Bundle 1.11.375, -AWS Java SDK for AWS KMS 1.11.375, -AWS Java SDK for Amazon S3 1.11.375, -AWS Java SDK for AWS STS 1.11.375, +AWS SDK for Java - Bundle 1.11.549, +AWS Java SDK for AWS KMS 1.11.549, +AWS Java SDK for Amazon S3 1.11.549, +AWS Java SDK for AWS STS 1.11.549, JMES Path Query library 1.0, which has the following notices: * This software includes third party software subject to the following diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index f4ec48ac0bf6cb..ae36e8f53f6eeb 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -146,7 +146,7 @@ 1.3.1 1.0-beta-1 900 - 1.11.375 + 1.11.549 2.3.4 1.6 2.1 From 36cbd8bef871cb159a0e65ee06afbfddd36e0a4a Mon Sep 17 00:00:00 2001 From: Steve Loughran Date: Fri, 31 May 2019 11:45:16 +0100 Subject: [PATCH 2/3] HADOOP-16117 Update AWS SDK to 1.11.563 +review and update the AWS SDK notice to be consistent with the artifacts we bundle and the current NOTICE file in the AWS SDK Change-Id: I43057100de9c9e8486099e5326fca3fce7ffd55a --- NOTICE.txt | 15 +++++---------- hadoop-project/pom.xml | 2 +- 2 files changed, 6 insertions(+), 11 deletions(-) diff --git a/NOTICE.txt b/NOTICE.txt index 37e80d70d40031..2cda1556051b4e 100644 --- a/NOTICE.txt +++ b/NOTICE.txt @@ -8,17 +8,12 @@ following notices: * Copyright 2011 FuseSource Corp. http://fusesource.com The binary distribution of this product bundles binaries of -AWS SDK for Java - Bundle 1.11.549, -AWS Java SDK for AWS KMS 1.11.549, -AWS Java SDK for Amazon S3 1.11.549, -AWS Java SDK for AWS STS 1.11.549, -JMES Path Query library 1.0, +AWS SDK for Java - Bundle 1.11.563 (https://github.com/aws/aws-sdk-java), which has the following notices: - * This software includes third party software subject to the following - copyrights: - XML parsing and utility functions from JetS3t - Copyright - 2006-2009 James Murty. - JSON parsing and utility functions from JSON.org - - Copyright 2002 JSON.org. - PKCS#1 PEM encoded private key parsing and utility - functions from oauth.googlecode.com - Copyright 1998-2010 AOL Inc. + +This software includes third party software subject to the following copyrights: +- XML parsing and utility functions from JetS3t - Copyright 2006-2009 James Murty. +- PKCS#1 PEM encoded private key parsing and utility functions from oauth.googlecode.com - Copyright 1998-2010 AOL Inc. The binary distribution of this product bundles binaries of Gson 2.2.4, diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml index ae36e8f53f6eeb..dc8cc8d6baf43d 100644 --- a/hadoop-project/pom.xml +++ b/hadoop-project/pom.xml @@ -146,7 +146,7 @@ 1.3.1 1.0-beta-1 900 - 1.11.549 + 1.11.563 2.3.4 1.6 2.1 From 01e36bb83a34cd50b50d0f60861986ee49f4579b Mon Sep 17 00:00:00 2001 From: Steve Loughran Date: Wed, 5 Jun 2019 23:02:44 +0100 Subject: [PATCH 3/3] HADOOP-16117 Update the testing.md doc with some more commands to run. Change-Id: Ib6305366bfc11fe58996ffd9ee942e5867429818 --- .../hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md index f61b46a8da6ea0..41ef6570ff41a8 100644 --- a/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md +++ b/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md @@ -1329,11 +1329,13 @@ bin/hadoop fs -mkdir $BUCKET/dir-no-trailing # fails with S3Guard bin/hadoop fs -mkdir $BUCKET/dir-trailing/ bin/hadoop fs -touchz $BUCKET/file +echo hello | bin/hadoop fs -cat $BUCKET/file bin/hadoop fs -ls $BUCKET/ bin/hadoop fs -mv $BUCKET/file $BUCKET/file2 # expect "No such file or directory" bin/hadoop fs -stat $BUCKET/file bin/hadoop fs -stat $BUCKET/file2 +bin/hadoop fs -mkdir $BUCKET/dir-no-trailing bin/hadoop fs -mv $BUCKET/file2 $BUCKET/dir-no-trailing bin/hadoop fs -stat $BUCKET/dir-no-trailing/file2 # treated the same as the file stat @@ -1348,6 +1350,9 @@ bin/hadoop fs -test -d $BUCKET/dir-no-trailing/file2 ; echo $? bin/hadoop fs -checksum $BUCKET/dir-no-trailing/file2 # expect "etag" + a long string bin/hadoop fs -D fs.s3a.etag.checksum.enabled=true -checksum $BUCKET/dir-no-trailing/file2 +bin/hadoop fs -expunge -immediate -fs $BUCKET +bin/hdfs fetchdt --webservice $BUCKET secrets.bin +bin/hdfs fetchdt -D fs.s3a.delegation.token.binding=org.apache.hadoop.fs.s3a.auth.delegation.SessionTokenBinding --webservice $BUCKET secrets.bin ``` ### Other tests