HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs #2845

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

steveloughran merged 18 commits into apache:trunk from sumangala17:HADOOP-17618

Aug 4, 2021

Contributor

sumangala-patki commented Mar 31, 2021 •

edited

Loading

Delegation SAS tokens are created using various parameters for specifying details such as permissions and validity. The requests are logged, along with values of all the query parameters. This change will partially mask values logged for the following object IDs representing the security principal: skoid, saoid, suoid

sumangala17 added 3 commits

March 31, 2021 18:20


          mask oids

e0b34b3


          basic test

dc64112


          spotbug viz

8e3ea6e

steveloughran self-requested a review

March 31, 2021 15:53

Contributor

steveloughran commented Mar 31, 2021

this security/privacy issues, or just due diligence?

Contributor Author

sumangala-patki commented Apr 1, 2021

Hi @steveloughran this is for privacy (no issue yet); the values masked identify the security principal (user/app)

vinaysbadami suggested changes

View reviewed changes

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java Outdated

    
                public void maskSASObjectIDs() {

                  int oidStartIdx, ampIdx, oidEndIndex, qpStrIdx;

                  for (String qpKey : SAS_OID_PARAM_KEYS) {

                    qpStrIdx = maskedUrl.indexOf('&' + qpKey);

vinaysbadami Apr 1, 2021

this.maskedUrl to be consistent with rest of file

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java Outdated

    
                }

                public void maskSASObjectIDs() {

                  int oidStartIdx, ampIdx, oidEndIndex, qpStrIdx;

vinaysbadami Apr 1, 2021

move to point of first use

Contributor Author

sumangala-patki Apr 10, 2021

done

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java Outdated Show resolved Hide resolved

sumangala17 added 5 commits

April 9, 2021 11:24


          move to utils

57c437e


          Merge branch 'trunk' into HADOOP-17618

e97a2f9


          tests

407143f


          minor change

7af9f80


          fix test assuming unique sig

d72d490

Contributor Author

sumangala-patki commented Apr 9, 2021

TEST RESULTS

HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US

HNS OAuth

[INFO] Tests run: 93, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 512, Failures: 0, Errors: 0, Skipped: 71
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 50

HNS SharedKey

[INFO] Tests run: 93, Failures: 1, Errors: 0, Skipped: 0
[WARNING] Tests run: 506, Failures: 0, Errors: 0, Skipped: 26 
[ERROR] Errors: ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:631 ? TestTimedOut
[INFO]
[ERROR] Tests run: 251, Failures: 0, Errors: 1, Skipped: 40

Non-HNS SharedKey

[INFO] Tests run: 93, Failures: 1, Errors: 0, Skipped: 0
[WARNING] Tests run: 506, Failures: 0, Errors: 0, Skipped: 249
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 40

bilaharith approved these changes

View reviewed changes

bilaharith reviewed

View reviewed changes

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java Show resolved Hide resolved

bilaharith reviewed

View reviewed changes

...ols/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java Outdated

    
                  if (maskedEncodedUrl != null) {

                    return maskedEncodedUrl;

                  }

                  return UriUtils.encodedUrlStr(getMaskedUrl());

Contributor

bilaharith Apr 12, 2021

assign the value to this.maskedEncodedUrl, then return the same.

Contributor Author

sumangala-patki Apr 20, 2021

done

bilaharith reviewed

View reviewed changes

Contributor

bilaharith left a comment

Please check the comments

sumangala17 added 4 commits

April 16, 2021 13:00


          temp

42d76d4


          tests

3a95de3


          fix null query fail

d635f07


          rm =

1e39143

Contributor Author

sumangala-patki commented Apr 20, 2021

TEST RESULTS

HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US

HNS-OAuth

[INFO] Tests run: 95, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 494, Failures: 0, Errors: 0, Skipped: 71
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 50

HNS-SharedKey

[INFO] Tests run: 95, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 503, Failures: 0, Errors: 0, Skipped: 26
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 40

NonHNS-SharedKey

[INFO] Tests run: 95, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 506, Failures: 0, Errors: 0, Skipped: 249
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 40

sumangala-patki commented

View reviewed changes

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/UriUtils.java Outdated Show resolved Hide resolved

sumangala-patki commented

View reviewed changes

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/UriUtils.java Outdated Show resolved Hide resolved

sumangala-patki commented

View reviewed changes

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/UriUtils.java Outdated Show resolved Hide resolved

sumangala-patki commented

View reviewed changes

hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/UriUtils.java Show resolved Hide resolved

sumangala-patki commented

View reviewed changes

Contributor Author

sumangala-patki left a comment

check comments


          pr revw

apache deleted a comment from hadoop-yetus

apache deleted a comment from hadoop-yetus

sumangala17 added 3 commits

April 23, 2021 18:17


          chkstyle

cb46432


          chkstyle


          Merge branch 'trunk' into HADOOP-17618

6a75563

Contributor Author

sumangala-patki commented Apr 23, 2021

TEST RESULTS

HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US

HNS OAuth

[INFO] Tests run: 99, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 519, Failures: 0, Errors: 0, Skipped: 70
[WARNING] Tests run: 261, Failures: 0, Errors: 0, Skipped: 50

HNS SharedKey

[INFO] Tests run: 99, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 507, Failures: 0, Errors: 0, Skipped: 26
[WARNING] Tests run: 251, Failures: 0, Errors: 0, Skipped: 40

Non-HNS SharedKey

[INFO] Tests run: 99, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 519, Failures: 0, Errors: 0, Skipped: 248
[ERROR] Errors:
[ERROR]   ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:62->testReadWriteAndSeek:78 ? TestTimedOut
[ERROR] Tests run: 261, Failures: 0, Errors: 1, Skipped: 40

JIRA to track timeout failure: ITestAbfsReadWriteAndSeek

sumangala-patki marked this pull request as ready for review

April 23, 2021 17:05

sumangala-patki requested a review from steveloughran

April 26, 2021 06:12

apache deleted a comment from hadoop-yetus

apache deleted a comment from hadoop-yetus

Contributor Author

sumangala-patki commented Apr 29, 2021

Hi @steveloughran, thanks for the review. Have addressed the comments, please take a look. Thank you!

apache deleted a comment from hadoop-yetus

apache deleted a comment from hadoop-yetus

apache deleted a comment from hadoop-yetus

apache deleted a comment from hadoop-yetus

steveloughran approved these changes

View reviewed changes

Contributor

steveloughran left a comment

I'm happy with this.

@sumangala-patki -can you rebase/merge with trunk and push up for yetus to recheck?

+1 pending yetus being happy


          Merge branch 'trunk' into HADOOP-17618

2cd7519

Contributor Author

sumangala-patki commented Aug 4, 2021

TEST RESULTS

HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US

AppendBlob-HNS-OAuth

[WARNING] Tests run: 105, Failures: 0, Errors: 0, Skipped: 1
[ERROR] Failures: 
[ERROR]   ITestAbfsStreamStatistics.testAbfsStreamOps:140->Assert.assertTrue:42->Assert.fail:89 The actual value of 99 was not equal to the expected value
[ERROR] Errors: 
[ERROR]   ITestAzureBlobFileSystemLease.testTwoWritersCreateAppendNoInfiniteLease:178->twoWriters:166 » IO
[ERROR] Tests run: 558, Failures: 1, Errors: 1, Skipped: 98
[WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 76

HNS-OAuth

[WARNING] Tests run: 105, Failures: 0, Errors: 0, Skipped: 1
[WARNING] Tests run: 558, Failures: 0, Errors: 0, Skipped: 98
[WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 52

HNS-SharedKey

[WARNING] Tests run: 105, Failures: 0, Errors: 0, Skipped: 2
[ERROR] Failures: 
[ERROR]   ITestAbfsRestOperationException.testCustomTokenFetchRetryCount:93->testWithDifferentCustomTokenFetchRetry:122->Assert.assertTrue:42->Assert.fail:89 Number of token fetch retries (4) done, does not match with fs.azure.custom.token.fetch.retry.count configured (0)
[ERROR] Tests run: 558, Failures: 1, Errors: 0, Skipped: 54
[WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 40

NonHNS-SharedKey

[ERROR] Failures: 
[ERROR]   TestAbfsClientThrottlingAnalyzer.testManySuccessAndErrorsAndWaiting:171->fuzzyValidate:49 The actual value 16 is not within the expected range: [5.60, 8.40].
[ERROR] Tests run: 105, Failures: 1, Errors: 0, Skipped: 2
[WARNING] Tests run: 558, Failures: 0, Errors: 0, Skipped: 276
[WARNING] Tests run: 259, Failures: 0, Errors: 0, Skipped: 40

JIRAs to track failures: Appendblob lease, TestAbfsClientThrottlingAnalyzer, StreamOps

steveloughran merged commit 3450522 into apache:trunk

Contributor

steveloughran commented Aug 4, 2021

thanks, merged to trunk. Do a test run after cherrypicking to 3.3 and we can merge it there too

sumangala-patki added a commit to sumangala17/hadoop that referenced this pull request


          HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs (apach…

af4adc2

…e#2845)

Contributed by Sumangala Patki

(cherry picked from commit 3450522)

apache deleted a comment from hadoop-yetus

apache deleted a comment from hadoop-yetus

apache deleted a comment from hadoop-yetus

apache deleted a comment from hadoop-yetus

apache deleted a comment from hadoop-yetus

apache deleted a comment from hadoop-yetus

apache deleted a comment from hadoop-yetus

apache deleted a comment from hadoop-yetus

apache deleted a comment from hadoop-yetus

steveloughran pushed a commit that referenced this pull request


          HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs (#2845)

1cb9e74

Contributed by Sumangala Patki

(cherry picked from commit 3450522)

kiran-maturi pushed a commit to kiran-maturi/hadoop that referenced this pull request


          HADOOP-17618. ABFS: Partially obfuscate SAS object IDs in Logs (apach…

ecf9713

…e#2845)


Contributed by Sumangala Patki

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet