Skip to content

Conversation

@adoroszlai
Copy link
Contributor

What changes were proposed in this pull request?

S3 Gateway logs exception in S3ErrorTable.newError, but the stack trace is wrong.

s3g_1       | <Error>
s3g_1       |   <Code>InvalidBucketName</Code>
s3g_1       |   <Message>The specified bucket is not valid.</Message>
s3g_1       |   <Resource>a_b_c</Resource>
s3g_1       |   <RequestId/>
s3g_1       | </Error>
s3g_1       |
s3g_1       | org.apache.hadoop.ozone.s3.exception.OS3Exception
s3g_1       |    at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.<clinit>(S3ErrorTable.java:88)
s3g_1       |    at org.apache.hadoop.ozone.s3.endpoint.EndpointBase.getBucket(EndpointBase.java:91)
s3g_1       |    at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.head(BucketEndpoint.java:294)
s3g_1       | <Error>
s3g_1       |   <Code>NoSuchBucket</Code>
s3g_1       |   <Message>The specified bucket does not exist</Message>
s3g_1       |   <Resource>dont-exist</Resource>
s3g_1       |   <RequestId/>
s3g_1       | </Error>
s3g_1       |
s3g_1       | org.apache.hadoop.ozone.s3.exception.OS3Exception
s3g_1       |    at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.<clinit>(S3ErrorTable.java:51)
s3g_1       |    at org.apache.hadoop.ozone.s3.endpoint.EndpointBase.getBucket(EndpointBase.java:91)
s3g_1       |    at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.head(BucketEndpoint.java:294)

Note that both errors have the same stack trace. It always points to the operation that first accessed S3ErrorTable class (i.e. the one that triggered the first error, in this case a head-bucket operation).

This change fixes the stack trace by passing the original exception, if available, to the method which logs, instead of using the predefined OS3Exception's stack trace.

https://issues.apache.org/jira/browse/HDDS-6257

How was this patch tested?

Set log level to debug:

log4j.logger.org.apache.hadoop.ozone.s3.exception.S3ErrorTable=DEBUG

Tested with the following commands. Verified that stack traces are unique.

aws s3api --endpoint http://s3g:9878 head-bucket --bucket dont-exist
s3g_1       | 2022-02-09 14:41:46,981 [qtp690052870-22] DEBUG exception.S3ErrorTable: <?xml version="1.0" encoding="UTF-8"?>
s3g_1       | <Error>
s3g_1       |   <Code>NoSuchBucket</Code>
s3g_1       |   <Message>The specified bucket does not exist</Message>
s3g_1       |   <Resource>dont-exist</Resource>
s3g_1       |   <RequestId/>
s3g_1       | </Error>
s3g_1       |
s3g_1       | BUCKET_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Bucket not found
s3g_1       |    at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:654)
s3g_1       |    at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getBucketInfo(OzoneManagerProtocolClientSideTranslatorPB.java:503)
s3g_1       |    at org.apache.hadoop.ozone.client.rpc.RpcClient.getBucketDetails(RpcClient.java:771)
s3g_1       |    at org.apache.hadoop.ozone.client.OzoneVolume.getBucket(OzoneVolume.java:406)
s3g_1       |    at org.apache.hadoop.ozone.client.ObjectStore.getS3Bucket(ObjectStore.java:122)
s3g_1       |    at org.apache.hadoop.ozone.s3.endpoint.EndpointBase.getBucket(EndpointBase.java:89)
s3g_1       |    at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.head(BucketEndpoint.java:293)
aws s3api --endpoint http://s3g:9878 delete-bucket --bucket dont-exist
s3g_1       | 2022-02-09 14:42:29,766 [qtp690052870-16] DEBUG exception.S3ErrorTable: <?xml version="1.0" encoding="UTF-8"?>
s3g_1       | <Error>
s3g_1       |   <Code>NoSuchBucket</Code>
s3g_1       |   <Message>The specified bucket does not exist</Message>
s3g_1       |   <Resource>dont-exist</Resource>
s3g_1       |   <RequestId/>
s3g_1       | </Error>
s3g_1       |
s3g_1       | BUCKET_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Bucket not exists
s3g_1       |    at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:654)
s3g_1       |    at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.deleteBucket(OzoneManagerProtocolClientSideTranslatorPB.java:856)
s3g_1       |    at org.apache.hadoop.ozone.client.rpc.RpcClient.deleteBucket(RpcClient.java:756)
s3g_1       |    at org.apache.hadoop.ozone.client.OzoneVolume.deleteBucket(OzoneVolume.java:444)
s3g_1       |    at org.apache.hadoop.ozone.client.ObjectStore.deleteS3Bucket(ObjectStore.java:133)
s3g_1       |    at org.apache.hadoop.ozone.s3.endpoint.EndpointBase.deleteS3Bucket(EndpointBase.java:140)
s3g_1       |    at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.delete(BucketEndpoint.java:308)
aws s3api --endpoint http://s3g:9878 create-bucket --bucket 'a_b_c'
s3g_1       | 2022-02-09 14:43:02,028 [qtp690052870-17] DEBUG exception.S3ErrorTable: <?xml version="1.0" encoding="UTF-8"?>
s3g_1       | <Error>
s3g_1       |   <Code>InvalidBucketName</Code>
s3g_1       |   <Message>The specified bucket is not valid.</Message>
s3g_1       |   <Resource>a_b_c</Resource>
s3g_1       |   <RequestId/>
s3g_1       | </Error>
s3g_1       |
s3g_1       | INVALID_BUCKET_NAME org.apache.hadoop.ozone.om.exceptions.OMException: Bucket or Volume name has an unsupported character : _
s3g_1       |    at org.apache.hadoop.ozone.client.rpc.RpcClient.verifyBucketName(RpcClient.java:582)
s3g_1       |    at org.apache.hadoop.ozone.client.rpc.RpcClient.createBucket(RpcClient.java:521)
s3g_1       |    at org.apache.hadoop.ozone.client.rpc.RpcClient.createBucket(RpcClient.java:512)
s3g_1       |    at org.apache.hadoop.ozone.client.OzoneVolume.createBucket(OzoneVolume.java:385)
s3g_1       |    at org.apache.hadoop.ozone.client.ObjectStore.createS3Bucket(ObjectStore.java:118)
s3g_1       |    at org.apache.hadoop.ozone.s3.endpoint.EndpointBase.createS3Bucket(EndpointBase.java:119)
s3g_1       |    at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.put(BucketEndpoint.java:240)
ozone sh bucket create /s3v/bucket1
ozone sh key put /s3v/bucket1/key1 /etc/passwd
aws s3api --endpoint http://s3g:9878 get-object --bucket bucket1 --key key1 --range bytes=10000-10000 /dev/null
s3g_1       | 2022-02-09 14:56:33,771 [qtp690052870-22] DEBUG exception.S3ErrorTable: <?xml version="1.0" encoding="UTF-8"?>
s3g_1       | <Error>
s3g_1       |   <Code>InvalidRange</Code>
s3g_1       |   <Message>The requested range is not satisfiable</Message>
s3g_1       |   <Resource>bytes=10000-10000</Resource>
s3g_1       |   <RequestId/>
s3g_1       | </Error>

CI:
https://github.com/adoroszlai/hadoop-ozone/actions/runs/1818745426

@adoroszlai adoroszlai self-assigned this Feb 9, 2022
Copy link
Contributor

@sodonnel sodonnel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes LGTM

@adoroszlai adoroszlai merged commit 594c4a9 into apache:master Feb 10, 2022
@adoroszlai adoroszlai deleted the HDDS-6257 branch February 10, 2022 18:07
@adoroszlai
Copy link
Contributor Author

Thanks @sodonnel for the review.

errose28 added a commit to errose28/ozone that referenced this pull request Mar 30, 2022
* HDDS-4944: (268 commits)
  HDDS-6366. [Multi-Tenant] Disallow specifying custom accessId in OzoneManager (apache#3166)
  HDDS-6275. [Multi-Tenant] Add feature documentation and CLI quick start guide (apache#3095)
  HDDS-6063. [Multi-Tenant] Use VOLUME_LOCK in read and write requests, and some minor refactoring (apache#3051)
  HDDS-6214. [Multi-Tenant] Fix KMS Encryption/Decryption (apache#3010)
  HDDS-6322. Fix Recon getting inccorrect sequenceNumber from OM (apache#3090)
  HDDS-5913. Avoid integer overflow when setting dfs.container.ratis.lo… (apache#2785)
  HDDS-6313. Remove replicas in ContainerStateMap when a container is deleted (apache#3086)
  HDDS-6186. Selective checks: skip integration check for unit test changes (apache#3061)
  HDDS-6310. Update json-smart to 2.4.7. (apache#3080)
  HDDS-6190. Cleanup unnecessary datanode id path checks. (apache#2993)
  HDDS-6304. Add enforcer to make sure ozone.version equals project.version (apache#3075)
  HDDS-6309. Update ozone-runner version to 20220212-1 (apache#3079)
  HDDS-6293. Allow using custom ozone-runner image (apache#3072)
  HDDS-4126. Freon key generator should support >2GB files. (apache#3054)
  HDDS-6088. Implement O3FS/OFS getFileChecksum() using file checksum helpers - addendum: fix checkstyle
  HDDS-6088. Implement O3FS/OFS getFileChecksum() using file checksum helpers. (apache#2935)
  HDDS-6084. [Multi-Tenant] Handle upgrades to version supporting S3 multi-tenancy (apache#3018)
  HDDS-6257. Wrong stack trace for S3 errors (apache#3066)
  HDDS-6278 Improve memory profile for listStatus API call. (apache#3053)
  HDDS-6285. ozonesecure-mr intermittently failing with timeout downloading packages (apache#3057)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants