Skip to content

HDDS-5676. When use s3g and goofys, some write operation will failed. #2583

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zhengchenyu wants to merge 1 commit into from
Closed

HDDS-5676. When use s3g and goofys, some write operation will failed. #2583

zhengchenyu wants to merge 1 commit into from

Conversation

@zhengchenyu
Copy link
Contributor

@zhengchenyu zhengchenyu commented Aug 26, 2021 •
edited
Loading

What changes were proposed in this pull request?

S3g should return FILE or Directory not found with 404.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5676

How was this patch tested?

manual tests.

Reproduce case:

截屏2021-08-30 下午5 30 36

File 'abc' in bucket tmp is not created.

logs

goofys related logs:

-----------------------------------------------------
2021/08/30 17:32:34.104385 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 500 Server Error
Connection: close
Content-Length: 380
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=iso-8859-1
Pragma: no-cache
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block


-----------------------------------------------------
2021/08/30 17:32:34.104455 s3.DEBUG DEBUG: Validate Response s3/ListObjects failed, not retrying, error SerializationError: failed to decode S3 XML error response
	status code: 500, request id: , host id:
caused by: expected element type <Error> but have <html>
2021/08/30 17:32:34.104499 s3.DEBUG LIST abc/ = resource temporarily unavailable
2021/08/30 17:32:34.104521 fuse.DEBUG <-- LookUpInode 1 abc resource temporarily unavailable
2021/08/30 17:32:34.104571 fuse.DEBUG Op 0x00000003        connection.go:493] -> Error: "resource temporarily unavailable"
2021/08/30 17:32:34.104593 fuse.ERROR *fuseops.LookUpInodeOp error: resource temporarily unavailable

s3g related logs:

Caused by: FILE_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Unable to get file status: volume: s3v bucket: bucket key: abc/
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:613)
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.listStatus(OzoneManagerProtocolClientSideTranslatorPB.java:1508)
	at org.apache.hadoop.ozone.client.rpc.RpcClient.listStatus(RpcClient.java:1278)

@kerneltime
Copy link
Contributor

kerneltime commented Aug 28, 2021

  • Can you paste the exact cli invocation? mv /mount/bucket1/abc /mount/bucket1/xyz should not result in a 404.

  • We should not extend the S3 Error Messages unless we have to. Returning 404 not found for both should be suitable.

@zhengchenyu
Copy link
Contributor Author

zhengchenyu commented Aug 30, 2021 •
edited
Loading

  • Can you paste the exact cli invocation? mv /mount/bucket1/abc /mount/bucket1/xyz should not result in a 404.
  • We should not extend the S3 Error Messages unless we have to. Returning 404 not found for both should be suitable.

I have reproduce the problem, please help me review it.

@zhengchenyu zhengchenyu mentioned this pull request Aug 30, 2021
Closed
@kerneltime
Copy link
Contributor

kerneltime commented Aug 30, 2021

echo hello > /mount/tmp/abc should create an object under the bucket mounted at tmp called abc. Why do you think abc should be a directory?
In the steps outlined, there was a bucket created called tmp using AWS CLI but the goofyfs command used a bucket named bucket and not tmp to be mounted under /mount/tmp..

@zhengchenyu
Copy link
Contributor Author

zhengchenyu commented Aug 31, 2021 •
edited
Loading

@kerneltime

echo hello > /mount/tmp/abc should create an object under the bucket mounted at tmp called abc. Why do you think abc should be a directory?

First, I didn't said that abc is a directory. Here abc is not created. Because s3g returned 500 means internal error. goofys can't resovle it, so failed.

In the steps outlined, there was a bucket created called tmp using AWS CLI but the goofyfs command used a bucket named bucket and not tmp to be mounted under /mount/tmp..

Here, there are two test:
(1) mv /mount/bucket1/abc /mount/bucket1/xyz
(2) echo hello > /mount/tmp/abc

I use (2) to reproduce detail info.
Note: Many write operation dir will fail. Please ignore (1), that's about #2584 which is closed.

@zhengchenyu
Copy link
Contributor Author

zhengchenyu commented Aug 31, 2021

I found only ozone.om.enable.filesystem.paths=true and ozone.om.metadata.layout=PREFIX, operation will failed. As #1418 and #2584 mentioned, we should make sure bucketlayout is OBJECT_STORE. So expect the progress of https://issues.apache.org/jira/browse/HDDS-5672. I will close it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@zhengchenyu @kerneltime