HDDS-3752. Patch for o3fs issue with not listing bucket contents WITHOUT trailin… #2088
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…g slash. Originally when ls -R o3fs://bucket.vol issued, resulted in No such file/directory error. Fixed path so that contents of bucket listed when command issued without trailing slash. Patch affects one file, the BasicOzoneFileSystem filesystem abstraction. Patch overrides the fixRelativePart method to allow Path to be resolved correctly for OzoneFileSystem implementation.
avijayanhwx
reviewed
Mar 29, 2021
Contributor
avijayanhwx
left a comment
avijayanhwx
left a comment
There was a problem hiding this comment.
Thanks for working on this. LGTM +1 (pending CI)
…s coding style checkstyle.sh. Simplification to code with method override providing implementation specific modifications to fixRelativePart and calling "super" for non implementation specific.
Contributor
Author
|
Thanks @avijayanhwx . Pushed changes to pass coding style checks. Simplification of fixRelativePart method (calling super for non implementation specific operations for method). |
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Apr 7, 2021
* HDDS-3698-nonrolling-upgrade: (144 commits) fix project name in NOTICE.txt (apache#2112) HDDS-5066. Use fixed vesion from pnpm to build recon (apache#2115) HDDS-5014. Add non-rolling upgrade design docs. HDDS-5035. Use default config values to solve generated config file conflict (apache#2087) HDDS-5032. DN stopped to load containers on volume after a container load exception. (apache#2109) HDDS-4504. Datanode deletion config should be based on number of blocks (apache#1885) Fix ozone-ha acceptance test. HDDS-5058. Make getScmInfo retry for a duration. HDDS-4506. Support query parameter based v4 auth in S3g (apache#1628) HDDS-4553. ChunkInputStream should release buffer as soon as last byte in the buffer is read (apache#2062) HDDS-5022. SCM get roles command should provide Ratis Leader/Follower… (apache#2098) HDDS-5033. SCM may not be able to know full port list of Datanode after Datanode is started. (apache#2090) HDDS-3752. Fix o3fs list bucket contents issue when without tailing "/" (apache#2088) HDDS-4901. Remove OmOzoneAclMap from OmVolumeArgs to avoid OzoneAcl conversions (apache#1992) HDDS-4987. Import container should not delete container contents if container already exists (apache#2077) Checkstyle fix. Intialize DN layout version before security init. HDDS-4915. [SCM HA Security] Integrate CertClient. (apache#2000) HDDS-5049. Add timeout support for ratis requests in SCM HA. (apache#2099) trigger new CI check ...
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Apr 9, 2021
* HDDS-3698-nonrolling-upgrade: (150 commits) HDDS-5056. Avoid false positiver error messages during pipeline creations (apache#2105) HDDS-5027. [SCM HA Security] Handle leader changes during bootstrap. (apache#2113) HDDS-5032. Fix findbugs (apache#2120) HDDS-5062. Add a config to bypass clusterId validation for bootstrapping SCM. (apache#2114) HDDS-5011. Introduce Java based ReplicationConfig implementation (apache#2089) HDDS-4925. Introduce ContainerBalancer in SCM with start/stop capabilities. (apache#2097) fix project name in NOTICE.txt (apache#2112) HDDS-5066. Use fixed vesion from pnpm to build recon (apache#2115) HDDS-5014. Add non-rolling upgrade design docs. HDDS-5035. Use default config values to solve generated config file conflict (apache#2087) HDDS-5032. DN stopped to load containers on volume after a container load exception. (apache#2109) HDDS-4504. Datanode deletion config should be based on number of blocks (apache#1885) Fix ozone-ha acceptance test. HDDS-5058. Make getScmInfo retry for a duration. HDDS-4506. Support query parameter based v4 auth in S3g (apache#1628) HDDS-4553. ChunkInputStream should release buffer as soon as last byte in the buffer is read (apache#2062) HDDS-5022. SCM get roles command should provide Ratis Leader/Follower… (apache#2098) HDDS-5033. SCM may not be able to know full port list of Datanode after Datanode is started. (apache#2090) HDDS-3752. Fix o3fs list bucket contents issue when without tailing "/" (apache#2088) HDDS-4901. Remove OmOzoneAclMap from OmVolumeArgs to avoid OzoneAcl conversions (apache#1992) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
...Originally when ls -R o3fs://bucket.vol issued, resulted in No such file/directory error. Fixed path so that contents of bucket listed when command issued without trailing slash. Patch affects one file, the BasicOzoneFileSystem filesystem abstraction. Patch overrides the fixRelativePart method to allow Path to be resolved correctly for OzoneFileSystem implementation.
What changes were proposed in this pull request?
Patch fixed issue with OzoneFileSystem implementation of FileSystem for o3fs space when -ls command request without trailing slash of 2nd level bucket (vol/bucket). Problem is that request interpreted as a 'relative' address resolving to the 'working directory' which is evaluated to an error 'Directory or File Not Found'.
Patch to BasicOzoneFileSystem overrides the fixRelativePart method to resolve the Path to the correct Path (adding a trailing slash). BasicOzoneFileSystem implementation of FileSystem then correctly evaluates the path.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3752
How was this patch tested?
Passed unit and integration tests.
Manual testing:
Tested though ozone fs cli commands (see attached images).
Pre-patch:
Fix with patch:
