Skip to content

HDDS-8543. FSO directory should reflect bucket/cluster default replication #4947

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
adoroszlai merged 8 commits into from
Jun 22, 2023
Merged

HDDS-8543. FSO directory should reflect bucket/cluster default replication #4947

adoroszlai merged 8 commits into from
Jun 22, 2023

Conversation

@adoroszlai
Copy link
Contributor

@adoroszlai adoroszlai commented Jun 21, 2023

What changes were proposed in this pull request?

Change key info for FSO directory to reflect the bucket's default replication config, if set, otherwise the cluster-wide default setting OM uses for new keys. Previously it always showed RATIS/ONE.

https://issues.apache.org/jira/browse/HDDS-8543

How was this patch tested?

Added assertions in integration and acceptance tests.

$ ozone sh key list vol23126/ratis   
[ {
  "volumeName" : "vol23126",
  "bucketName" : "ratis",
  "name" : "dir/",
  "dataSize" : 0,
  "replicationConfig" : {
    "replicationFactor" : "THREE",
    "requiredNodes" : 3,
    "replicationType" : "RATIS"
  },
  ...
},
  ...

$ ozone sh key list vol23126/ecbucket
[ {
  "volumeName" : "vol23126",
  "bucketName" : "ecbucket",
  "name" : "dir/",
  "dataSize" : 0,
  "replicationConfig" : {
    "data" : 3,
    "parity" : 2,
    "ecChunkSize" : 1048576,
    "codec" : "RS",
    "replicationType" : "EC",
    "requiredNodes" : 5
  },
  ...
},
  ...

https://github.com/adoroszlai/hadoop-ozone/actions/runs/5335943835

@adoroszlai adoroszlai self-assigned this Jun 21, 2023
@adoroszlai adoroszlai changed the title HDDS-8543. New key prefix in EC bucket with FSO reports RATIS replication HDDS-8543. FSO directory should reflect bucket/cluster default replication Jun 21, 2023
@adoroszlai adoroszlai requested a review from sadanand48 June 22, 2023 09:30
sadanand48
sadanand48 approved these changes Jun 22, 2023
View reviewed changes
Copy link
Contributor

@sadanand48 sadanand48 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @adoroszlai for the changes. LGTM

sodonnel
sodonnel reviewed Jun 22, 2023
View reviewed changes
Copy link
Contributor

@sodonnel sodonnel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good to me, but I am not really up to speed on OM. Happy to defer to @sadanand48 on his review as he knows the area better than me.

@adoroszlai adoroszlai merged commit 05c1e85 into apache:master Jun 22, 2023
@adoroszlai adoroszlai deleted the HDDS-8543 branch June 22, 2023 18:42
@adoroszlai
Copy link
Contributor Author

adoroszlai commented Jun 22, 2023

Thanks @sadanand48, @sodonnel for the review.

errose28 added a commit to errose28/ozone that referenced this pull request Jun 25, 2023
@errose28
Merge branch 'master' into tmp-dir-refactor
e502a07 
* master: (96 commits)
  HDDS-8586 Recon. - API for Count of deletePending keys and amount of data mapped to such keys. (apache#4923)
  HDDS-8908. Intermittent failure in TestBlockDeletion#testBlockDeletion (apache#4958)
  HDDS-8910. Replace LockManager with striped lock in ContainerStateManager (apache#4962)
  HDDS-8917. Move protobuf conversion out of the lock in PipelineStateManagerImpl (apache#4965)
  HDDS-8825. Use apache/hadoop 3.3.5 docker image (apache#4963)
  HDDS-8906. Avoid stream when getting in-service healthy nodes (apache#4960)
  HDDS-8907. Store volume count when storage report is updated (apache#4957)
  HDDS-8905. PipelineManager metrics should not be synchronized (apache#4959)
  HDDS-8553. Improve scanner integration tests. (apache#4936)
  HDDS-8854. Avoid unnecessary DatanodeDetails creation for NodeStateManager lookup (apache#4925)
  HDDS-8315. [Snapshot] Added unit tests for SnapshotDiffManager (apache#4716)
  HDDS-7968. [Snapshot] Improve KeyDeletingService to reclaim eligible key blocks in snapshot's deletedTable (apache#4935)
  HDDS-8838. Update default datanode check empty containter on disk to false (apache#4937)
  HDDS-8763. Support RocksDB iterator with ByteBuffer. (apache#4942)
  HDDS-8543. FSO directory should reflect bucket/cluster default replication (apache#4947)
  HDDS-8898. Replication limit should not be less than reconstruction weight (apache#4954)
  HDDS-8739. Snapdiff should return complete absolute path in Diff Entry (apache#4823)
  HDDS-8908. Mark TestBlockDeletion#testBlockDeletion as flaky
  HDDS-8534. Support asynchronous service logging (apache#4663)
  HDDS-8879. Cleanup SecurityConfig and related class initialization (apache#4921)
  ...
errose28 added a commit to errose28/ozone that referenced this pull request Jun 26, 2023
@errose28
Merge branch 'tmp-dir-refactor' into improve-volume-scanner
fa4207c 
* tmp-dir-refactor: (99 commits)
  HDDS-8586 Recon. - API for Count of deletePending keys and amount of data mapped to such keys. (apache#4923)
  Fix SCM HA finalization compat test
  HDDS-8908. Intermittent failure in TestBlockDeletion#testBlockDeletion (apache#4958)
  HDDS-8910. Replace LockManager with striped lock in ContainerStateManager (apache#4962)
  HDDS-8917. Move protobuf conversion out of the lock in PipelineStateManagerImpl (apache#4965)
  HDDS-8825. Use apache/hadoop 3.3.5 docker image (apache#4963)
  HDDS-8906. Avoid stream when getting in-service healthy nodes (apache#4960)
  HDDS-8907. Store volume count when storage report is updated (apache#4957)
  HDDS-8905. PipelineManager metrics should not be synchronized (apache#4959)
  HDDS-8553. Improve scanner integration tests. (apache#4936)
  HDDS-8854. Avoid unnecessary DatanodeDetails creation for NodeStateManager lookup (apache#4925)
  HDDS-8315. [Snapshot] Added unit tests for SnapshotDiffManager (apache#4716)
  HDDS-7968. [Snapshot] Improve KeyDeletingService to reclaim eligible key blocks in snapshot's deletedTable (apache#4935)
  HDDS-8838. Update default datanode check empty containter on disk to false (apache#4937)
  HDDS-8763. Support RocksDB iterator with ByteBuffer. (apache#4942)
  HDDS-8543. FSO directory should reflect bucket/cluster default replication (apache#4947)
  HDDS-8898. Replication limit should not be less than reconstruction weight (apache#4954)
  HDDS-8739. Snapdiff should return complete absolute path in Diff Entry (apache#4823)
  HDDS-8908. Mark TestBlockDeletion#testBlockDeletion as flaky
  HDDS-8534. Support asynchronous service logging (apache#4663)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sodonnel sodonnel sodonnel left review comments

@sadanand48 sadanand48 sadanand48 approved these changes

@siddhantsangwan siddhantsangwan Awaiting requested review from siddhantsangwan

Assignees

@adoroszlai adoroszlai

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@adoroszlai @sodonnel @sadanand48