HDDS-8771. Refactor volume level tmp directory for generic usage. #4838

errose28 · 2023-06-07T00:04:34Z

What changes were proposed in this pull request?

Most of the diff is renames and moving methods to different classes.

Move volume level tmp directory management to StorageVolume.
- Volume types other than HddsVolume may need to use the tmp directory. For example, syncing volume health check files, which will come after this PR.
Move deleted container directory management to HddsVolume.
- This uses the tmp directory from the StorageVolume parent class for container deletion, which is specific to HddsVolumes.
Rename methods to better reflect tmp directory and deleted container directory distinction.
Rename on disk directory from container_delete_service to deleted-containers to better reflect its contents.
- There have been no releases with HDDS-6449, so changing this directory name will not affect released clusters.
- Any clusters built off master between HDDS-6449 and now will ignore the old directory after this change. It could be manually deleted if desired.

What is the link to the Apache JIRA

HDDS-8771

How was this patch tested?

Existing tests updated
Fix issue preventing restart with schema v3
- TestStorageVolumeUtil#testCheckVolumeNoDupDbStoreCreated and TestEndpoint#testDeletedContainersClearedOnStartup are currently failing for this reason.
Manual testing with docker
CI passing on my fork

Change was lost in the cherry-pick process

Subdirs will now be created on upgraded clusters as well.

* master: (73 commits) HDDS-8587. Test that CertificateClient can store multiple rootCA certificates (apache#4852) HDDS-8801. ReplicationManager: Add metric to count how often replication is throttled (apache#4864) HDDS-8477. Unit test for Snapdiff using tombstone entries (apache#4678) HDDS-7507. [Snapshot] Implement List Snapshot API Pagination (apache#4065) (apache#4861) HDDS-8373. Document that setquota doesn't accept decimals (apache#4856) HDDS-8779. Recon - Expose flag for enable/disable of heatmap. (apache#4845) HDDS-8677. Ozone admin OM CLI command for block tokens (apache#4760) HDDS-8164. Authorize secret key APIs (apache#4597) HDDS-7945. Integrate secret keys to SCM snapshot (apache#4549) HDDS-8003. E2E integration test cases for block tokens (apache#4547) HDDS-7831. Use symmetric secret key to sign and verify token (apache#4417) HDDS-7830. SCM API for OM and Datanode to get secret keys (apache#4345) HDDS-7734. Implement symmetric SecretKeys lifescycle management in SCM (apache#4194) HDDS-8679. Add dedicated, configurable thread pool for OM gRPC server (apache#4771) HDDS-8790. Split EC acceptance tests (apache#4855) HDDS-8714. TestScmHAFinalization: mark testFinalizationWithRestart as flaky, enable other test cases HDDS-8787. Reduce ozone sh calls in robot tests (apache#4854) HDDS-8774. Log allocation stack trace for leaked CodecBuffer (apache#4840) HDDS-8729. Add metric for count of blocks pending deletion on datanode (apache#4800) HDDS-8780. Leak of ManagedChannel in HASecurityUtils (apache#4850) ...

sumitagrawl

@errose28 Thanks for working over this, have few minor comments

...-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/StorageVolumeUtil.java

...op-hdds/container-service/src/test/java/org/apache/hadoop/ozone/TestHddsDatanodeService.java

sumitagrawl

@errose28 Thanks for working, LGTM +1

errose28 · 2023-06-22T16:57:36Z

Thanks for the review @sumitagrawl. I will fix the failing unit test and merge conflicts.

* master: (96 commits) HDDS-8586 Recon. - API for Count of deletePending keys and amount of data mapped to such keys. (apache#4923) HDDS-8908. Intermittent failure in TestBlockDeletion#testBlockDeletion (apache#4958) HDDS-8910. Replace LockManager with striped lock in ContainerStateManager (apache#4962) HDDS-8917. Move protobuf conversion out of the lock in PipelineStateManagerImpl (apache#4965) HDDS-8825. Use apache/hadoop 3.3.5 docker image (apache#4963) HDDS-8906. Avoid stream when getting in-service healthy nodes (apache#4960) HDDS-8907. Store volume count when storage report is updated (apache#4957) HDDS-8905. PipelineManager metrics should not be synchronized (apache#4959) HDDS-8553. Improve scanner integration tests. (apache#4936) HDDS-8854. Avoid unnecessary DatanodeDetails creation for NodeStateManager lookup (apache#4925) HDDS-8315. [Snapshot] Added unit tests for SnapshotDiffManager (apache#4716) HDDS-7968. [Snapshot] Improve KeyDeletingService to reclaim eligible key blocks in snapshot's deletedTable (apache#4935) HDDS-8838. Update default datanode check empty containter on disk to false (apache#4937) HDDS-8763. Support RocksDB iterator with ByteBuffer. (apache#4942) HDDS-8543. FSO directory should reflect bucket/cluster default replication (apache#4947) HDDS-8898. Replication limit should not be less than reconstruction weight (apache#4954) HDDS-8739. Snapdiff should return complete absolute path in Diff Entry (apache#4823) HDDS-8908. Mark TestBlockDeletion#testBlockDeletion as flaky HDDS-8534. Support asynchronous service logging (apache#4663) HDDS-8879. Cleanup SecurityConfig and related class initialization (apache#4921) ...

* master: (79 commits) HDDS-8914. Datanode may fail to start due to duplicate VolumeInfoMetrics (apache#4966) HDDS-8921. Add support for EC in Freon SCM block generator (apache#4982) HDDS-8927. Metadata scanner should not scan unhealthy containers. (apache#4976) HDDS-8929. Avoid list allocation for pipeline search (apache#4980) HDDS-8778. Support recursive volume delete using Ozone sh command. (apache#4842) HDDS-8885. Quota repair count enable quota feature for old bucket/volume. (apache#4941) HDDS-8771. Refactor volume level tmp directory for generic usage. (apache#4838) HDDS-8922. Random EC read pipeline ID causes XceiverClient cache churn (apache#4971) HDDS-8586 Recon. - API for Count of deletePending keys and amount of data mapped to such keys. (apache#4923) HDDS-8908. Intermittent failure in TestBlockDeletion#testBlockDeletion (apache#4958) HDDS-8910. Replace LockManager with striped lock in ContainerStateManager (apache#4962) HDDS-8917. Move protobuf conversion out of the lock in PipelineStateManagerImpl (apache#4965) HDDS-8825. Use apache/hadoop 3.3.5 docker image (apache#4963) HDDS-8906. Avoid stream when getting in-service healthy nodes (apache#4960) HDDS-8907. Store volume count when storage report is updated (apache#4957) HDDS-8905. PipelineManager metrics should not be synchronized (apache#4959) HDDS-8553. Improve scanner integration tests. (apache#4936) HDDS-8854. Avoid unnecessary DatanodeDetails creation for NodeStateManager lookup (apache#4925) HDDS-8315. [Snapshot] Added unit tests for SnapshotDiffManager (apache#4716) HDDS-7968. [Snapshot] Improve KeyDeletingService to reclaim eligible key blocks in snapshot's deletedTable (apache#4935) ...

* master: HDDS-8914. Datanode may fail to start due to duplicate VolumeInfoMetrics (apache#4966) HDDS-8921. Add support for EC in Freon SCM block generator (apache#4982) HDDS-8927. Metadata scanner should not scan unhealthy containers. (apache#4976) HDDS-8929. Avoid list allocation for pipeline search (apache#4980) HDDS-8778. Support recursive volume delete using Ozone sh command. (apache#4842) HDDS-8885. Quota repair count enable quota feature for old bucket/volume. (apache#4941) HDDS-8771. Refactor volume level tmp directory for generic usage. (apache#4838) HDDS-8922. Random EC read pipeline ID causes XceiverClient cache churn (apache#4971)

…ache#4838)

errose28 added 13 commits June 5, 2023 13:17

Refactor volume level tmp dir so it can be used for disk test files

34f6e5a

Consolidate volume shutdown

1b3375d

More refactoring of volume tmp dir

0d15e8a

Change tmpdir to File

8572b34

Change was lost in the cherry-pick process

Another compilation fix after cherry-pick

7dced52

Update TestContainerPersistence and fix directory create bug

442dde9

Refactor working dir and subdirs on startup

04384c1

Subdirs will now be created on upgraded clusters as well.

Rename dir delete_container_service -> deleted-containers

72c0d31

Move tmp dir cleanup inside HddsVolume

b12bcd2

Update existing tests

44dbd47

Add startup/shutdown tests to TestHddsVolume

908ee85

Checkstyle

e38ec9c

Fix assignment to tmp dir name fields

4ceff35

errose28 mentioned this pull request Jun 9, 2023

HDDS-8782. Improve Volume Scanner Health checks. #4867

Merged

3 tasks

errose28 added 3 commits June 9, 2023 17:02

Checkstyle

d516b19

Restore endpoint test

0369054

errose28 requested a review from sumitagrawl June 13, 2023 03:14

sumitagrawl reviewed Jun 14, 2023

View reviewed changes

...-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/StorageVolumeUtil.java Outdated Show resolved Hide resolved

...op-hdds/container-service/src/test/java/org/apache/hadoop/ozone/TestHddsDatanodeService.java Show resolved Hide resolved

Separate DB store init from tmp dir creation

1b20307

errose28 marked this pull request as ready for review June 20, 2023 06:09

errose28 requested a review from sumitagrawl June 20, 2023 06:09

sumitagrawl approved these changes Jun 22, 2023

View reviewed changes

errose28 added 2 commits June 24, 2023 15:46

Fix SCM HA finalization compat test

d579586

sumitagrawl merged commit 956385a into apache:master Jun 26, 2023

vtutrinov pushed a commit to Cyrill/ozone that referenced this pull request Jul 3, 2023

HDDS-8771. Refactor volume level tmp directory for generic usage. (ap…

2df8ccf

…ache#4838)

vtutrinov pushed a commit to Cyrill/ozone that referenced this pull request Jul 3, 2023

HDDS-8771. Refactor volume level tmp directory for generic usage. (ap…

7ee48e2

…ache#4838)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-8771. Refactor volume level tmp directory for generic usage. #4838

HDDS-8771. Refactor volume level tmp directory for generic usage. #4838

Uh oh!

errose28 commented Jun 7, 2023 •

edited

Loading

Uh oh!

sumitagrawl left a comment

Uh oh!

Uh oh!

Uh oh!

sumitagrawl left a comment

Uh oh!

errose28 commented Jun 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

HDDS-8771. Refactor volume level tmp directory for generic usage. #4838

HDDS-8771. Refactor volume level tmp directory for generic usage. #4838

Uh oh!

Conversation

errose28 commented Jun 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

errose28 commented Jun 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

errose28 commented Jun 7, 2023 •

edited

Loading