Skip to content

HDDS-6764: EC: DN ability to create RECOVERING containers for EC reconstruction. #3458

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
umamaheswararao merged 3 commits into from
Jun 2, 2022
Merged

HDDS-6764: EC: DN ability to create RECOVERING containers for EC reconstruction. #3458

umamaheswararao merged 3 commits into from
Jun 2, 2022

Conversation

@umamaheswararao
Copy link
Contributor

@umamaheswararao umamaheswararao commented May 26, 2022

What changes were proposed in this pull request?

This PR proposes to add the createREcoveringContainer API in ContainerProtocolCalls, which can be used to store reconstruction in progress containers.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-6764

How was this patch tested?

Added tests to verify.

adoroszlai
adoroszlai reviewed Jun 1, 2022
View reviewed changes
Copy link
Contributor

@adoroszlai adoroszlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @umamaheswararao for the patch.

hadoop-hdds/interface-client/src/main/proto/DatanodeClientProtocol.proto Outdated
Copy link
Contributor

@adoroszlai adoroszlai Jun 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add this?

Copy link
Contributor Author

@umamaheswararao umamaheswararao Jun 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed it. Below state is there.

hadoop-hdds/common/src/test/java/org/apache/hadoop/ozone/container/ContainerTestHelper.java Outdated
Copy link
Contributor

@adoroszlai adoroszlai Jun 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is eof=true needed here? I think caller can set it later.

Copy link
Contributor Author

@umamaheswararao umamaheswararao Jun 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I think I added this while testing.

...iner-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java Outdated
Comment on lines 477 to 478
Copy link
Contributor

@adoroszlai adoroszlai Jun 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to add a method for handling OPEN and RECOVERING state the same way in specific cases? (I'm not sure what a good name for the method might be, though.)

Copy link
Contributor Author

@umamaheswararao umamaheswararao Jun 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried to export into a method. Let's see if that make sense.

@umamaheswararao
Copy link
Contributor Author

umamaheswararao commented Jun 2, 2022

Thanks @adoroszlai for the review. Please check the latest updates.

adoroszlai
adoroszlai approved these changes Jun 2, 2022
View reviewed changes
Copy link
Contributor

@adoroszlai adoroszlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @umamaheswararao for updating the patch, LGTM.

@adoroszlai
Copy link
Contributor

adoroszlai commented Jun 2, 2022 •
edited
Loading

@umamaheswararao it looks like TestOzoneECClient is failing consistently in CI. Tried it locally, no issues.

https://github.com/apache/ozone/runs/6702568728#step:5:1644
https://github.com/apache/ozone/runs/6705930506#step:5:1654
https://github.com/apache/ozone/runs/6709532641#step:5:1654
https://github.com/apache/ozone/runs/6710406479#step:5:1654
https://github.com/apache/ozone/runs/6711102895#step:5:1654

@umamaheswararao
Copy link
Contributor Author

umamaheswararao commented Jun 2, 2022

That tests got added recently. I don't think it's related, but there is some flakeyness in the tests. Tests were added from this HDDS-6808

cc: @kaijchen
Should we revert that tests until we figure out the cause?

@kaijchen
Copy link
Member

kaijchen commented Jun 2, 2022

cc: @kaijchen
Should we revert that tests until we figure out the cause?

I hope #3476 fixes the problem. If not, we can add a @Ignore tag to it.

@umamaheswararao
Copy link
Contributor Author

umamaheswararao commented Jun 2, 2022

I have just rebased with that change. Let's see. If this reappears, then I will add @ignore tag.

@umamaheswararao umamaheswararao merged commit 88f4101 into apache:master Jun 2, 2022
@umamaheswararao
Copy link
Contributor Author

umamaheswararao commented Jun 2, 2022

Thanks @adoroszlai for the reviews!

errose28 added a commit to errose28/ozone that referenced this pull request Jun 7, 2022
@errose28
Merge branch 'master' into HDDS-6760
049d7bc 
* master: (87 commits)
  HDDS-6686. Do Leadship check before SASL token verification. (apache#3382)
  HDDS-4364: [FSO]List FileStatus : startKey can be a non-existed path (apache#3481)
  HDDS-6091. Add file checksum to OmKeyInfo (apache#3201)
  HDDS-6706. Exposing Volume Information Metrics to the DataNode UI (apache#3478)
  HDDS-6759: Add listblock API in MockDatanodeStorage (apache#3452)
  HDDS-5821 Container cache management for closing RockDB  (apache#3426)
  HDDS-6683. Refactor OM server bucket layout configuration usage (apache#3477)
  HDDS-6824. Revert changes made in proto.lock by HDDS-6768. (apache#3480)
  HDDS-6811. Bucket create message with layout type (apache#3479)
  HDDS-6810. Add a optional flag to trigger listStatus as part of listKeys for FSO buckets. (apache#3461)
  HDDS-6828. Revert RockDB version pending leak fixes (apache#3475)
  HDDS-6764: EC: DN ability to create RECOVERING containers for EC reconstruction. (apache#3458)
  HDDS-6795: EC: PipelineStateMap#addPipeline should not have precondition checks post db updates (apache#3453)
  HDDS-6823. Intermittent failure in TestOzoneECClient#testExcludeOnDNMixed (apache#3476)
  HDDS-6820. Bucket Layout Post-Finalization Validators for ACL Requests. (apache#3472)
  HDDS-6819. Add LEGACY to AllowedBucketLayouts in CreateBucketHandler (apache#3473)
  HDDS-4859. [FSO]ListKeys: seek all the files/dirs from startKey to keyPrefix (apache#3466)
  HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved (apache#3430)
  HDDS-6474. Add test to cover the FSO bucket list status with beyond batch boundary and cache. (apache#3379). Contributed by aswinshakil
  HDDS-6280. Support Container Balancer HA (apache#3423)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adoroszlai adoroszlai adoroszlai approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@umamaheswararao @adoroszlai @kaijchen