Skip to content

HDDS-3773. Add OMDBDefinition to define structure of om.db. #1076

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
elek merged 2 commits into from
Jun 25, 2020
Merged

HDDS-3773. Add OMDBDefinition to define structure of om.db. #1076

elek merged 2 commits into from
Jun 25, 2020

Conversation

@sadanand48
Copy link
Contributor

@sadanand48 sadanand48 commented Jun 15, 2020

What changes were proposed in this pull request?

In order to support the rocksdb tool which displays data for a given db file (HDDS-3622) , the appropriate DBDefinition file must be defined. The change here is to extend the tool for om.db.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-3773

How was this patch tested?

Manual test.

@codecov-commenter
Copy link

codecov-commenter commented Jun 15, 2020

Codecov Report

Merging #1076 into master will decrease coverage by 0.03%.
The diff coverage is 0.00%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #1076      +/-   ##
============================================
- Coverage     70.46%   70.43%   -0.04%     
- Complexity     9242     9247       +5     
============================================
  Files           961      963       +2     
  Lines         48130    48155      +25     
  Branches       4676     4675       -1     
============================================
+ Hits          33917    33920       +3     
- Misses        11960    11985      +25     
+ Partials       2253     2250       -3
Impacted Files Coverage Δ Complexity Δ
...g/apache/hadoop/ozone/om/codec/OMDBDefinition.java 0.00% <0.00%> (ø) 0.00 <0.00> (?)
...apache/hadoop/ozone/debug/DBDefinitionFactory.java 0.00% <0.00%> (ø) 0.00 <0.00> (?)
.../java/org/apache/hadoop/ozone/debug/DBScanner.java 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...hdds/scm/container/CloseContainerEventHandler.java 72.41% <0.00%> (-17.25%) 6.00% <0.00%> (ø%)
...er/common/transport/server/GrpcXceiverService.java 70.00% <0.00%> (-10.00%) 3.00% <0.00%> (ø%)
...va/org/apache/hadoop/hdds/utils/db/RDBMetrics.java 85.71% <0.00%> (-7.15%) 13.00% <0.00%> (-1.00%)
...e/commandhandler/CreatePipelineCommandHandler.java 87.23% <0.00%> (-4.26%) 8.00% <0.00%> (ø%)
...apache/hadoop/hdds/server/events/EventWatcher.java 77.77% <0.00%> (-4.17%) 14.00% <0.00%> (ø%)
...ent/algorithms/SCMContainerPlacementRackAware.java 76.69% <0.00%> (-3.01%) 31.00% <0.00%> (-2.00%)
...java/org/apache/hadoop/hdds/utils/db/RDBTable.java 58.66% <0.00%> (-2.67%) 19.00% <0.00%> (-1.00%)
... and 16 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 57a8388...c27fd85. Read the comment docs.

@avijayanhwx
Copy link
Contributor

avijayanhwx commented Jun 15, 2020

Thank you for the patch @sadanand48. Any reason why we are not using the OMDBDefinition to actually create the OM DB, like in the SCM?

@sadanand48
Copy link
Contributor Author

sadanand48 commented Jun 16, 2020

Thank you for the patch @sadanand48. Any reason why we are not using the OMDBDefinition to actually create the OM DB, like in the SCM?

@avijayanhwx I think the plan was to use DBStore interface instead of the MetadataDataStore and in order to do that DBDefinition must be defined [(HDDS-3172)] .(#700 (comment)).
OM currently uses the MetadataStore Interface so there is no OMDBDefinition defined.

@bshashikant
Copy link
Contributor

bshashikant commented Jun 17, 2020

Thank you for the patch @sadanand48. Any reason why we are not using the OMDBDefinition to actually create the OM DB, like in the SCM?

@avijayanhwx I think the plan was to use DBStore interface instead of the MetadataDataStore and in order to do that DBDefinition must be defined [(HDDS-3172)] .(#700 (comment)).
OM currently uses the MetadataStore Interface so there is no OMDBDefinition defined.

I think we probably need to follow the same model as that in SCM. Though the patch looks good, other changes related to using DB interface rather than MetadataDataStore interface, can be done in separate jira after further analysis.

@avijayanhwx , what do you think?

@elek
Copy link
Member

elek commented Jun 24, 2020

I think we probably need to follow the same model as that in SCM. Though the patch looks good, other changes related to using DB interface rather than MetadataDataStore interface, can be done in separate jira after further analysis.

+1 Sounds like a reasonable plan for me.

@avijayanhwx
Copy link
Contributor

avijayanhwx commented Jun 24, 2020

@sadanand48 / @bshashikant I am OK with handling the next set of OM changes in a different JIRA.

@avijayanhwx avijayanhwx self-requested a review June 24, 2020 17:05
@elek
Copy link
Member

elek commented Jun 25, 2020

Great. I think the proposal is to do something similar to #997 in the next patch.

Now I am merging this one. Thanks the patch @sadanand48 and the review @avijayanhwx and @bshashikant

@elek elek merged commit 5e7b2b6 into apache:master Jun 25, 2020
errose28 added a commit to errose28/ozone that referenced this pull request Jun 25, 2020
@errose28
Merge remote-tracking branch 'upstream/master'
615f58c 
* upstream/master: (56 commits)
  HDDS-3264. Fix TestCSMMetrics.java. (apache#1120)
  HDDS-3858. Remove support to start Ozone and HDFS datanodes in the same JVM (apache#1117)
  HDDS-3704. Update all the documentation to use ozonefs-hadoop2/3 instead of legacy/current (apache#1099)
  HDDS-3773. Add OMDBDefinition to define structure of om.db. (apache#1076)
  Revert "HDDS-3263. Fix TestCloseContainerByPipeline.java. (apache#1119)" (apache#1126)
  HDDS-3821. Disable Ozone SPNEGO should not fall back to hadoop.http.a… (apache#1101)
  HDDS-3819. OzoneManager#listVolumeByUser ignores userName parameter when ACL is enabled (apache#1087)
  HDDS-3779. Add csi interface documents to show how to use ozone csi (apache#1059)
  HDDS-3857. Datanode in compose/ozonescripts can't be started (apache#1116)
  HDDS-3430. Enable TestWatchForCommit test cases. (apache#1114)
  HDDS-3263. Fix TestCloseContainerByPipeline.java. (apache#1119)
  HDDS-3512. s3g multi-part-upload saved incorrect content using streaming (apache#1092)
  HDDS-3836. Modify ContainerPlacementPolicyFactory JavaDoc (apache#1097)
  HDDS-3780. Replace the imagePullPolicy from always to IfNotPresent (apache#1055)
  HDDS-3847. Change OMNotLeaderException logging to DEBUG (apache#1118)
  HDDS-3745. Improve OM and SCM performance with 64% by avoid collect datanode information to s3g (apache#1031)
  HDDS-3286. BasicOzoneFileSystem  support batchDelete. (apache#814)
  HDDS-3850. Update the admin document to let user know how to show the status of all rules. (apache#1109)
  HDDS-3848. Add ratis.thirdparty.version in main pom.xml (apache#1108)
  HDDS-3815. Avoid buffer copy in ContainerCommandRequestProto. (apache#1085)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@avijayanhwx avijayanhwx avijayanhwx approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@sadanand48 @codecov-commenter @avijayanhwx @bshashikant @elek