HDDS-8326. Container Level Info - OM DB Insights. #4509

devmadhuu · 2023-03-31T13:31:10Z

Container Level Info (Tab1): A new API will present the container level info showing mismatch between SCM and OM. Some containers are deleted in SCM but seem to be referred to by files/keys in OM, So recon needs to detect if any container is missing or not from OM perspective. This is a data loss situation. This API will be consumed by Recon new tab/page. This API will only identify and list such containers along with number of keys. Recon UI can later query "api/v1/containers//keys" to get details of keys mapped to such containers.

Rest API Response for "api/v1/containers/mismatch"

[
  {
  "containerId" : 1,
  "numberOfKeys" : 3,
  "pipelines" : [ 
        pipelineId: 1423ghjds832403232,
	pipelineId: 32vds94943fsdh4443,
	pipelineId: 32vds94943fsdhs443
   ],
  "existsAt" : "OM"
}
]

Here in this tab, below list of containers will be shown along with their key mappings.

So this PR covers identifying and listing of following containers:

Containers are present in OM, but not in SCM.
Containers are present in SCM, but not in OM.

https://issues.apache.org/jira/browse/HDDS-8326

How was this patch tested?

This patch was tested using Junit test cases.

devmadhuu · 2023-03-31T13:32:01Z

@errose28, @smengcl - pls review.

sumitagrawl

@devmadhuu Thanks for working over this, please check few minor comments.

sumitagrawl · 2023-04-04T10:53:48Z

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/ContainerEndpoint.java

+            pipelines.add(pipelineManager.getPipeline(pipelineID));
+          }
+        } catch (ContainerNotFoundException e) {
+          throw new RuntimeException(e);


For containerNotFound exception, we can log and continue as parallel can be removed from SCM. No need throw exeption

@sumitagrawl - pls have re-look, Changes done and pushed.

sumitagrawl · 2023-04-04T10:54:23Z

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/ContainerEndpoint.java

+        } catch (ContainerNotFoundException e) {
+          throw new RuntimeException(e);
+        } catch (PipelineNotFoundException e) {
+          throw new RuntimeException(e);


For pipelineNotFound exception, pipeline may have been closed, so we can ignore this with debug log, and continue reporting this.

@sumitagrawl - pls have re-look, Changes done and pushed.

sumitagrawl · 2023-04-04T10:54:50Z

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/ContainerEndpoint.java

+
+  @GET
+  @Path("insights/containermismatch")
+  public Response getContainerInsights() {


method name can be changed, getContainerMismatchInsights

@sumitagrawl - pls have re-look, Changes done and pushed.

krishnaasawa1 · 2023-04-17T09:37:18Z

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/ContainerEndpoint.java

  }
+
+  @GET
+  @Path("insights/containermismatch")


path can be /containers/mismatch which aligns with current container APIs. In API "insights" is not required.

Changes done and pushed

krishnaasawa1 · 2023-04-17T09:38:12Z

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/ContainerEndpoint.java

+    try {
+      Map<Long, ContainerMetadata> omContainers =
+          reconContainerMetadataManager.getContainers(-1, -1);
+      List<Long> scmContainers = containerManager.getContainers().stream()


Assume this scmContainers are which are is not in DELETED state?

Yes, this is SCM container list irrespective of their state.

1-In this case all SCM Containers when compared with OM Containers , will show all Deleted State containers as discrepancy i.e missing in OM
2-Also OM will never report any container "available in OM but missing in SCM" as deleted container is still in SCM list.
We need to have only Non-Deleted State SCM containers.
@sumitagrawl can you help check

@devmadhuu @krishnaasawa1 Yes, we need consider non-deleted state of containers. This is like a dataloss only if OM have referred container but that container is not present / deleted.

OM Containers
@sumitagrawl @krishnaasawa1 first point is fixed where "Missing in OM and present in SCM" case.

krishnaasawa1 · 2023-04-17T09:41:45Z

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/ContainerEndpoint.java

+              .collect(
+                  Collectors.toList());
+
+      notSCMContainers.forEach(nonSCMContainer -> {


notSCMContainers List will be empty in normal scenario and only if some thing went wrong will have it populated.
Here we need to check notSCMContainers size is not zero then only do further steps.

foreach will not execute if list is empty, so no check required.

krishnaasawa1 · 2023-04-17T09:42:41Z

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/ContainerEndpoint.java

+          .collect(Collectors.toList());
+
+      List<Pipeline> pipelines = new ArrayList<>();
+      nonOMContainers.forEach(nonOMContainerId -> {


Here again need a check if nonOMContainers empty

foreach will not execute if list is empty, so no check required.

sumitagrawl

@devmadhuu Thanks working over this, LGTM

sumitagrawl

@devmadhuu Given one minor comment

sumitagrawl · 2023-04-28T11:28:54Z

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/ContainerEndpoint.java

+              HddsProtos.LifeCycleState.DELETED))
+          .map(containerInfo -> containerInfo.getContainerID()).collect(
+              Collectors.toList());
+      List<Long> scmNonDeletedContainers =


scmNonDeletedContainers and scmAllContainers are same, duplicate

scmNonDeletedContainers and scmAllContainers are same, duplicate

Its fixed. Pls re-review.

krishnaasawa1 · 2023-05-05T10:25:01Z

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/ContainerEndpoint.java

+      // Filter list of container Ids which are present in OM but not in SCM.
+      List<Map.Entry<Long, ContainerMetadata>> notSCMContainers =
+          omContainers.entrySet().stream().filter(containerMetadataEntry ->
+                  !(scmAllContainers.contains(containerMetadataEntry.getKey())))


This still looks incorrect. If a container has got Deleted in SCM and still present in OM, it will never get flagged as scmAllContainers has Deleted SCM containers as well. so no discrepancy will ever get reported.
you need to have scmNonDeletedContainers here as well instead of scmAllContainers. What we are trying to figure out is container is still referred in OM but in SCM that container has got deleted.
@devmadhuu @sumitagrawl

This still looks incorrect. If a container has got Deleted in SCM and still present in OM, it will never get flagged as scmAllContainers has Deleted SCM containers as well. so no discrepancy will ever get reported. you need to have scmNonDeletedContainers here as well instead of scmAllContainers. What we are trying to figure out is container is still referred in OM but in SCM that container has got deleted. @devmadhuu @sumitagrawl

IMO, we want to identify mismatch of container referred in OM but present in SCM, now why the container is in deleted state in SCM, that is different issue. But any container which cannot be queried for metadata by OM to SCM is a data loss situation. @sumitagrawl - I think your 1st point is what we discussed. Let us know what discussion happened with @errose28

@devmadhuu We should not count deleted SCM container for comparing to OM also, deleted container is just a bookmark, to avoid re-created same container at DN due to issue with sync at DN or Old DN.

This needs to be removed in comparison here tool.

sumitagrawl

@devmadhuu Need avoid deleted container in comparision.

sumitagrawl · 2023-05-08T03:47:20Z

hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/ContainerEndpoint.java

+      // Filter list of container Ids which are present in OM but not in SCM.
+      List<Map.Entry<Long, ContainerMetadata>> notSCMContainers =
+          omContainers.entrySet().stream().filter(containerMetadataEntry ->
+                  !(scmAllContainers.contains(containerMetadataEntry.getKey())))


@devmadhuu We should not count deleted SCM container for comparing to OM also, deleted container is just a bookmark, to avoid re-created same container at DN due to issue with sync at DN or Old DN.

This needs to be removed in comparison here tool.

devmadhuu · 2023-05-08T04:43:23Z

Containers are present in OM, but not in SCM.

Containers are present in SCM, but not in OM.

Updated the patch, pls review

ChenSammi · 2023-05-09T09:38:53Z

...ne/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/ContainerDiscrepancyInfo.java

+
+  @JsonProperty("scmContainerState")
+  @JsonInclude(JsonInclude.Include.NON_EMPTY)
+  private long scmContainerState;


It looks like scmContainerState and omContainerState are not used in anywhere.

It looks like scmContainerState and omContainerState are not used in anywhere.

Thanks for pointing out Sammi, I have removed them, earlier I was thinking to use, but I don't have a mechanism to for states of mismatched containers. Pls review again.

HDDS-8326. Container Level Info - OM DB Insights.

f4bc318

devmadhuu marked this pull request as draft March 31, 2023 14:35

deveshsingh added 3 commits March 31, 2023 21:46

HDDS-8326. Container Level Info - OM DB Insights.

caf38d0

HDDS-8326. Container Level Info - OM DB Insights.

de7d1cd

HDDS-8326. Container Level Info - OM DB Insights.

b6ff464

devmadhuu marked this pull request as ready for review March 31, 2023 16:25

devmadhuu marked this pull request as draft April 3, 2023 04:05

HDDS-8326. Container Level Info - OM DB Insights.

950b756

devmadhuu marked this pull request as ready for review April 3, 2023 05:45

adoroszlai added the recon label Apr 3, 2023

kerneltime requested a review from ChenSammi April 3, 2023 16:24

sumitagrawl requested changes Apr 4, 2023

View reviewed changes

deveshsingh added 2 commits April 5, 2023 20:27

HDDS-8326. Container Level Info - OM DB Insights.

f658127

HDDS-8326. Container Level Info - OM DB Insights.

2fd3e9d

krishnaasawa1 reviewed Apr 17, 2023

View reviewed changes

sumitagrawl approved these changes Apr 17, 2023

View reviewed changes

deveshsingh added 2 commits April 24, 2023 09:54

HDDS-8326. Container Level Info - OM DB Insights.

5400a71

HDDS-8326. Container Level Info - OM DB Insights.

f15b19c

sumitagrawl reviewed Apr 28, 2023

View reviewed changes

deveshsingh added 2 commits April 28, 2023 19:55

HDDS-8326. Container Level Info - OM DB Insights.

f664544

HDDS-8326. Container Level Info - OM DB Insights.

5fc7e78

krishnaasawa1 reviewed May 5, 2023

View reviewed changes

sumitagrawl reviewed May 8, 2023

View reviewed changes

HDDS-8326. Container Level Info - OM DB Insights.

dc93c63

krishnaasawa1 approved these changes May 8, 2023

View reviewed changes

ChenSammi reviewed May 9, 2023

View reviewed changes

HDDS-8326. Container Level Info - OM DB Insights.

13639c0

ChenSammi approved these changes May 10, 2023

View reviewed changes

ChenSammi merged commit e8fb174 into apache:master May 10, 2023

smitajoshi12 mentioned this pull request Jun 20, 2023

HDDS-8415. Recon UI:OM DB Insights Changes #4940

Merged

HDDS-8326. Container Level Info - OM DB Insights. #4509

HDDS-8326. Container Level Info - OM DB Insights. #4509

Uh oh!

Conversation

devmadhuu commented Mar 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How was this patch tested?

Uh oh!

devmadhuu commented Mar 31, 2023

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devmadhuu commented May 8, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

devmadhuu commented Mar 31, 2023 •

edited

Loading