Skip to content

HDDS-8411. Show the total number of CLOSING, QUASI_CLOSED, CLOSED, DELETING Containers in Recon #4591

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
targetoee wants to merge 1 commit into from
Closed

HDDS-8411. Show the total number of CLOSING, QUASI_CLOSED, CLOSED, DELETING Containers in Recon #4591

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/ClusterStateEndpoint.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,22 @@ public Response getClusterState() {
this.containerManager.getContainerStateCount(
HddsProtos.LifeCycleState.OPEN));

containerStateCounts.setClosingContainersCount(
this.containerManager.getContainerStateCount(
HddsProtos.LifeCycleState.CLOSING));

containerStateCounts.setQuasiClosedContainersCount(
this.containerManager.getContainerStateCount(
HddsProtos.LifeCycleState.QUASI_CLOSED));

containerStateCounts.setClosedContainersCount(
this.containerManager.getContainerStateCount(
HddsProtos.LifeCycleState.CLOSED));

containerStateCounts.setDeletingContainersCount(
this.containerManager.getContainerStateCount(
HddsProtos.LifeCycleState.DELETING));

containerStateCounts.setDeletedContainersCount(
this.containerManager.getContainerStateCount(
HddsProtos.LifeCycleState.DELETED));
Expand Down Expand Up @@ -180,6 +196,12 @@ public Response getClusterState() {
.setTotalDatanodes(datanodeDetails.size())
.setHealthyDatanodes(healthyDatanodes)
.setOpenContainers(containerStateCounts.getOpenContainersCount())
.setClosingContainers(containerStateCounts.getClosingContainersCount())
.setQuasiClosedContainers(
containerStateCounts.getQuasiClosedContainersCount())
.setClosedContainers(containerStateCounts.getClosedContainersCount())
Copy link
Contributor

@devmadhuu devmadhuu Apr 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may not be needed, as difference of total and open will give us closed container count. Rest all information on container states can be easily found out using Ozone admin CLI because we don't want to expose the transitioning container states to other users, and should be exposed to Admin users only.

Copy link
Contributor

@ArafatKhan2198 ArafatKhan2198 Apr 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @devmadhuu
@sumitagrawl has suggested that we include additional states in our state counts, such as close, closing, and quasi_closed. The state closing is especially important and can provide valuable insights into the situation being analyzed.

Copy link
Contributor

@devmadhuu devmadhuu Apr 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ArafatKhan2198 , these state transitions are too dynamic and quick, let's discuss this, not sure if in so dynamic transitions, if it is really useful for admins. Moreover these are directly queried using admin CLI, so better we should avoid such data and load recon

Copy link
Contributor

@devmadhuu devmadhuu Apr 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also closed is not needed because it can be derived from difference

Copy link
Contributor

@sumitagrawl sumitagrawl Apr 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, Quasi-Closed can be important state having special meaning. Also Closing / Deleting if stuck can be identified easily from this perspective.

Copy link

@krishnaasawa1 krishnaasawa1 May 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes in normal scenarios transition will be quick, but have seen large number of containers stuck and remaining in Closing state in some scenarios. So should be useful for analyzing, if easily visible. But we should be careful of cluttering overview page and should have at appropriate place.

.setDeletingContainers(
containerStateCounts.getDeletingContainersCount())
.setDeletedContainers(containerStateCounts.getDeletedContainersCount())
Copy link
Contributor

@devmadhuu devmadhuu Apr 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be duplicate of PR for HDDS-8127

Copy link
Contributor

@sumitagrawl sumitagrawl Apr 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deleted Container and Deleting container shows different info ... and not to be excluded from container count. Only Deleted Container needs to be excluded,

.build();
return Response.ok(response).build();
Expand Down
71 changes: 71 additions & 0 deletions ...one/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/ClusterStateResponse.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,30 @@ public final class ClusterStateResponse {
@JsonProperty("openContainers")
private int openContainers;

/**
* Total count of closing containers in the cluster.
*/
@JsonProperty("closingContainers")
private int closingContainers;

/**
* Total count of quasi closed containers in the cluster.
*/
@JsonProperty("quasiClosedContainers")
private int quasiClosedContainers;

/**
* Total count of closed containers.
*/
@JsonProperty("closedContainers")
private int closedContainers;

/**
* Total count of deleting containers.
*/
@JsonProperty("deletingContainers")
private int deletingContainers;

/**
* Total count of deleted containers in the cluster.
*/
Expand Down Expand Up @@ -124,6 +148,10 @@ private ClusterStateResponse(Builder b) {
this.openContainers = b.openContainers;
this.deletedKeys = b.deletedKeys;
this.deletedDirs = b.deletedDirs;
this.closingContainers = b.closingContainers;
this.quasiClosedContainers = b.quasiClosedContainers;
this.closedContainers = b.closedContainers;
this.deletingContainers = b.deletingContainers;
this.deletedContainers = b.deletedContainers;
}

Expand All @@ -139,6 +167,10 @@ public static final class Builder {
private int containers;
private int missingContainers;
private int openContainers;
private int closingContainers;
private int quasiClosedContainers;
private int closedContainers;
private int deletingContainers;
private int deletedContainers;
private long volumes;
private long buckets;
Expand All @@ -151,6 +183,10 @@ public Builder() {
this.containers = 0;
this.missingContainers = 0;
this.openContainers = 0;
this.closingContainers = 0;
this.quasiClosedContainers = 0;
this.closedContainers = 0;
this.deletingContainers = 0;
this.deletedContainers = 0;
this.volumes = 0;
this.buckets = 0;
Expand Down Expand Up @@ -197,6 +233,25 @@ public Builder setOpenContainers(int openContainers) {
return this;
}

public Builder setClosingContainers(int closingContainers) {
this.closingContainers = closingContainers;
return this;
}

public Builder setQuasiClosedContainers(int quasiClosedContainers) {
this.quasiClosedContainers = quasiClosedContainers;
return this;
}
public Builder setClosedContainers(int closedContainers) {
this.closedContainers = closedContainers;
return this;
}

public Builder setDeletingContainers(int deletingContainers) {
this.deletingContainers = deletingContainers;
return this;
}

public Builder setDeletedContainers(int deletedContainers) {
this.deletedContainers = deletedContainers;
return this;
Expand Down Expand Up @@ -264,6 +319,22 @@ public int getOpenContainers() {
return openContainers;
}

public int getClosingContainers() {
return closingContainers;
}

public int getQuasiClosedContainers() {
return quasiClosedContainers;
}

public int getClosedContainers() {
return closedContainers;
}

public int getDeletingContainers() {
return deletingContainers;
}

public int getDeletedContainers() {
return deletedContainers;
}
Expand Down
36 changes: 36 additions & 0 deletions ...one/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/ContainerStateCounts.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ public class ContainerStateCounts {
private int totalContainerCount;
private int missingContainerCount;
private int openContainersCount;
private int closingContainersCount;
private int quasiClosedContainersCount;
private int closedContainersCount;
private int deletingContainersCount;
private int deletedContainersCount;

public int getTotalContainerCount() {
Expand All @@ -52,6 +56,38 @@ public void setOpenContainersCount(int openContainersCount) {
this.openContainersCount = openContainersCount;
}

public int getClosingContainersCount() {
return closingContainersCount;
}

public void setClosingContainersCount(int closingContainersCount) {
this.closingContainersCount = closingContainersCount;
}

public int getQuasiClosedContainersCount() {
return quasiClosedContainersCount;
}

public void setQuasiClosedContainersCount(int quasiClosedContainersCount) {
this.quasiClosedContainersCount = quasiClosedContainersCount;
}

public int getClosedContainersCount() {
return closedContainersCount;
}

public void setClosedContainersCount(int closedContainersCount) {
this.closedContainersCount = closedContainersCount;
}

public int getDeletingContainersCount() {
return deletingContainersCount;
}

public void setDeletingContainersCount(int deletingContainersCount) {
this.deletingContainersCount = deletingContainersCount;
}

public int getDeletedContainersCount() {
return deletedContainersCount;
}
Expand Down
24 changes: 20 additions & 4 deletions ...ne/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/overview/overview.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ interface IClusterStateResponse {
buckets: number;
keys: number;
openContainers: number;
deletingContainers: number;
}

interface IOverviewState {
Expand All @@ -58,6 +59,10 @@ interface IOverviewState {
lastUpdatedOMDBFull: number;
omStatus: string;
openContainers: number;
closingContainers: number,
quasiClosedContainers: number,
closedContainers: number,
deletingContainers: number;
}

export class Overview extends React.Component<Record<string, object>, IOverviewState> {
Expand All @@ -84,7 +89,11 @@ export class Overview extends React.Component<Record<string, object>, IOverviewS
lastUpdatedOMDBDelta: 0,
lastUpdatedOMDBFull: 0,
omStatus: '',
openContainers: 0
openContainers: 0,
closingContainers: 0,
quasiClosedContainers:0,
closedContainers:0,
deletingContainers: 0
};
this.autoReload = new AutoReloadHelper(this._loadData);
}
Expand Down Expand Up @@ -115,6 +124,10 @@ export class Overview extends React.Component<Record<string, object>, IOverviewS
keys: clusterState.keys,
missingContainersCount,
openContainers: clusterState.openContainers,
closingContainers: clusterState.closingContainers,
quasiClosedContainers: clusterState.quasiClosedContainers,
closedContainers: clusterState.closedContainers,
deletingContainers: clusterState.deletingContainers,
lastRefreshed: Number(moment()),
lastUpdatedOMDBDelta: omDBDeltaObject && omDBDeltaObject.lastUpdatedTimestamp,
lastUpdatedOMDBFull: omDBFullObject && omDBFullObject.lastUpdatedTimestamp
Expand Down Expand Up @@ -158,7 +171,8 @@ export class Overview extends React.Component<Record<string, object>, IOverviewS

render() {
const {loading, datanodes, pipelines, storageReport, containers, volumes, buckets,
keys, missingContainersCount, lastRefreshed, lastUpdatedOMDBDelta, lastUpdatedOMDBFull, omStatus, openContainers } = this.state;
keys, missingContainersCount, lastRefreshed, lastUpdatedOMDBDelta, lastUpdatedOMDBFull, omStatus,
openContainers, closingContainers, quasiClosedContainers, closedContainers,deletingContainers } = this.state;

const datanodesElement = (
<span>
Expand All @@ -168,6 +182,8 @@ export class Overview extends React.Component<Record<string, object>, IOverviewS
const containersTooltip = missingContainersCount === 1 ? 'container is missing' : 'containers are missing';
const containersLink = missingContainersCount > 0 ? '/MissingContainers' : '/Containers';
const duLink = '/DiskUsage';
const transitionalContainers = closingContainers + quasiClosedContainers + closedContainers + deletingContainers;
const containersElementTitle = missingContainersCount > 0 ? "Containers" : "Containers(Open/Transition)"
const containersElement = missingContainersCount > 0 ? (
<span>
<Tooltip placement='bottom' title={missingContainersCount > 1000 ? `1000+ Containers are missing. For more information, go to the Containers page.` : `${missingContainersCount} ${containersTooltip}`}>
Expand All @@ -179,7 +195,7 @@ export class Overview extends React.Component<Record<string, object>, IOverviewS
<div>
<span>{containers.toString()} </span>
<Tooltip placement='bottom' title='Number of open containers'>
<span>({openContainers})</span>
<span>({openContainers}/{transitionalContainers})</span>
</Tooltip>
</div>
const clusterCapacity = `${size(storageReport.capacity - storageReport.remaining)}/${size(storageReport.capacity)}`;
Expand Down Expand Up @@ -212,7 +228,7 @@ export class Overview extends React.Component<Record<string, object>, IOverviewS
</Col>
<Col xs={24} sm={18} md={12} lg={12} xl={6}>
<OverviewCard
loading={loading} title='Containers' data={containersElement}
loading={loading} title={containersElementTitle} data={containersElement}
icon='container'
error={missingContainersCount > 0} linkToUrl={containersLink}/>
</Col>
Expand Down
25 changes: 24 additions & 1 deletion ...ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestContainerStateCounts.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,9 @@ public class TestContainerStateCounts extends AbstractReconSqlDBTest {
private static final int NUM_OPEN_CONTAINERS = 3;
private static final int NUM_DELETED_CONTAINERS = 4;
private static final int NUM_CLOSED_CONTAINERS = 3;
private static final int NUM_QUASI_CLOSED_CONTAINERS = 5;
private static final int NUM_CLOSING_CONTAINERS = 6;
private static final int NUM_DELETING_CONTAINERS = 7;


@BeforeEach
Expand Down Expand Up @@ -117,22 +120,42 @@ public void testGetContainerCounts() throws Exception {
HddsProtos.LifeCycleState.DELETED);
putContainerInfos(NUM_CLOSED_CONTAINERS,
HddsProtos.LifeCycleState.CLOSED);
putContainerInfos(NUM_CLOSING_CONTAINERS,
HddsProtos.LifeCycleState.CLOSING);
putContainerInfos(NUM_QUASI_CLOSED_CONTAINERS,
HddsProtos.LifeCycleState.QUASI_CLOSED);
putContainerInfos(NUM_DELETING_CONTAINERS,
HddsProtos.LifeCycleState.DELETING);

// Get the cluster state using the ClusterStateEndpoint
Response response1 = clusterStateEndpoint.getClusterState();
ClusterStateResponse clusterStateResponse1 =
(ClusterStateResponse) response1.getEntity();

// Calculate expected counts
int expectedTotalContainers = NUM_OPEN_CONTAINERS + NUM_CLOSED_CONTAINERS;
int expectedTotalContainers = NUM_OPEN_CONTAINERS + NUM_CLOSED_CONTAINERS
+ NUM_CLOSING_CONTAINERS + NUM_QUASI_CLOSED_CONTAINERS
+ NUM_DELETING_CONTAINERS;
int expectedOpenContainers = NUM_OPEN_CONTAINERS;
int expectedClosingContainers = NUM_CLOSING_CONTAINERS;
int expectedQuasiClosedContainers = NUM_QUASI_CLOSED_CONTAINERS;
int expectedClosedContainers = NUM_CLOSED_CONTAINERS;
int expectedDeletingContainers = NUM_DELETING_CONTAINERS;
int expectedDeletedContainers = NUM_DELETED_CONTAINERS;

// Verify counts using assertions
Assertions.assertEquals(expectedTotalContainers,
clusterStateResponse1.getContainers());
Assertions.assertEquals(expectedOpenContainers,
clusterStateResponse1.getOpenContainers());
Assertions.assertEquals(expectedClosingContainers,
clusterStateResponse1.getClosingContainers());
Assertions.assertEquals(expectedQuasiClosedContainers,
clusterStateResponse1.getQuasiClosedContainers());
Assertions.assertEquals(expectedClosedContainers,
clusterStateResponse1.getClosedContainers());
Assertions.assertEquals(expectedDeletingContainers,
clusterStateResponse1.getDeletingContainers());
Assertions.assertEquals(expectedDeletedContainers,
clusterStateResponse1.getDeletedContainers());
}
Expand Down