HDDS-13092. Container scanner should trigger volume scan when marking a container unhealthy

[Tejaskriya]

What changes were proposed in this pull request?

If any of the container scanners (background or on-demand, data or metadata) find an unhealthy container, they should trigger an on-demand volume scan to check if the underlying volume has a larger issue beyond that container.

This PR triggers a volume check for when a container is marked unhealthy by the scanners

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13092

How was this patch tested?

Added unit test coverage

[Tejaskriya]

@ptlrs Could you please review this PR?

[adoroszlai]

nit: any scan "triggered" by some event is by definition an "on-demand" scan (vs. background scan, which runs on schedule). So can we omit "on-demand" from the title?

[Tejaskriya]

@adoroszlai makes sense, I'll change the title. Thanks!

[aryangupta1998]

Thanks for the patch @Tejaskriya.
IIUC, containers of a volume are scanned in a single iteration. If a volume has multiple unhealthy containers, we may end up triggering a volume scan multiple times for the same volume using:
StorageVolumeUtil.onFailure(containerData.getVolume());
To avoid redundant volume scans, we can track which volumes have already been scanned in the current iteration. One way is to pass a Set to scanContainer():

public void scanContainer(Container<?> c, Set<Path> volumesAlreadyChecked)

Then, within scanContainer(), only trigger the volume scan if it hasn't already been triggered:

Path volumePath = containerData.getVolume().getStorageDir().getPath();
if (volumesAlreadyChecked.add(volumePath)) {
    LOG.info("Triggering a volume scan for volume [{}] as unhealthy container [{}] was on it.",
        volumePath, containerId);
    StorageVolumeUtil.onFailure(containerData.getVolume());
}

[Tejaskriya]

@aryangupta1998 in the scheduling logic, we have throttling. In org.apache.hadoop.ozone.container.common.volume.ThrottledAsyncChecker#schedule, if a request for multiple scans of the same volume comes within a timeframe, then it is skipped.
This durations seems to be defaulting to 10m:
@Config(key = "disk.check.min.gap", defaultValue = "10m",

[Tejaskriya]

@ptlrs requesting a review for this PR

[ptlrs]

Thanks @Tejaskriya for the PR, it mostly looks good.

[Tejaskriya]

Thanks for the review and approval @ptlrs . I have made the changes.
@aryangupta1998 could you please review and approve the PR if it seems good?

[aryangupta1998]

LGTM!

[Tejaskriya]

@ptlrs @errose28 @aryangupta1998 could you please review the patch again?

[aryangupta1998]

Thanks for updating the patch @Tejaskriya, LGTM!

[errose28]

Just one minor comment here and in this thread then I think this is good to go.

[ptlrs]

Thanks for the updates @Tejaskriya.

[errose28]

Thanks for working on this @Tejaskriya

[adoroszlai]

Please feel free remove co-author information when the only contribution is merging master into the PR branch.