HDDS-13238. Scan container and volume in container import & export, EC reconstruction, reconciliation

[Tejaskriya]

What changes were proposed in this pull request?

This PR adds container scans in the following cases:

• EC reconstruction: for success
• Reconciler: for success and failure
• Export container: for failure

And volume scan is triggered during the following cases:

• Import container : for failure

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13238

How was this patch tested?

Added unit tests

[errose28]

In this PR, let's also update this location which gets triggered when the container file cannot be updated. We should do an on-demand container scan first, and only a volume scan if that fails. However, we don't want to trigger the on-demand container scan in this exact spot in case this is being called to mark the container unhealthy, or it will get stuck in a loop. Instead we probably want to trigger it here.

Note that if a container scan fails due to permission errors, the result will only be kept in memory. Persistence will fail since we don't have permission to update the file, but that is desirable since permissions can be restored on restart. Let's add a comment explaining that in this scan location as well.

[errose28]

Thanks for adding this @Tejaskriya. Can you address the merge conflict? I think the reconciliation test will need to be updated after #8908 because the failure being injected is in peer communication, which no longer causes a complete failure of reconciliation. Instead you could introduce a local issue like removing the container's directory from the disk before trying to reconcile, which should cause early calls like this to fail.

[Tejaskriya]

@errose28 I have made the changes as described in your review comments above, please take a look again, thanks!

[errose28]

Prod code LGTM, just minor comments on tests.

[errose28]

Thanks for adding this @Tejaskriya.