Open mfdataset enchancement

[pratiman-91]

• Closes better handling of invalid files in open_mfdataset #6736
• User visible changes (including notable bug fixes) are documented in whats-new.rst

Added new argument in open_mfdataset to better handle the invalid files.

errors : {'ignore', 'raise', 'warn'}, default 'raise'
        - If 'raise', then invalid dataset will raise an exception.
        - If 'ignore', then invalid dataset will be ignored.
        - If 'warn', then a warning will be issued for each invalid dataset.

[welcome]

Thank you for opening this pull request! It may take us a few days to respond here, so thank you for being patient.
If you have questions, some answers may be found in our contributing guidelines.

[max-sixty]

I'm not the expert, but this looks reasonable! Any other thoughts?

Assuming no one thinks it's a bad idea, we would need tests.

[headtr1ck]

I think it is a good idea.

But the way it is implemented here seems overly complicated and repetitive.
I would suggest to revert the logic: first build up the list wrapped in a single try and then handle the three cases in the except block.

[headtr1ck]

This is never being raised because it is catched by the outer try-except.

Also, to keep the complexity lower, check this before entering the if else block: if errors not in ("raise", "warn", "ignore"): raise ValueError(...)

[pratiman-91]

Thanks @headtr1ck for pointing this out. I have made changes to the logic based on your suggestions.

[headtr1ck]

Almost there.

Also, we should add tests for this.

[pratiman-91]

@headtr1ck Thanks for the suggestions. I have added two tests (ignore and warn). Also, while testing, I found that a new argument broke combine="nested" due to invalid ids. I have now modified it to reflect the correct ids, and it is passing the tests. Please review the tests and the latest version.

[headtr1ck]

Does this work with nested datasets? E.g. a 2x2 set where one file is missing

[pratiman-91]

Thanks for pointing this out. I did not know about 2x2 functionality. Now, I have modified the logic to remove the invalid ids and paths and added tests for the same.

[pratiman-91]

Hi @headtr1ck, I have been thinking about the handling of ids. Current version looks like a patch work (I am not happy with it.). I think we can create ids after removing all the invalid datasets from path1d within the combine==nested block. Please let me know what do you think.
Thanks!