improve error msg when optional packages not available

[wyli]

Is your feature request related to a problem? Please describe.

from monai.bundle import ConfigParser                                                                                          
                                                                                                                               
config = {"training": "$monai.handlers"}                                                                                       
parser = ConfigParser(config=config)                                                                                         
parser.get_parsed_content("training")   

when ignite is not installed, the error is:

Traceback (most recent call last):
  File "test.py", line 5, in <module>
    parser.get_parsed_content("training")
  File "/Users/Documents/MONAI/monai/bundle/config_parser.py", line 237, in get_parsed_content
    return self.ref_resolver.get_resolved_content(id=id, **kwargs)
  File "/Users/Documents/MONAI/monai/bundle/reference_resolver.py", line 182, in get_resolved_content
    return self._resolve_one_item(id=id, **kwargs)
  File "/Users/Documents/MONAI/monai/bundle/reference_resolver.py", line 166, in _resolve_one_item
    item.evaluate(globals={f"{self._vars}": self.resolved_content}) if run_eval else item
  File "/Users/Documents/MONAI/monai/bundle/config_item.py", line 367, in evaluate
    return eval(value[len(self.prefix) :], globals_, locals)
  File "<string>", line 1, in <module>
AttributeError: module 'monai' has no attribute 'handlers'

cc @dongyang0122 @Nic-Ma

[Nic-Ma]

Hi @wyli ,

Oh, I see. Thanks for raising this ticket.
I will try to improve the error message for it.

Thanks.

[Nic-Ma]

After some investigation, the root cause of the issue is that: if any optional dependency is unavailable, then the related modules will not be imported:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/utils/module.py#L194-L195
So monai module doesn't have the handlers property.
It can be easily reproduced by:

>>> import monai
>>> 
>>> print(monai.handlers)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'monai' has no attribute 'handlers'

I did some tests but didn't get a good solution for it, @wyli @ericspod do you guys have any idea to improve it?

Thanks.

[ericspod]

Is this problem related to the special case here? Does this problem come up for the engines directory as well which also should rely on ignite in the same way? Maybe there's a way around the reason behind excluding handlers and bundle from the loading process.

[Nic-Ma]

Hi @ericspod ,

It has the same issue for engines as well. Do you have any idea to fix it with a simple work around?

Thanks in advance.

[ericspod]

Are the excludes in the __init__.py file I mentioned there because of the decorators used? We have a problem then if our optional import scheme doesn't work well with decorators, we have to create a mechanism to generate fake decorators when the import isn't present so that the class can be defined, this would eliminate the need for this exclusion unless there's some other reason why that's there.

[Nic-Ma]

Hi @wyli ,

Do you know some deep reason for @ericspod 's question? I don't remember why we added the exclude..

Thanks in advance.

[wyli]

@ericspod @Nic-Ma you are right, there are decorators and base classes:

MONAI/monai/handlers/ignite_metric.py

Line 32 in 22924f5

class IgniteMetric(Metric): # type: ignore[valid-type, misc] # due to optional_import

MONAI/monai/handlers/ignite_metric.py

Line 32 in 22924f5

class IgniteMetric(Metric): # type: ignore[valid-type, misc] # due to optional_import

[ericspod]

If base classes and decorators are the two issues we can possibly solve them both with a slight change to how the optional import works. Instead of returning an instance of _LazyRaise the optional_import function can return the type itself which can be inherited but raises the exception on instantiation. For decorators we may need a different function which returns a decorator proxy or we change _LazyRaise to fill that role somehow.

I think we should look into a larger change the whole import system to clean up some older code we possibly don't need. We can also clean up duplication in __init__.py files where we have members listed rather than just have from foo import * and use the __ALL__ list to filter what gets imported.

[Nic-Ma]

Hi @ericspod ,

Would you like to contribute a PR for your optimization ideas? I think you have already thought very deeply about it.

Thanks.

[ericspod]

It will take time and maybe best not to upset things for the upcoming release, I'm working on bundles for you right now too. The way that we're importing everything at once with load_submodules perhaps should go but that will need a good bit of testing to make sure that everything continues to behave, so let's leave this for next version and live with a bad error message for now?

[Nic-Ma]

Hi @ericspod ,

I totally agree with it, sounds like a good plan.
@wyli Do you have any other concerns? I think let's keep this ticket open for a while.

Thanks.