Logging doesn't propagate: trying to get all of ax messages to file

[ksanjeevan]

I'm trying to run trials remotely and log the ax output to a file. From my main script I have:

...
fh = logging.FileHandler('out.log')
fh.setLevel(logging.INFO)

logging.getLogger('ax').addHandler(fh)
...

In other libraries this is enough to have all the logging be propagated to the root and then since I've added the handler, push them to a file. But with ax the log file is empty. I can see all the ax logger names:

print([k for k in logging.Logger.manager.loggerDict.keys() if k.startswith('ax')])
>>> ['ax.core.objective', 'ax.core', 'ax', 'ax.core.outcome_constraint', 'ax.core.experiment', 'ax.core.multi_type_experiment', 'ax.models.torch.botorch', 'ax.models.torch', 'ax.models', 'ax.modelbridge.transforms.int_to_float', 'ax.modelbridge.transforms', 'ax.modelbridge', 'ax.models.discrete.eb_thompson', 'ax.models.discrete', 'ax.models.discrete.full_factorial', 'ax.modelbridge.registry', 'ax.modelbridge.factory', 'ax.modelbridge.generation_strategy', 'ax.modelbridge.dispatch_utils', 'ax.service.utils.best_point', 'ax.service.utils', 'ax.service', 'ax.service.managed_loop', 'ax.benchmark.benchmark_problem', 'ax.benchmark', 'ax.benchmark.benchmark_result']

But adding a handler only works if I set it for each one individually, not just for the .getLogger('ax') root.
Looking at the code:

Ax/ax/utils/common/logger.py

Line 14 in 4cea49b

def get_logger(

it seems that in every submodule handlers and formatters are being set? Is that what's causing it not to propagate?

What are the best practices to follow if I want all of ax logging to be dumped to a file?

Thanks.

[Jakepodell]

Hi @ksanjeevan! Thanks for bringing this to our attention. We have a few ideas of what the problem might be, but we will need to look into this further to get you a reliable solution. Thanks for your patience while we work on getting you an answer here!

[stevemandala]

Quick update @ksanjeevan: We refactored the core logger libraries in 44fdecb such that all Ax loggers inherit (and propagate) to the root logger "ax". With those change, this (similar to your example) should work:

# Same as logging.getLogger('ax')
from ax.utils.common.logger import get_root_logger()

fh = logging.FileHandler('out.log')
fh.setLevel(logging.INFO)

get_root_logger().addHandler(fh)

This is currently on master and should get included in the next release.

[ksanjeevan]

Thanks for the update @stevemandala, I'll give it a shot!