Add job return logging

[jieru-hu]

I tried a fews ways to add the logging in hydra core without a callback. However didn't find a clean way to add the logging in both RUN and MULTIRUN mode without adding much more noise to the console.

Add logging as a callback means: 1) job return will be logged together with the rest of the application log, 2) users can opt in.

related to #2100

[pixelb]

Is this a callback we might be able to enable by default?
I.e. what are the disadvantages of enabling it?
Does it result in more console output?

In any case the code looks good. thanks

[Jasha10]

I think users might be confused when this "Examples" button redirects to the hydra/experimental directory; there are some things in that directory that are unrelated to callbacks.

What if we create a file hydra/experimental/callbacks.py that contains all the callback implementations (instead of using separate files log_job_return_callback.py and pickle_job_info_callback.py)? The ExampleGithubLink in the docs could then point to that specific file callbacks.py.
This would make the yaml files simpler too: users would write _target_: hydra.experimental.callbacks.LogJobReturnCallback instead of _target_: hydra.experimental.log_job_return_callback.LogJobReturnCallback.

[jieru-hu]

That's a great idea! Since this would mean touching the PickleJobCallback as well, I will create that PR as a follow up.
For now I will remove this change.

[Jasha10]

Is this a callback we might be able to enable by default? I.e. what are the disadvantages of enabling it? Does it result in more console output?

It does result in more console output. Here is an example:

$ cd examples/tutorials/basic/running_your_hydra_app/5_basic_sweep
$ python my_app.py -m db=mysql,postgresql '+hydra.callbacks.foo._target_=hydra.experimental.log_job_return_callback.LogJobReturnCallback' hydra.job.chdir=true
[2022-03-27 09:14:38,045][HYDRA] Launching 4 jobs locally
[2022-03-27 09:14:38,045][HYDRA]        #0 : db=mysql db.timeout=5
driver=mysql, timeout=5
[2022-03-27 09:14:38,141][hydra.experimental.log_job_return_callback][INFO] - Succeeded with return value: None
[2022-03-27 09:14:38,142][HYDRA]        #1 : db=mysql db.timeout=10
driver=mysql, timeout=10
[2022-03-27 09:14:38,233][hydra.experimental.log_job_return_callback][INFO] - Succeeded with return value: None
[2022-03-27 09:14:38,233][HYDRA]        #2 : db=postgresql db.timeout=5
driver=postgresql, timeout=5
[2022-03-27 09:14:38,323][hydra.experimental.log_job_return_callback][INFO] - Succeeded with return value: None
[2022-03-27 09:14:38,324][HYDRA]        #3 : db=postgresql db.timeout=10
driver=postgresql, timeout=10
[2022-03-27 09:14:38,414][hydra.experimental.log_job_return_callback][INFO] - Succeeded with return value: None