feat: reinit="allow" setting for multiple simultaneous runs in one process

[timoffex]

• WB-23108
• Docs: Document new reinit='allow' setting docs#1145

Adds an allow option to the reinit setting which causes wandb.init() to create a new run even if other runs exist. This should eventually become the default, and the other options ("return_previous" and "finish_previous") should be deprecated.

Although it's possible to pass reinit as a wandb.init() argument, the best way to use this is with wandb.setup().

Here is an example that uses the allow setting to (serially) run multiple sub-experiments while logging their results to another run:

wandb.setup(wandb.Settings(reinit="allow"))

with wandb.init() as experiment_results_run:
    for ...:
        with wandb.init() as run:
            # The do_experiment() function logs fine-grained metrics
            # to the given run and then returns result metrics that
            # we'd like to track separately.
            experiment_results = do_experiment(run)

            # After each experiment, we log its results to a parent
            # run. Each point in the parent run's charts corresponds
            # to one experiment's results.
            experiment_results_run.log(experiment_results)

[timoffex]

• feat: reinit="allow" setting for multiple simultaneous runs in one process #9562 👈 (View in Graphite)
• refactor: share gpu_stats process between multiple runs #9632
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.44%. Comparing base (526f4dc) to head (0da4fca).
Report is 5 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9562      +/-   ##
==========================================
- Coverage   80.48%   80.44%   -0.04%     
==========================================
  Files         737      738       +1     
  Lines       78967    79029      +62     
==========================================
+ Hits        63554    63573      +19     
- Misses      14683    14725      +42     
- Partials      730      731       +1     

Flag	Coverage Δ
func	46.20% <59.45%> (+0.19%)	⬆️
system	64.80% <98.75%> (-0.01%)	⬇️
unit	65.26% <27.02%> (-0.10%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
tests/conftest.py	98.76% <100.00%> (-0.41%)	⬇️
tests/system_tests/test_core/test_wandb_init.py	100.00% <ø> (ø)
...s/system_tests/test_core/test_wandb_init_reinit.py	100.00% <100.00%> (ø)
tests/system_tests/test_core/test_wandb_setup.py	100.00% <100.00%> (ø)
wandb/sdk/wandb_init.py	84.52% <100.00%> (+0.04%)	⬆️
wandb/sdk/wandb_run.py	88.73% <100.00%> (+0.05%)	⬆️
wandb/sdk/wandb_settings.py	91.66% <ø> (ø)
wandb/sdk/wandb_setup.py	88.78% <100.00%> (+1.07%)	⬆️

... and 26 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kptkin]

is it interesting to test combinations of different reinit modes in the same script?

[timoffex]

This file includes tests for interactions that are special; is there anything missing?

[kptkin]

let me read it more carefully again!

[kptkin]

i think this behavior of global run is going to cause some confusion with users, as it will require to carefully track when they create and finish each run, maybe a better solution is to disallow global run when we have multiple active runs?

[kptkin]

or have multiple global runs? and they need to use id or something, not sure if this approach is better, but an alternative to fully disabling global run

[timoffex]

This behavior is for integrations; end users should avoid using wandb.run.

An alternative I considered is to make accesses to wandb.run raise an error after reinit="allow" is used. But we check wandb.run often in our own code (for example, to get its settings), so without fixing all those instances, it would break the SDK.

So wandb.run needs to either be None or a run. If starting a reinit="allow" run were to change it to None or the new run, then integrations relying on wandb.run that were initialized before the run would break or misbehave.

The rule in this PR is that wandb.run can only change when you call wandb.init() and refers to the same run until that run finishes.

[kptkin]

hmmm i see... i guess i was thinking of integrations in third party repo, where i don't think they change these settings, but i guess there is a case for integrations in our code base.

i guess it's fine, we probably should document it and discourage certain usage patterns especially with integrations?

[timoffex]

We don't mention using wandb.run in our guide for integrations, and I recently updated those docs and some other top-level docs to not use the global run.