Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

_config object contains state not suitable for multiprocessing #646

Open
jmartin-tech opened this issue May 1, 2024 · 3 comments
Open
Assignees
Labels
bug Something isn't working enhancement Architectural upgrades
Milestone

Comments

@jmartin-tech
Copy link
Collaborator

Overview

Many garak._config namespace variables such as transient, run, reporting are currently populated at state during execution. The management of these values between testing runs for a long running service does not lend to a singleton as then only one of each can exist at a time in the python interpreter. Further since this data is populated into a namespace variable reload of the namespace will not maintain these values.

The behavior of the multiprocessing package is more akin to launching another new python process that uses only the files directly connected to the source objects passed.

#645 addresses an instance that exposed this concern.

Current state example

In the case of the probe class and its associated class hierarchy being called the only argument is defined in the same class and state of config is not passed.

Desired state

Any stateful _config likely needs to either be passed or consistently reloaded in new processes. Investigation and understanding of use cases needs occur to consolidate on patterns either in code standards or library framework supporting task execution with all required context.

@leondz leondz added bug Something isn't working enhancement Architectural upgrades labels May 1, 2024
@jmartin-tech jmartin-tech changed the title _config object contains state suitable of multiprocessing _config object contains state not suitable of multiprocessing May 1, 2024
@jmartin-tech jmartin-tech changed the title _config object contains state not suitable of multiprocessing _config object contains state not suitable for multiprocessing May 1, 2024
@leondz
Copy link
Collaborator

leondz commented May 8, 2024

When addressing this:

@leondz leondz added this to the release 0.9.1 milestone May 13, 2024
@leondz leondz changed the title _config object contains state not suitable for multiprocessing _config object contains state not suitable for multiprocessing May 13, 2024
@leondz
Copy link
Collaborator

leondz commented Sep 4, 2024

How should we triage/roadmap this fella?

@jmartin-tech
Copy link
Collaborator Author

There is some ideation about possibly injecting a shared memory location to sync config across process and thread boundaries, maybe set a goal to introduce this in this quarter.

The idea proposed is for module level code in _config to search for an existing shared memory object that would be created in the parent runtime when _config is locked to start a run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement Architectural upgrades
Projects
None yet
Development

No branches or pull requests

2 participants