You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PR #1828 takes a first step towards engine configuration within SMARTS. This, however, does not translate to providing configuration to worker processes. To do that we can take the initial configuration in the main process and any configuration that is not already an environment variable to one so that all subprocesses and workers inherit those values.
Solution
On load of the entry process we can convert all configurations into environment variables.
We can also set an specialized environment variable (e.g. SMARTS_MAIN_LOADED_ENVS) as a way to only run this in the entry process.
Context
We had some discussion regarding this issue.
> 1. Loop through the configuration and use them to set environment variables. (this should pass to subprocesses and workers but it would also mean only a single engine configuration of SMARTS could ever run on the same thread)
This seems viable, as it may allow us to make the engine configuration propagate across machines if a program propagates across workers (using something like `ray`) as long as the environment configuration resolves before the workers are generated (before something like `ray.init()`.) It looks like this might be viable to implement.
With that said, any user using a cluster management system could just inline the configuration using environment variables.
...
> 5. Set a series of configuration paths (global user, local user, local directory) that are resolved in order of closeness to the project. Sub-processes will know about it inherently and each process lazy loads the configuration independently.
This seems like the most reasonable option for now. The exact implementation will resolve each of the configuration in this priority:
1. Individual environment variables (e.g. SMARTS_SENSOR_WORKER_COUNT)
2. Local directory engine configuration (`./smarts_engine.ini` )
3. Local user engine configuration (`~/.smarts/smarts_engine.ini` ) if local directory configuration is not found.
4. Global engine configuration (`/etc/smarts/smarts_engine.ini` ) if local configuration is not found.
5. Package default configuration (`${PYTHON_ENV}/lib/${PYTHON_VERSION}/site-packages/smarts/core/smarts_engine.ini`).
> 6. Configure the engine through `scl`.
This is viable regardless of other features. It allows us in the case of `scl run` to provide a `--configuration` flag which we could pass in a configuration from a custom location and immediately resolve it to environment variables before anything else is evaluated.
## Direction
We decided to go with option 5 and combine it with option 2. At a later date we may want to bring in option 6.
Issue
PR #1828 takes a first step towards engine configuration within SMARTS. This, however, does not translate to providing configuration to worker processes. To do that we can take the initial configuration in the main process and any configuration that is not already an environment variable to one so that all subprocesses and workers inherit those values.
Solution
On load of the entry process we can convert all configurations into environment variables.
We can also set an specialized environment variable (e.g.
SMARTS_MAIN_LOADED_ENVS
) as a way to only run this in the entry process.Context
Originally posted by @Gamenot in #1828 (comment)
The text was updated successfully, but these errors were encountered: