feat(spider-execution-manager): Forward RUST_LOG and configured environment variables to spawned task executors.

[LinZhihao-723]

Description

This PR lets the execution manager forward environment variables into the spider-task-executor subprocesses its process pool spawns. RUST_LOG is always forwarded so an executor's log verbosity matches the execution manager's, and a new env_keys configuration field lets operators forward an arbitrary list of additional variables (for example credentials or feature flags a task package needs). Each variable's value is read from the execution manager's own environment at spawn time. Forwarding is additive: the child already inherits the full parent environment, so this only sets the selected keys explicitly on top.

env_keys configuration (config.rs)

• Adds env_keys: Vec<String> to TaskExecutorConfig, deserialized from the task_executor section of the YAML config. It names the environment variables to forward to executors, beyond the always-forwarded RUST_LOG. The field is #[serde(default)], so it is optional and defaults to an empty list when omitted.
• Config::runtime_config() forwards the list into the derived RuntimeConfig.

Threading through the runtime (runtime.rs)

• Adds env_keys to RuntimeConfig, and passes it into ProcessPoolConfig when Runtime::create builds the process pool.

Environment forwarding at spawn (process_pool.rs)

• Adds env_keys to ProcessPoolConfig.
• spawn_executor now populates the child Command's environment before spawning: RUST_LOG is always forwarded when it is set in the execution manager's environment, and is silently skipped when unset (it is commonly unset, so a missing value is not worth a warning). Each variable named in env_keys is then forwarded when set; a key that is unset or holds a non-Unicode value is skipped with a warn-level log that carries the key and the underlying std::env::VarError (which distinguishes not-present from non-Unicode).
• The forward is additive rather than an allowlist: no env_clear() is issued, so the child keeps the inherited parent environment and the explicit forwards remain correct even if the child environment is later cleared.

Tests (em-runtime-tests, task-executor-tests)

• Both test crates construct these config structs with struct literals; each now sets env_keys: Vec::new(), preserving the existing no-forwarding behavior.

Notes

• RUST_LOG is treated as a built-in and is not part of env_keys; the two paths differ deliberately on a missing value (silent for RUST_LOG, warned for a configured key, since a configured key was explicitly requested).
• env_keys is optional (#[serde(default)]): a task_executor config that omits it deserializes with an empty list — no forwarding beyond RUST_LOG — so existing config files keep loading unchanged.

Checklist

• The PR satisfies the contribution guidelines.
• This is a breaking change and that has been indicated in the PR title, OR this isn't a
  breaking change.
• Necessary docs have been updated, OR no docs need to be updated.

Validation performed

• Ensure all workflows pass.

Summary by CodeRabbit

• New Features

  • Added configuration support for forwarding selected environment variables to spawned task executors.
  • Automatically forwards RUST_LOG when it is set.
  • Unavailable variables are skipped with a warning.

• Tests

  • Updated runtime and process pool test configurations to support the new environment forwarding settings.

[coderabbitai]

Walkthrough

Changes

The execution manager adds an inherited_env configuration allowlist, propagates it through runtime and process-pool configuration, and forwards configured values plus RUST_LOG when spawning task executors. Missing or invalid values are skipped with warnings.

Executor environment forwarding

Layer / File(s)	Summary
Environment forwarding configuration components/spider-execution-manager/src/config.rs, components/spider-execution-manager/src/runtime.rs, tests/huntsman/em-runtime/tests/test_runtime.rs, tests/huntsman/task-executor/tests/test_process_pool.rs	inherited_env is added to configuration types, propagated into the process pool, and initialized in test configuration helpers.
Child environment forwarding components/spider-execution-manager/src/process_pool.rs	Executor spawning forwards RUST_LOG and configured environment values, warning when configured values are unavailable or non-Unicode.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Config
  participant Runtime
  participant ProcessPool
  participant TaskExecutor
  Config->>Runtime: propagate inherited_env
  Runtime->>ProcessPool: initialize ProcessPoolConfig
  ProcessPool->>TaskExecutor: spawn with RUST_LOG and configured environment values

Loading

Possibly related PRs

• y-scope/spider#326: Modifies the process-pool supervisor and task-executor spawn lifecycle.

Suggested reviewers: sitaowang1998

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: forwarding RUST_LOG and configured environment variables to spawned task executors.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[LinZhihao-723]

The current way for passing env var for CLP is confirmed with @junhaoliao: we'd prefer to pass only the key, not the entire kv pair.

[sitaowang1998]

I don't think env_keys is a good name in the config. Since these envs are to be passed into task executor., how about executor_env_keys.

[LinZhihao-723]

I don't fully agree. While these keys are indeed passed into the executor, it's also required that the execution manager has these environment variables defined. A more precise name would probably be inherited_env_keys, but that sounds too developer-oriented.

[LinZhihao-723]

Actually, maybe env or environment should be sufficient: these are how k8s and docker-compose specify the env keys (without values), respectively.

[sitaowang1998]

Is there any risk a user of Spider is not a developer or a coding agent? lol.

I am ok with no keys. How about inherited_env?

[sitaowang1998]

We need to check and skip RUST_LOG.

[LinZhihao-723]

I don't think we need to do an extra check since:

• command.env is an upsert; without passing the explicit values, this upsert should be a no-op.
• When RUST_LOG is explicitly given in this list and it's not set, we should print a warning to make this absence visible. While in the default path, it is allowed to have RUST_LOG unset.