op-devstack: run rust services as sub-processes in sysgo

[protolambda]

Description

While the devstack offers flexibility between remote networks and local sysgo networks, the feedback loop of the remote network and the configuration challenges make it difficult to use.

Sysgo also provides additional network customization, since the network is set up for the testing specifically, and can is not constrained to kurtosis / other external things.

The rust stack needs a faster feedback loop, and coverage of these sysgo tests, so a different approach was taken than kurtosis here.

This PR introduces the idea of running Kona / op-reth as a "sub-process". By swapping the Go service that normally runs in sysgo for a managed sub-process, the sysgo testing works all the same, and in-process Go services can interact with the sub-process rust services.

This PR:

• Introduces a "logpipe" package to funnel the structured-logging std-out/err output of the sub-process into the labeled logger of the component. All the same log-filters will work for the sub-process, unified with the Go logging.
  • TODO: revisit the structured-log decoding; need to match the rust structured log format correctly
• Introduces a "sysgo.SubProcess" component to manage the lifetime of the sub-process, and support the start/stop functionality that other sysgo components have.
• Generalizes the L2 CL sysgo configuration, to not tightly couple to op-node anymore
• Adds the option to add a KonaNode, as sub-process
• Generalizes the L2 EL sysgo configuration, to not tightly couple to op-geth anymore
• Adds the option to add an op-reth, as sub-process
• Adopt the TCP proxy setup from op-geth sysgo to apply to op-node, kona-node and op-reth. This way the RPC addresses stay valid across restarts.

Usage:

• env: DEVSTACK_L2CL_KIND=kona to select kona
• env: DEVSTACK_L2EL_KIND=op-reth to select op-reth
• env: KONA_NODE_EXEC_PATH=/home/USERHERE/projects/kona/target/debug/kona-node to select the kona-node executable to run
• env: OP_RETH_EXEC_PATH=/home/USERHERE/projects/reth/target/release/op-reth to select the op-reth executable to run

Defaults are to op-node / op-geth

Tests

• Tested log-pipe functionality
• Tested sub-process functionality
• Devstack tests can run with Kona / op-reth. TestTransfer is confirmed locally, others are more experimental. TBD on CI strategy.

Additional context

Metadata

Fix #16725

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.00%. Comparing base (5511151) to head (d186907).
⚠️ Report is 1 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff              @@
##           develop   #16726       +/-   ##
============================================
- Coverage    45.82%        0   -45.83%     
============================================
  Files         1468        0     -1468     
  Lines       119406        0   -119406     
============================================
- Hits         54719        0    -54719     
+ Misses       60854        0    -60854     
+ Partials      3833        0     -3833     

Flag	Coverage Δ
cannon-go-tests-64	?
contracts-bedrock-tests	?

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1468 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[protolambda]

Copy-paste from conversation thread, progress update on Kona test running:

Trying to test-run now, but Kona-node is not updating its L1 finalized/safe data, and because of that not its L2 safe/finalized. Meanwhile L1 initially responds with generic error, because upon genesis it doesn't immediately have a finalized/safe block yet, but those are set shortly after startup in our in-Go setup. Still, it's valid to not have a finalized/safe block in L1 after genesis (the poor generic error by L1 geth is bad though).

Logs:

t=2025-07-24T15:43:28.320+0200 lvl=warn msg="Served eth_getBlockByNumber" conn=127.0.0.1:35426 reqid=4 duration=15.519µs err="finalized block not found" global=true
t=2025-07-24T15:43:28.320+0200 lvl=warn msg="failed to query l1 provider for latest finalized block" src=stdout error="ErrorResp(ErrorPayload { code: -32000, message: \"finalized block not found\", data: None })" scope=/pkg chainID=901 kind=L2CLNode id=L2CLNode-sequencer-901
t=2025-07-24T15:43:28.320+0200 lvl=warn msg="Served eth_getBlockByNumber" conn=127.0.0.1:35426 reqid=5 duration=17.582µs err="safe block not found" global=true
t=2025-07-24T15:43:28.320+0200 lvl=warn msg="failed to query l1 provider for latest safe block" src=stdout error="ErrorResp(ErrorPayload { code: -32000, message: \"safe block not found\", data: None })" scope=/pkg chainID=901 kind=L2CLNode id=L2CLNode-sequencer-901

Could it be that kona-node internally caches the failed RPC result somehow? Or stops trying?
After 6 attempts with "failed to query l1 provider for latest safe block", just 10s of milliseconds appart without backoff, it stops logging this. Or maybe it did pick up the change in L1?

It gets in a loop with:

t=2025-07-24T15:43:32.822+0200 lvl=warn msg="empty BlockRef in sync status" syncStatus.headL1=7ce960..23468a:0 syncStatus.currentL1=7ce960..23468a:0 syncStatus.localSafeL2=000000..000000:0 syncStatus.safeL2=000000..000000:0 syncStatus.unsafeL2=000000..000000:0 scope=/pkg chainID=901 kind=L2Batcher id=L2Batcher-main-901
t=2025-07-24T15:43:32.822+0200 lvl=warn msg="Sequencer is out of sync, retrying next tick." scope=/pkg chainID=901 kind=L2Batcher id=L2Batcher-main-901
t=2025-07-24T15:43:32.835+0200 lvl=info msg="Skipping proposal for genesis block" scope=/pkg chainID=901 kind=L2Proposer id=L2Proposer-main-901

I.e. the kona-node doesn't label anything as safe, and this stops the batcher (and proposer too). And without batcher, it won't batch-submit, and never mark something safe.

EDIT: for context, I'm running TestTransfer (pkg: github.com/ethereum-optimism/optimism/op-acceptance-tests/tests/base) with DEVSTACK_L2CL_KIND=kona

---

And we need to make some slight adjustment to parse the ip:port, remember the port, and configure the next restart to re-use the same port instead of the 0 system port selection. That way the tests that rely on restarts of the node will be able to work.

Update: @theochap fixed these issues in Kona

[mslipper]

proxy changes look fine

[protolambda]

There's something wrong with TestSubProcess in CI, a simple echo invocation as sub-process doesn't seem to work as expected, but I can't reproduce it locally. Maybe something odd about the CI environment?

[protolambda]

Rebased, and fixed the TestSubProcess and some usage things