op-node: single aggregate safe-head advancement FCU message per channel

[0x416e746f6e]

Is your feature request related to a problem? Please describe.

we observe the following problem:

• when batcher is running with considerable --max-channel-duration, for example 50 (equivalent of 10m), it will publish l2 information on l1 (most of time) at those intervals.
• op-node running in sequencer mode reads the above updates at said intervals, and advances the safe head accordingly.
• currently it does it in sequential manner, block by block.
  • in other words, for channel duration of 50 (10m) and l2 block time of 1s, sequencer will send ~600 FCU messages in a row to its EL.
  • this happens in the span of 1-3s, which creates a congestion in the network connection b/w sequencer and the EL.
    • in the sense that the "block-building" FCU message (with extra param) would be sent by the sequencer among the bulk of other FCUs (advancing the safe head), and processes by the EL in sequence, which could result in the interval between "block-building" FCU and consecutive getPayload call being significantly less than usual
      • we saw cases where the gap was on the scale of 10ms vs. the usual 950ms
      • or even worse when FCU would be delivered after the typical timestamp-offset of the getPayload calls

Describe the solution you'd like

would it be possible for the op-node sequencer, instead of advancing safe head step-by-step, just do it once?

e.g. by going through the whole batch published on l1 and then sending only the last derived safe head in 1 FCU message?

[protolambda]

We can probably add a debounce to the engine-controller FCU code, to handle when these API calls are really prohibitively expensive/inefficient. The current FCU handling on the engine side, assuming the FCU is all for incremental blocks on the same chain against a local node, should be relatively quick compared to all the actual block processing however, so omitting the calls may not save that much time. What's your experience with processing cost here?

[0x416e746f6e]

it's a function of round-trip time on the network.

• when the EL is local (RTT ~0.01ms), sending 600 FCUs in a row is fast (~6ms)
• but when EL is not so local, although still very close (RTT 5ms), sending 600 FCUs in a row is going to take prohibitively long time (3s in current example)

[ferranbt]

@protolambda One question I have is that if this FCU calls are being made so close to each other, why are they not sent with the no_tx_pool flag set? I would not expect the builder doing much work anyway.

[protolambda]

@ferranbt

as sequencer:

• no-tx-pool is currently only set when running out of sequencer time drift
• we can more eagerly disable the tx-pool, when we know we're behind the wallclock significantly, but it's more of an optimization. You're welcome to submit a PR to change the behavior here, as long as we don't shut down tx-inclusion too eagerly.

as verifier:

• no-tx-pool is always set, since blocks are reproduced from exact inputs from L1

[ferranbt]

when the EL is local (RTT ~0.01ms), sending 600 FCUs in a row is fast (~6ms)

Hey @protolambda! We want to make a PR to fix this. Can you point me to the specific part in the driver that might be handling this?

[SozinM]

@protolambda Additional context to this:
When i set max channel to 10 locally roughly every 40 sec i would see the following:
Log line t=2025-05-14T10:54:41+0000 lvl=info msg="created new channel" stage=channel origin=0x0375b1214959ae86e7cd68612455d833a03a0ddea0057b6fdb4f210acaca18c2:1461
Then i would see ~20 of these in 1 sec (logging does not show ms):

t=2025-05-14T10:54:41+0000 lvl=info msg="decoded singular batch from channel" batch_type=SingularBatch batch_timestamp=1747220033 parent_hash=0x3a95df275b772949024f2e3af2cdbde9293b82f885c3dc34b4732c7086bfc167 batch_epoch=0x3cb59eec3af1e82e65c3fbef7d9a4ba06741cb76cc2e8727c1cb7a877a7f0fa3:1450 txs=0 compression_algo=zlib stage_origin=0x0375b1214959ae86e7cd68612455d833a03a0ddea0057b6fdb4f210acaca18c2:1461

This would correlate to ~20 FCU calls received by EL

[SozinM]

More context:
This FCU spam happens because every pending safe update events comes with Concluding = true and this triggers FCU
When we update safe head we will update pending event on every L2 block, and i thing it should be following logic:
all in-between promote pending safe should come with Concluding = false, and the last one should be with Concluding = true.

		if x.Concluding && x.Ref.Number > d.ec.LocalSafeL2Head().Number {
			d.emitter.Emit(PromoteLocalSafeEvent{
				Ref:    x.Ref,
				Source: x.Source,
			})
		}

[sebastianst]

I think this is related to #12695 there are other benefits to only advancing the safe head after a full channel (or even full L1 block, which in practice is usually the same) has been derived. I won't find the time in the near future to investigate this though.

[clabby]

We've implemented this over in kona-node, only sending one FCU per span batch in the happy-path (sad-path being the Holocene forwards-invalidation rule, where we send an FCU after the replaced block is built): https://github.com/op-rs/kona/blob/main/crates/node/engine/src/task_queue/tasks/consolidate/task.rs#L88-L111

[teddyknox]

We merged this today which may help.

#18234