feat(node/sequencer) Break apart Engine operations to prevent blocking

[op-will]

Description

The main goal of this change is to allow engine processing to continue after block building has been initiated but before it has been sealed.

This is accomplished by:

• Updating the EngineActor to
  • Make the BuildTask only initiate block building
  • Add a new SealTask to seal and insert the initiated block
• Updating the SequencerActor loop to
  • Seal the last block and start the next block
  • Dynamically set the loop interval to account for sealing duration

Importantly the SequencerActor's build_block function no longer starts and completes block building in a single call but rather seals the last block if there was one, and starts the next. This means that the time spent waiting for the next tick will be during block building.

This also changes the EngineTask pattern within the SealTask such that if there was a caller (i.e. there exists a Sender to use to relay results), the task may not be retried if there is a "temporary" task error. While this diverges from existing patterns, it is a meaningful step toward #3021, which aims to pull logic out of the engine and into the appropriate actor. This gives the Sequencer more insight into and control over block building and timing.

Considerations

This PR creates a number of possible race conditions. One race condition that did not previously exist but has been introduced and mitigated is Build (FCU) -> Reset -> Seal.

A worthwhile exercise for a reviewer is to consider the other possible race conditions introduced and evaluate the feasibility of them occurring and impact if they happen.

Testing

While this PR is ready to review and passes existing e2e and acceptance tests, a byproduct of breaking apart the BuildTask is a number of possible race conditions. This PR aimed to address a number of them (e.g. detecting if the unsafe head changed between Build and Seal operations and retrying), but a worthwhile exercise will be to:

1. List all possible EngineActor task race conditions that could result from this change
2. Define expected behavior for each of those conditions
3. Confirm via unit tests that behavior matches expected behavior.

There is also value in unit testing the logic in the different functions within SequencerActor and the affected EngineActor tasks.

Unfortunately the act of making the SequencerActor, EngineActor, and EngineTasks unit testable is a considerable amount of work. A different PR will be created, branching off of this PR to attempt that. This PR can be merged as is, as it meets current standards, but another reasonable option is to wait until kona is more unit testable and unit tests exist to demonstrate that the changes work as intended.

Other

Fixes #2978
Relates to #2848, #3021

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.

[codecov]

Codecov Report

❌ Patch coverage is 63.21070% with 110 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.3%. Comparing base (daa3535) to head (2ac3bfa).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...ates/node/engine/src/task_queue/tasks/seal/task.rs	55.8%	53 Missing ⚠️
crates/node/service/src/actors/sequencer/actor.rs	79.3%	19 Missing ⚠️
...tes/node/engine/src/task_queue/tasks/build/task.rs	30.0%	14 Missing ⚠️
...tes/node/engine/src/task_queue/tasks/seal/error.rs	0.0%	12 Missing ⚠️
...e/engine/src/task_queue/tasks/consolidate/error.rs	0.0%	6 Missing ⚠️
crates/node/engine/src/task_queue/tasks/task.rs	50.0%	4 Missing ⚠️
...de/engine/src/task_queue/tasks/consolidate/task.rs	85.7%	1 Missing ⚠️
crates/node/service/src/metrics/mod.rs	0.0%	1 Missing ⚠️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.