Add more PCR0 extraction pattern and fix tee script#244
Merged
dailinsubjam merged 1 commit intocelo-integration-rebase-13.2from Oct 9, 2025
Merged
Conversation
shenkeyao
approved these changes
Oct 9, 2025
Member
|
Can we merge it after Artemii's PR? Just to make sure your fix to the TEE proposal doesn't get lost if any conflict resolving is needed. |
dailinsubjam
merged commit 556fc2f
into
celo-integration-rebase-13.2
32 of 33 checks passed
shenkeyao
added a commit
that referenced
this pull request
Mar 18, 2026
* Fix caff-node stalling (#213) * Revert timing changes for beacon * just command to run the devnet tests. * Comment out running the devnet tests in CI. --------- Co-authored-by: Philippe Camacho <philippe@espressosys.com> Co-authored-by: Keyao Shen <shenkeyao@gmail.com> * Support timestamp env var (#218) * IL3 Remove redundant "Walking back L1Block" and "will retry" logs (#221) * Remove logs * Restore driver log * Remove retry log * Restore a log * Skip BatchFuture (#217) * Fix length check (#216) * IA1.6.1 Add batcher service running in TEE (#205) * a working script without args * a working script without args * everything works in the scripts despite the args * fix socat proxy script * working op-batcher inside docker-compose * rename the script to build batcher enclave image * cleanup and profile the op-batcher-non-tee * use port number from env and shorten nc listener timeout as it will not be used in most cases * fix dasel format * remove uneeded ESPRESSO_RUN_ENCLAVE_TESTS * fix scripts * Add op-batcher-tee image in CI (#210) * push op-batcher-tee image init * fix tag and push * test image creation without enclaver * try to use env * fix enclaver download * use env in docker images yml * restore other task * remove unneeded steps * special case to common case * use default for op-batcher and tee for op-batcher-tee * fix double ports mapping * fix batcher restart test * add a script to use enclave tool * works to some extend * also works for passing in arguments from cmd * try to upload the image * add my branch patter * fix dockerfile * a simplified version * adding packages/contracts-bedrock/forge-artifacts to op-batcher-enclave-target * PCR0 registered in op-batcher-tee docker compose and add monitor for enclave logs * copy deployment/ to op-batcher-enclave-target * fix docker-images * Remove unneeded script * remove unneeded script and cleanup readme * fix overlapping ports and move long cmd of op-batcher-tee to script * update readme * Fix batcher restart test (#222) * Fix batcher restart test * Tune parameters to be more realistic (in particular, increasing parallelism to reduce bottleneck on slow L1) * Improve logging * Fix go lint * Download binaries for appropriate architecture in Docker images (#223) * Add key rotation tests (#224) * Remove a Caff node comment (#225) * Remove a comment * Restore devnet test in CI * Disable the CI again * Bump github.com/ulikunitz/xz from 0.5.12 to 0.5.14 (#220) Bumps [github.com/ulikunitz/xz](https://github.com/ulikunitz/xz) from 0.5.12 to 0.5.14. - [Commits](ulikunitz/xz@v0.5.12...v0.5.14) --- updated-dependencies: - dependency-name: github.com/ulikunitz/xz dependency-version: 0.5.14 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Test a challenge game in the docker devnet (#228) * Add test for devnet challenge game * Fixes * Make sure batcher key used by default in the devnet tests is the same one registered in the inbox contract * Remove check in batcher that prevents it from sending transactions to Espresso immediately * Ensures that the deployment files are deleted before building a new devnet. Update README_ESPRESSO.md to remind running docker as a non root user. * Run devnet tests on CI again. * Ensure deployment files are not written by the root user. * Ignore rotate batcher key and change batch inbox owmer tests. * Clean way of setting UID and GID. * Ignore devnet tests for now so that we can merge. * Add fallback values for UID and GID. * Pinpoint forge version in CI as the linter is complaining. * Add comment regarding the number of claims. * Add comment to function TaggedWriter.Write. --------- Co-authored-by: Philippe Camacho <philippe@espressosys.com> * Run smoke devnet test in CI (#231) * Add smoke test for devnet. * Run Game Challenge and smoke test in CI. * Run all the devnet tests locally. * push (#232) * forget this commit (#233) * Add a Buffered Streamer around Espresso Streamer for batcher (#230) * Add a Buffered Streamer around Espresso Streamer for batcher Because the `EspressoStreamer` is getting `Reset` during the Batcher process when building a batch to submit to the `L1`, it hinders progress of the chain in a reasonable amount of time, which ultimately causes it to stop creating non-empty blocks. There are a number of factors that are contributing to this issue, but ultimately the `Reset` is causing the `EspressoStreamer` to restart from `0` and it takes too long before it catches back up to the next expected batch. To remedy this, a Buffer can be used to mitigate this `Reset` and revert to a point that is much closer to the desired next batch. In testing it has been observed that the `SafeL2` can sometimes move backwards. To safe guard against this, it is better to `Reset` to the `FinalizedL2` position instead of the `SafeL2` as this behavior has not been observed there. * Rename EspressoStreamer and EspressoStreamerIFace Based on feedback provided by @QuentinI in PR review: #230 (comment) The name `EspressoStreamerIFace` is quite a long name, and the `IFace` suffix isn't necessary since one could tell it's an `interface` by inspection, or using an `LSP`. The feedback provided by @QuentinI suggested to rename `EspressoStreamerIFace` to just `EspressoStreamer` so that it falls in line with our other code approaches. This change renames `EspressoStreamer` to `BatchStreamer` This change renames `EspressoStreamerIFace` to `EspressoStreamer`. * Remove `RemainingBatchesLen` method Based on feedback received from @QuentinI: #230 (comment) The only reason `RemainingBatchesLen` exists is to serve as a check, and issue a warning when things are running. Even though this matches the existing behavior this log seems to overlap with the logs corresponding to Undecided Batches which already log warnings or errors. As a result this method, and the log that utilizes it, seem to be unnecessary and should be removed to eliminate noise. This change removes `RemainingBatchesLen` and the uses of it * Add `RefreshSafeL1Origin` to `EspressoStreamer` interface Based on feedback provided by @QuentinI: #230 (comment) Since the `RefreshSafeL1Origin` method can potentially be utilized in some places that do not require a full `Refresh`, and for convenience, it makes sense to allow it to be a separately exposed method distinct from `Refresh`. This change adds `RefreshSafeL1Origin` as a required method in the `EspressoStreamer` interface. * Fix missed renamed references * Rename enclave smoke test * Fix refreshSafeL1Origin logic for Buffered Streamer The buffered streamer is resetting its read position far more than it needs to, ultimately reproducing the same issue that was already occurring with the unbuffered version. In inspecting the behavior with a debugger, it seems we're resetting the reset position unnecessarily when we receive the same safeL1Origin again. Additionally, the logic for determining the read position when the safeL1Origin advances also seems flawed, in that it is very likely to reset too far in the past. We really want to keep our relative read position unless we're explicitly told to Reset. This change addresses these issues in order to try and smooth out the batches being returned, and avoid unnecessary reprocessing of previous batches. * Add Unit Tests for BufferedEspressoStreamer Fix BufferedEspressoStreamer behavior While adding unit tests for the BufferedEspressoStreamer it was noticed that the position of the L2 and the L1 for the Buffer were being mixed together at times. This would ultimately lead to very difficult to detect bugs based on observed behavior alone. With the addition of the unit tests identifying the issue, the buffer adjustment behavior has been adjusted to apply to the L2 position in isolation away from the L1 positions. The L1 positions will cause a larger Reset in the underlying logic. * Rename tests (#236) * Remove unneeded service http proxy for docker compose (#238) * Fix `TestSmoke` failing on CI/CD (#237) The sigp/lighthouse docker image was upgraded from version `v7.1.0` to `v8.0.0-rc.0` on `2025-09-29`. Since the image isn't anchored to a version, this update gets pulled in, and it seems to have breaking changes with our previous setup. This change sets the version of the docker image used specifically to `v7.1.0` so that the previous behavior we're used to is seen. Additionally, when `TestSmoke` is running, it initially **MUST** download the images for the docker containers that wer not built in the `Build Devnet` job. This delays the launch and running of the DevNet by quite a bit. Fix this delay by adding a `docker compose pull` setp to `Build Devnet` * Fix CI after rebasing celo-14 (#243) * fix fast tests * fix go version in dockerfile and ce in streamer_test.go * update prepare-allocs.sh * fix prepare-allocs.sh * try to fix l1-geth docker * fix op-stack dockerfile * fix op-stack dockerfile * try to fix l1-geth dockerfile * fix config read * try to fix l1-geth dockerfile * try to fix challenger gamer * TN5 withdrawal devnet test (#226) * Add smoke test for devnet. * Add test to ensure L2 funds can be withdrawn to L1. --------- Co-authored-by: Theodore Schnepper <tschnepper@espressosys.com> * Rename DevNet to E2eDevnet (#239) * Rename DevNet to E2eDevnet * Remove duplicate name * Fix low gasLimit in L1 genesis (#241) * update (#244) * Streaming streamer (#235) * Fix pcr0 extraction in docker compose script and correctly shutdown op-proposer-tee (#246) * fix pcr0 extraction * stopping op-proposer-tee in script * revert to old pcr0 extraction as the new pattern breaks the old pattern * try to fix pcr0 extraction * upload op-proposer-tee * Update metrics (#242) * Update log level (#247) * Decouple Espresso L1 & OP L1 (#248) * Add back forgotten config when rebase celo-sync-14 * Fix prepare-allocs after rebasing celo-sync-14 (#250) * remove doulbe init() * try to fix prepare-allocs.sh * enable more workflow * Skip TestChallengerGame and TestWithdrawal (#251) * Trigger docker-images workflow * uncomment dasel put * skip TestChallengeGame * skip TestWithdrawal * mise: Define fake install sources for disabled tools (ethereum-optimism#18109) (#254) Co-authored-by: Adrian Sutton <adrian@oplabs.co> (cherry picked from commit de37992) * Change the logging level for the safe L2 number (#252) * Update log level * Reduce logging level (cherry picked from commit dd97e4c) * Respect espresso.fetch-api flag (#253) (cherry picked from commit 6d00dcb) * Readd devnet tests to CI. (#257) * Readd devnet tests to CI. *Fix for batcher restart test in CI. (cherry picked from commit f37ea91) * Add a log debouncer to op-service.log package (#259) (cherry picked from commit 788d28f) * Add netcat-openbsd to Dockerfile (#262) * Update log level * Add duplicate command from Terraform (cherry picked from commit b775430) * Add a devnet cleanup script (#261) * Update log level * Add cleanup script * Remove unnecessary commands (cherry picked from commit dc40b6c) * Enable circleCI after rebase14 (#265) * Fix .circleci/config.yaml and lint and most of circleCI tests after rebasing celo-sync-14 * Skip tests (in TEST_PKGS) that need auth or celo-specific rpc specified --------- Co-authored-by: Artemii Gerasimovich <artemii@espressosys.com> * Add origin height to Espresso streamer (#255) * Clean up more loggings (#266) * Reduce logs * Remove one more log * Skip attestation verification (#263) * Update espresso-tee-contracts submodule to sishan/skip-attestation-verification * Skip attestation verification to reduce gas costs * Reduce L1 gas limit from 45M to 16M * Update snapshots for registerSignerWithoutAttestationVerification * Ignore lib/automate submodule directory * fix CI * Update espresso-tee-contracts submodule Remove onlyOwner modifier from registerSignerWithoutAttestationVerification * keep large gasLimit * circleci: Enable workflow on all branches via API trigger Allow CircleCI main workflow to run on any branch when triggered via API, not just webhook triggers. This enables go-lint and go-tests to run on feature branches. * Regenerate semver-lock.json after rebase The initCodeHash for BatchAuthenticator needed to be regenerated after rebasing onto celo-integration-rebase-14.1. --------- Co-authored-by: EC2 Default User <ec2-user@ip-172-31-47-181.us-east-2.compute.internal> * Fix test TestChangeBatchInboxOwner (#264) * Fix the test TestChangeBatchInboxOwner. * Ensure the owner of the Batch Authenticator contract is initialized. * Update README and relevant scripts (#269) * Remove unused metrics (#273) * Rename (#275) * Improve image versioning and repo consistency (#276) * Add githooks and env * update path * test hooks * change wording * include docker compose * Create enclave ami for enclave test (#277) * create enclave ami * use new ami and restore the ci workflow and update enclave prepare ami script * new AMI * update readme * Add devnet smoke test with TEE (#268) * Add devnet smoke test with TEE * Remove unnecessary extra timeout * Add consts, remove incorrect setting * Use consts * Add missed file * Blockscout running inside the local devnet (#281) * Extend the local devnet with blockscout. * Pinpoint versions of blockscout images. * Blockscout fetching blocks from caff node. * Build deployer image in CI * Upate CI utils * Saner 'confirmed' logging * Don't error out on light client issues * fix the name of deployer factory address * Add a workaround for query service lag in real-world networks * Generate more metadata * More faithful compiler output in verifier * Don't fall below hotshot origin height * Remove cache buster to speed up docker image builds * Adjust channel duration in devnet * Jump ahead when origin is too low * Add log line to matching Espresso txn to L2 block * Fix semver lock * Fix snapshot lock * Support environment variables for channel parameters * Enable EigenDaProxy & MEMSTORE (#274) * Enable EigenDaProxy & MEMSTORE * longer eigenda-proxy start period * enable eigenda at the op-node level * Don't copy artifacts to batcher image (#290) * Refactor: replace MultiNode majority rule with SingleNode client and skip deprecated test. * refactor: remove majority rule and switch to single Espresso client * Skip deprecated TestEnforceMajorityRule (deprecated under SingleNode) * Fallback Inbox contract changes (#278) * Implement changes in the Batch Inbox / Batch Authenticator contracts to support a TEE and non TEE batcher. * Add some unit tests for the Batch Authenticator and Batch Inbox contracts. * Remove the failing Circle CI tests * OP succint support (#287) Deploy OP Succint contracts on L1. Spins up op-succinct-challenger and op-succint-proposer services. Adjust the devnet test `TestChallengeGame`. Deletes the Demo we made to Celo Labs as we can now spin up a devnet with Blockscout. * Update error handling (#289) * Update error handling * Fix typo Co-authored-by: Phil <philippe@espressosys.com> --------- Co-authored-by: Phil <philippe@espressosys.com> * Document configuration of all services (#291) * Add readme for config * Insert image, add more description * Support Sepolia Devnet with TEE (#288) * update enclave-entrypoint.bash to correctly deal with external url * preserve host name for external url * Skip IsURLAvailable TCP check when using HTTP proxy * skip VerifyCertTransaction for now * reuse socat so that it can work for internal url * comment and skip TestE2eDevnetWithInvalidAttestation * OP Succinct: Making changes to the derivation pipeline (#293) * Document how to make changes to the kona repository and propagate them. * Reference new docker images for the op-succinct proposer and challenger. * Fix op-succinct dependencies diagram. (#297) * Simplify checks in the derivation pipeline (#296) * Remove the superfluous check about the batcher address as now the Batch Inbox contract verifies the sender is legitimate. * Removed nonexist logs (#298) * Add support for ZK attestation service (#294) * Add support for ZK attestation service * check attestation service url is not nil * upgrade espresso tee verifier contracts * fix contracts * fix merge * fix tests * bring back deploy aws nitro * add support for mock contract * add support for attestation verifier service * fix tee tests * use higher version of github runner * fix tee args * fix tee args * add healthcheck to attestation verifier zk * increase timeout * Invalid attestation test passing * small fixes * fix TestE2eDevnetWithUnattestedBatcherKey * fix health check * fix devnet test * use 127.0.0.1 * fix regex * debug * fix proof generation * debug * fix url * fix url * remove debug logs * resolve based on comments * address comments * update github runner enclave * fix based on suggestions * cleanup logs * Enable AltDA Espresso E2E using EigenDA Docker proxy (#295) * Integrate EigenDA via Docker proxy for AltDA Espresso E2E tests * Scope EigenDA lifecycle to the test and ensure clean startup/teardown * Extract EigenDA Docker port and image into constants * Downgrade Dasel (#303) * Make attestation service url optional * fix dasel * fix dasel * update dockerfiles * make attestation service required again * Make the withdraw devnet test pass again (#301) * Withdraw test passing again on devnet. * Faster CI * Deposit into L1 before requesting the withdrawal on L2. * Add migration related things to readme (#302) --------- Co-authored-by: Philippe Camacho <philippe@espressosys.com> * Reducing logging when outputting the batch (#304) * update logging for the batch * clean up * Document code sync procedure (#308) * Add code sync procedure * Update links * Fix format * Rename files * Update batchAuthenticator according to audit report (#309) * update batchAuthenticator according to audit report * gen bindings and fix fast-tests * Port ForcedTxs test into devnet test suite (#306) * Simplify the test as we cannot in practice reduce the window size. --------- Co-authored-by: Philippe Camacho <philippe@espressosys.com> * Reorder checks of isValidBatchTx in derivation pipeline (#310) * remove warning on every failed tx * reorder the checks * Add fallback mechanism test (#305) * Fallback mechanism test * Update op-e2e/system/e2esys/setup.go Co-authored-by: Phil <philippe@espressosys.com> --------- Co-authored-by: Philippe Camacho <philippe@espressosys.com> * Philippe/fix withdraw flakiness (#312) * Address flakiness. * Simplify the code * Fix CI --------- Co-authored-by: Keyao Shen <shenkeyao@gmail.com> * Use unified run-enclave.sh script for op-batcher-tee (#299) * update single run-enclave.sh * remove BATCHER_PRIVATE_KEY * update run-enclave.sh * Test fallback mechanism on devnet (#313) * Recovery from fallback batcher (#315) * Fallback recovery * Add caff node * Suggestions * Make ZK Verifier Optional for E2E Testing (#321) * Make Attestation Verifier Service optional When the Attestation Verifier Service was added to the integration it fundamentally modified the testing experience, requiring external environment variables to be populated in order to run the tests. Additionally, these environment variable requirements were not documented in the README_ESPRESSO.md file for reference. This change modifies the Attestation Verifier Service setup for the E2E testing environment to make it opt-in instead of being forced to be enabled. Additionally, the Verifier URL is no longer required to run the Batcher. This is a double-edged sword, however, as it means that we could potentially deploy the service without the configuration, and we would potentially be lacking the registered attestation. This may be resolvable with a slight modification to the service configuration, that we would ultimately disable for the E2E testing environment. * Fix misspelling Fix linting error that has caught a misspelling of the work 'Network'. * Modify configuration address to be required from CLI With the change of making the Espresso Attestation Service optional we removed the CLI configuration check that occurs on launch, so that the E2E tests can still be run. This has an unfortunate side-effect of allowing the Batcher to be launched in a state where it is unable to operate as intended due to user error. The only indication being a `WARN` log entry to inform him/her of his/her mistake. This sort of approach is generally discouraged, yet we still need to be able to bypass this check for testing purposes. As a result the `EspressoAttestationService` value has been modified from being a simple `string` to being an interface whose value is inspectable and not allowed to be empty by default. This allows for the test configurations to overwrite this behavior, and allow an optional value in the cases where it is needed. This should preserve the prior behavior of erroring on launch when the parameter is not configured or specified, and should also preserve the new behavior where it is explicitly disabled in tests. * Fix some nil references The EspressoAttestationService configuration value being an interface makes it a `nil`lable value by default. Care needs to be taken when accessing this value an referencing it. This change adds some additional care in referencing the value stored within. * Fix nil access error The `l1Client` being created assumes that the `sys` returned from the call is non-nil before checking the error. This is not guaranteed, and is most likley not ever the case. As a result there is a potential for an error do to attempting an access on a `nil` value. By moving the `l1Client` declaration after the error check, we avoid the potential for this issue. * Apply linting and formatting changes * Fix e2e tests - populate default EspressoAttestationService With the modification of the EspressoAttestationService to an interface instead of an individual value, we need to ensure that the default way of launching the Espresso E2E DevNet results in the value being populated with an empty allowed value. This still allows for extension and override, without requiring the value to be specified, which is our intention. This was missed when adding the capability originally. * Cleanup code practices We have duplicated code that makes the maintenance burden more difficult than it needs to be. In many of these scenarios the code that is duplicated differs by only a single line. Instead of making the system more flexible, we ended up duplicating code paths. This increases the maintenance burden by needing to ensure that these code paths match in every case where they do not differ, yet they are independent of each other. This is not a great approach. Additionally, we end up with multiple starting points for something that should not need them. We also end up storing a configuration that is unnecessary to store. This incurs conditional checks where some are not needed, and ends up making the approach be more confusing than it needs to be. This change aims to replace these approaches with one that adheres to the functional option approach and preserves the existing behavior. * Revert EspressoAttestionService to a `string` As it so happens we rely on the `CLIConfig` for `Espresso`, and the `Batcher` to be serializable. By utilizing an `interface`, we run into trouble doing this. Due this constraint, the `interface` constraint is not feasible. This change reverts the value back to a `string`, which should result in a smaller overall change. It also opts for a private configuration value that is inspectable by the `Verify` check, but not directly configurable. We expose a method to allow for it to be configured, so it can only occur within code within the code base itself. We should only invoke this via Testing where we need the value to be optional. This achieves the same result but in a different way. | NOTE: There may be a better approach to this as well, isntead of having this be a separate field, we could do something akin to sql.NullString, where we encode this value as a Marshable `struct`. The acess pattern becomes different, but we could directly encode the empty allowance into the struct itself. * Add Espresso Attestation Verifier Service to Enclave Test The Enclave tests are currently failing in CI. It is dying due to an error stemming from the lack of the EspressoAttestationService being configured. It is likely that this is required for the Enclave tests specifically. As a result, we need to add and enable it for the enclave tests. * Modify LaunchBatcherInEnclave option The LaunchBatcherInEnclave essentially launches the batcher externally within an enclave. This option actually relies on the Espresso Attestation Verifier Service to be running. This is due to the Espresso Attestation Service only being optional inside of a test environment. When launched externally, the Batcher is no longer considered to be in a "test environment", or configurable for testing. As a result, its configuration **MUST** be something that can actually be resolvable from a CLI launch. Since the Espresso Attestation Verifier Service check is only disabled within the testing environment, this means it **MUST** be enabled in the enclave. For convenience, this option has been added automatically as a part of the LaunchBatcherInEnclave option, since it depends on it. This will minimize accidentl misconfigurations. * Tee support for EigenDA (#319) * add eigenda_proxy_url to op-batcher-tee * fix the url to post to eigenDA * not hardcoding EIGENDA_PROXY_PORT * fix the block height config * Add Batcher Fallback: Channel Not Closed Test (#314) * Add test to check end of channel fallback Asana task: https://app.asana.com/1/1208976916964769/project/1209976130071762/task/1211892212379885?focus=true We need a test to check the fallback Batcher behavior in the event that the Espresso Batcher is able to submit a partial Channel that is im progress. The specific scenario we want to test for is one concerning a multi-frame channel that has had at least part of the full channel submitted to the L1 by the Espresso Batcher, then no more. After which we swap to the Fallback Batcher, and we should be able to pick up the missed / incomplete channel, and complete the transactions. * Rename helper function to match naming pattern * Fix lint issue with not checking error result of wait.For * Commit work in progress multi frame channel efforts * Adjust settings to successfully trigger multi-frame channels After a mob programming session @quentinl was able to help identify a a specific combiniation of parameters to successfully and consistently trigger multi-frames within the Batcher. This condition is a necessary precusor to the test being attempted. This commit updates the test with the information necessary to trigger this condition and sets the necessary test criteria that we are aiming to achieve. * Perform some code cleanup This change does a few things: - Address linting issue causing CI failure - Adjusts some golang forloop usage to be more modern - Adjust function call signatures to remove unused variables * Fix bug tracking unsuccessful frames in test In the `TxManagerIntercept` there is a bug that appends the successful frames to the unsuccessful ones. While this bug isn't great in the information that it taints, it doesn't actually have the large of an impact on the test as a whole, as the resulting failure condition would be triggered regardless. This bug does affect the accurate tracking of failed frames which could be valuable information for inspection. * Update espresso/environment/e2e_helpers.go Co-authored-by: Phil <philippe@espressosys.com> * Replace Disable Batcher setting references There are a number of places in our testing setup where we are explicitly preventing the Batcher from starting on launch. Instead of rewriting this same option every time we want to use it, we should reference a built in option that we can reference continually. This allows for non-repeated code and improved documentation as to the point and purpose of this option. * Refactor custom wait in test There's a condition being waited on in the switch to fallback batcher test. This wait is useful, and can be reused between tests. But the wait itself is somewhat hiding it's intention by being inline defined within the test itself. We should pull this wait out so it can be easily used, and its intention / purpose can be more easily documented. * Cleanup code reuse in frame decoding When decoding frame information for one of the Batcher fallback tests, there are similar code paths taken that result in most of the code being reused. We should clean up this code reuse so that we don't repeat ourselves in order to avoid diverging logic. Additionally, it allows us to reduce the amount of code needing to be maintained, and more clearly document the intention of the code, and the consistency with how we perform this frame decoding process. * Relocate deferred stop calls The Stop calls should occur as close to the launch of the environment as possible. As a result, any deferred calls to Stop for the system or the Espresso Dev Node should occur as close to their occurence as possible. * Modify Initial L2Verif wait to be longer With the specific Frame and Channel settings being specified in the `TestFallbackMechanismIntegrationTestChannelNotClosed` test, the initial startup check for the L2 Verifier is failing. This is due to our settings requiring the Verifier process to take a bit longer than normal. In general, we want to give it more time, but the time frame for the failure is hard-coded in the `wait` function being utilized. While we **could** add a simple `time.Sleep`, and this would work, this is generally a bad appraoch as it just adds an unchecked delay. Instead, we opt to utilize a simple `retry` for up to `n` times. In this case, we only need to wait up to `3x` the normal time, so ensure that we perform at least `3` times. * Fix failure in Batcher Fallback test The TestFallbackMechanismIntegrationTestChannelNotClosed test fails locally without stopping, in spite of the overall time limit being specified on the test. After some troubleshooting and debugging, We were able to chase down the cause to be due to the `RunSimpleMultiTransactions`. It's unclear as to why this was causing the process to hang for as long as it was. It seemed to not be handling timeout errors well for some reason. Either way, we fority this helper by setting an explicit time limit on it, and referncing the context whenever we're performing channel operations. This should allow the channel operations themselves not to block and hang the test. After this modification we were able to determine that this process was failing due to insufficient gas being provided. For some reason when running the transactions through this mechanism, we require even more gas than we're normally need. This seems a bit odd, perhaps it has to do with the differences in the transaction construction. In any case, we up the gas being provided so that this becomes a non-issue. * Fix linting issues * Update espresso/environment/tx_helpers.go Co-authored-by: Phil <philippe@espressosys.com> * Correct failure vs success in Send The triggered conditions for failures and successes are backwards in the `Send` method of `TxMangerIntercept`. Their specific frame markers should be switched. * Update espresso/environment/14_batcher_fallback_test.go Co-authored-by: Phil <philippe@espressosys.com> --------- Co-authored-by: Phil <philippe@espressosys.com> * Move diagram files (#326) * Update Succinct image versions, update diagram (#329) * Inactive Batcher Shouldn't Post (#316) * Check if the batcher is active before publishing to L1/DA * fix readme lint * more lint fixes * check batcher contract * Fix endless warning * add batch authenticator address to rollup config * handle contract undeployed error * attempt test in CI * add test to CI * Revert "add test to CI" This reverts commit 2a9678a. * add test to CI * remove jg/ from branches * attempt to clean up and make the test more reliable * fix ci error WaitUntilSafe undefined * revert 07a82bf * Fix `anvil_setBalance` not found error * Simplify isActive check * add batcher-active-publish-only to devnet tests justfile * - simplify test, one less batcher switch - increase timeouts for devnet test * Cleaned up the code, raise tx waiting time to 60s * Brought back original timeouts * started fallback batcher up + lint fix docker compose file * Ensure that in Espresso mode the batch authenticator address is set. * Removing all changes to driver.go and the tests are still passing. --------- Co-authored-by: Philippe Camacho <philippe@espressosys.com> * Removes PreRegisteredBatcher code (#327) * Remove pre authenticated batcher * fix test * Update Succinct images * Streamer namespace range 14.2 (#334) * Support namespace range endpoint (cherry picked from commit a73f7b6) * fix buils (cherry picked from commit e46909b) * update docker image (cherry picked from commit 0774898) * fix streamer tests (cherry picked from commit f752aa2) * fix streamer tests (cherry picked from commit 168426e) * fix tests (cherry picked from commit b942c28) * fix tests (cherry picked from commit b96622c) * use docker instead of cargo to generate allocs.json (cherry picked from commit efee3ac) * fix readme * address comments * remove fetch api * Enable and test Transparent proxy upgradability and batcher address update (#337) * Enable upgradability * Fix fmt * Fix file name * Fix tests * Clean up tests * Force clean build * Add temp artifact verification * More verification for artifact verification * Fix build command * Fix artifact * Fix artifact * Fix script * Fix and simplify the script * Fix proxyAdmin as well * Add back verification workflow * Fix more workflows * Restore version * Use EspressoTEEVerifierMock * Fix TeeType conversion * Fix fmt * Fix enum conflict * Fix version in test * Fix byte requirement * Add error * Enable batcher address update * Fix typo and remove redundant tests * Fix owner * Fix test build * transfer owner * Fix more tests * Fix devnet test * Fix unused param * Fix ec2 test * Fix enclave test * Fix fmt * Fix circleCI * Fix fmt again * Cleanup * Move events and errors to interface * Fix build * Update proxy admin permission * Description for TestBatcherSwitching (#335) * add description for TestBatcherSwitching * Update espresso/environment/14_batcher_fallback_test.go Co-authored-by: Keyao Shen <shenkeyao@gmail.com> * Update espresso/environment/14_batcher_fallback_test.go Co-authored-by: Keyao Shen <shenkeyao@gmail.com> * Update espresso/environment/14_batcher_fallback_test.go Co-authored-by: Keyao Shen <shenkeyao@gmail.com> * Update espresso/environment/14_batcher_fallback_test.go Co-authored-by: Keyao Shen <shenkeyao@gmail.com> * Update espresso/environment/14_batcher_fallback_test.go Co-authored-by: Phil <philippe@espressosys.com> * Update espresso/environment/14_batcher_fallback_test.go Co-authored-by: Phil <philippe@espressosys.com> --------- Co-authored-by: Keyao Shen <shenkeyao@gmail.com> Co-authored-by: Phil <philippe@espressosys.com> * Add cmd to shutdown all docker containers with TEE (#332) * cmd to shutdown all services * Small change to trigger CI --------- Co-authored-by: Keyao Shen <shenkeyao@gmail.com> * Guardians rebased (#345) Co-authored-by: OpenCode <noreply@opencode.ai> * Audit Document (#339) Co-authored-by: Philippe Camacho <philippe@espressosys.com> * Security Analysis (#342) --------- Co-authored-by: Jean Gal <jgatbkk@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Keyao Shen <shenkeyao@gmail.com> * Fix and improve steps in the code sync doc (#344) * Update doc * Update kona default branch and fix links * Fix typo * Typos --------- Co-authored-by: Philippe Camacho <philippe@espressosys.com> * Fix op-deployer build * Fix go mod and duplicate flag * Fix go-ffi * Fix prepare-allocs * Fix duplicate attribute in pipeline * Fix test slice * Fix builder version * Set timeout for docker-images CI * Fix devnet tests * Add missing file * Fix go version * Fix go version * Add missing address * Use generic way to generate slice * Fix go version for build-op CIs * Fix dockerfile * Continue devnet version fix * Fix dockerfile * More dockerfile fix * More dockerfile fix * Simplify changes * More dockerfile fix * More dockerfile fix * Add missing event type * Fix op-node * Fix op-batcher * Fix batcher TEE and proposer * Remove unnecessary changes * Address Gemini comment * Restore rootClaim fix * Remove duplicate flag * Fix devnet build * Fix go version, add timeout * Update moved crate * Fix the build after rebase (#352) * Fix op-deployer build * Fix go mod and duplicate flag * Fix go-ffi * Fix prepare-allocs * Fix duplicate attribute in pipeline * Fix test slice * Fix builder version * Set timeout for docker-images CI * Fix devnet tests * Add missing file * Fix go version * Fix go version * Add missing address * Use generic way to generate slice * Fix go version for build-op CIs * Fix dockerfile * Continue devnet version fix * Fix dockerfile * More dockerfile fix * More dockerfile fix * Simplify changes * More dockerfile fix * More dockerfile fix * Add missing event type * Fix op-node * Fix op-batcher * Fix batcher TEE and proposer * Remove unnecessary changes * Address Gemini comment * Restore rootClaim fix * Remove duplicate flag * Fix devnet build * Fix go version, add timeout * Remove duplicate import * Fix go module flakiness * Fix refactored types and functions * Set light client * Add timeouts for devnet tests * Investigate test failure * Fix integration tests 0 * Fix fallback batcher test * Fix duplicate devnet running issue * Specify artifact names * Fix fmt * Fix challenge game test * Try fix batcher restart test * Fix fallback test * Fix test build * Remove duplicate builds * Fix parsing * Fix duplicate l1-geth-image * Increase timeout * Fix fallback * Fix batcher restart test * Fix devnet tests 3 and 4 * Fix contracts * Fix fmt * More devnet tests * More contract tests * Update version for contract tests * Fix fmt * Fix foundry * Fix CI for devnet tests * Ignore warning * Fix remaining contract tests * Fix script * Fix EOA path * Fix devnet test command * more yaml fix * Fix docker compose spinup * Remove blockscout * Move blockscount to monitoring profile * Free space * More docker fix * Fix more * Fix more * Add investigation log * Fix beacon * Fix timeout * Fix docker compose dir * Fix path * More sequencer fixes * Fix sequencer * More CI fix * Revert devnet test fixes * Restore more devnet files * Add back image fix * Restore streamer * Restore a devnet fix * Restore ec2 test fix * Restore l1 geth fix * Fix throttle * Restrict throttle fix scope * Remove isActiveBatcher * Remove unnecessary changes * Restore fallback path fix * Restore fmt fixes * Ignore cache error * Revert foundry version and fmt fixes * remove fmt check * fix: mise install (#366) * Add back necessary contract files * Fix contract workflow * Address comments for espresso/docker * Update espresso/scripts/run-tests-github-actions.sh Co-authored-by: Theodore Schnepper <Ayiga@MSN.com> * Remove use of output file * Remove path from contract names * Restore more files * Restore more files * Revert IproxyAdmin changes * Remove unneeded IProxyAdmin uses * Remove more files * Add back toml * Add lock * Replace build * Fix build timeout * Fix duplicate build * Fix more * Match Celo's changes * Restore contract files * Update batcher fallback test * Improvement the comment * Remove dead code * More conflict fixes * Fix test 11 * Address gemini comments * Fix test 8 * Restore gotestsum * Undo unnecessary changes * Remove helper functions * Add a missing file * Improve sha256 installation * Fix Dockerfile build * Restore artifactsfs * Fix Go version --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Theodore Schnepper <Ayiga@MSN.com> Co-authored-by: Philippe Camacho <philippe@espressosys.com> Co-authored-by: Sishan Long <dlsubjam@gmail.com> Co-authored-by: Jeb Bearer <jeb.bearer@gmail.com> Co-authored-by: Artemii Gerasimovich <artemii@espressosys.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Theodore Schnepper <tschnepper@espressosys.com> Co-authored-by: Adrian Sutton <adrian@oplabs.co> Co-authored-by: EC2 Default User <ec2-user@ip-172-31-47-181.us-east-2.compute.internal> Co-authored-by: Sneh Koul <snehkoul1999@gmail.com> Co-authored-by: Jean Gal <45081726+jjeangal@users.noreply.github.com> Co-authored-by: miguelCyclone <miguelhongo90@gmail.com> Co-authored-by: Sneh Koul <35871990+Sneh1999@users.noreply.github.com> Co-authored-by: OpenCode <noreply@opencode.ai> Co-authored-by: Jean Gal <jgatbkk@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
shenkeyao
added a commit
that referenced
this pull request
Mar 20, 2026
* Fix caff-node stalling (#213) * Revert timing changes for beacon * just command to run the devnet tests. * Comment out running the devnet tests in CI. --------- Co-authored-by: Philippe Camacho <philippe@espressosys.com> Co-authored-by: Keyao Shen <shenkeyao@gmail.com> * Support timestamp env var (#218) * IL3 Remove redundant "Walking back L1Block" and "will retry" logs (#221) * Remove logs * Restore driver log * Remove retry log * Restore a log * Skip BatchFuture (#217) * Fix length check (#216) * IA1.6.1 Add batcher service running in TEE (#205) * a working script without args * a working script without args * everything works in the scripts despite the args * fix socat proxy script * working op-batcher inside docker-compose * rename the script to build batcher enclave image * cleanup and profile the op-batcher-non-tee * use port number from env and shorten nc listener timeout as it will not be used in most cases * fix dasel format * remove uneeded ESPRESSO_RUN_ENCLAVE_TESTS * fix scripts * Add op-batcher-tee image in CI (#210) * push op-batcher-tee image init * fix tag and push * test image creation without enclaver * try to use env * fix enclaver download * use env in docker images yml * restore other task * remove unneeded steps * special case to common case * use default for op-batcher and tee for op-batcher-tee * fix double ports mapping * fix batcher restart test * add a script to use enclave tool * works to some extend * also works for passing in arguments from cmd * try to upload the image * add my branch patter * fix dockerfile * a simplified version * adding packages/contracts-bedrock/forge-artifacts to op-batcher-enclave-target * PCR0 registered in op-batcher-tee docker compose and add monitor for enclave logs * copy deployment/ to op-batcher-enclave-target * fix docker-images * Remove unneeded script * remove unneeded script and cleanup readme * fix overlapping ports and move long cmd of op-batcher-tee to script * update readme * Fix batcher restart test (#222) * Fix batcher restart test * Tune parameters to be more realistic (in particular, increasing parallelism to reduce bottleneck on slow L1) * Improve logging * Fix go lint * Download binaries for appropriate architecture in Docker images (#223) * Add key rotation tests (#224) * Remove a Caff node comment (#225) * Remove a comment * Restore devnet test in CI * Disable the CI again * Bump github.com/ulikunitz/xz from 0.5.12 to 0.5.14 (#220) Bumps [github.com/ulikunitz/xz](https://github.com/ulikunitz/xz) from 0.5.12 to 0.5.14. - [Commits](ulikunitz/xz@v0.5.12...v0.5.14) --- updated-dependencies: - dependency-name: github.com/ulikunitz/xz dependency-version: 0.5.14 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Test a challenge game in the docker devnet (#228) * Add test for devnet challenge game * Fixes * Make sure batcher key used by default in the devnet tests is the same one registered in the inbox contract * Remove check in batcher that prevents it from sending transactions to Espresso immediately * Ensures that the deployment files are deleted before building a new devnet. Update README_ESPRESSO.md to remind running docker as a non root user. * Run devnet tests on CI again. * Ensure deployment files are not written by the root user. * Ignore rotate batcher key and change batch inbox owmer tests. * Clean way of setting UID and GID. * Ignore devnet tests for now so that we can merge. * Add fallback values for UID and GID. * Pinpoint forge version in CI as the linter is complaining. * Add comment regarding the number of claims. * Add comment to function TaggedWriter.Write. --------- Co-authored-by: Philippe Camacho <philippe@espressosys.com> * Run smoke devnet test in CI (#231) * Add smoke test for devnet. * Run Game Challenge and smoke test in CI. * Run all the devnet tests locally. * push (#232) * forget this commit (#233) * Add a Buffered Streamer around Espresso Streamer for batcher (#230) * Add a Buffered Streamer around Espresso Streamer for batcher Because the `EspressoStreamer` is getting `Reset` during the Batcher process when building a batch to submit to the `L1`, it hinders progress of the chain in a reasonable amount of time, which ultimately causes it to stop creating non-empty blocks. There are a number of factors that are contributing to this issue, but ultimately the `Reset` is causing the `EspressoStreamer` to restart from `0` and it takes too long before it catches back up to the next expected batch. To remedy this, a Buffer can be used to mitigate this `Reset` and revert to a point that is much closer to the desired next batch. In testing it has been observed that the `SafeL2` can sometimes move backwards. To safe guard against this, it is better to `Reset` to the `FinalizedL2` position instead of the `SafeL2` as this behavior has not been observed there. * Rename EspressoStreamer and EspressoStreamerIFace Based on feedback provided by @QuentinI in PR review: #230 (comment) The name `EspressoStreamerIFace` is quite a long name, and the `IFace` suffix isn't necessary since one could tell it's an `interface` by inspection, or using an `LSP`. The feedback provided by @QuentinI suggested to rename `EspressoStreamerIFace` to just `EspressoStreamer` so that it falls in line with our other code approaches. This change renames `EspressoStreamer` to `BatchStreamer` This change renames `EspressoStreamerIFace` to `EspressoStreamer`. * Remove `RemainingBatchesLen` method Based on feedback received from @QuentinI: #230 (comment) The only reason `RemainingBatchesLen` exists is to serve as a check, and issue a warning when things are running. Even though this matches the existing behavior this log seems to overlap with the logs corresponding to Undecided Batches which already log warnings or errors. As a result this method, and the log that utilizes it, seem to be unnecessary and should be removed to eliminate noise. This change removes `RemainingBatchesLen` and the uses of it * Add `RefreshSafeL1Origin` to `EspressoStreamer` interface Based on feedback provided by @QuentinI: #230 (comment) Since the `RefreshSafeL1Origin` method can potentially be utilized in some places that do not require a full `Refresh`, and for convenience, it makes sense to allow it to be a separately exposed method distinct from `Refresh`. This change adds `RefreshSafeL1Origin` as a required method in the `EspressoStreamer` interface. * Fix missed renamed references * Rename enclave smoke test * Fix refreshSafeL1Origin logic for Buffered Streamer The buffered streamer is resetting its read position far more than it needs to, ultimately reproducing the same issue that was already occurring with the unbuffered version. In inspecting the behavior with a debugger, it seems we're resetting the reset position unnecessarily when we receive the same safeL1Origin again. Additionally, the logic for determining the read position when the safeL1Origin advances also seems flawed, in that it is very likely to reset too far in the past. We really want to keep our relative read position unless we're explicitly told to Reset. This change addresses these issues in order to try and smooth out the batches being returned, and avoid unnecessary reprocessing of previous batches. * Add Unit Tests for BufferedEspressoStreamer Fix BufferedEspressoStreamer behavior While adding unit tests for the BufferedEspressoStreamer it was noticed that the position of the L2 and the L1 for the Buffer were being mixed together at times. This would ultimately lead to very difficult to detect bugs based on observed behavior alone. With the addition of the unit tests identifying the issue, the buffer adjustment behavior has been adjusted to apply to the L2 position in isolation away from the L1 positions. The L1 positions will cause a larger Reset in the underlying logic. * Rename tests (#236) * Remove unneeded service http proxy for docker compose (#238) * Fix `TestSmoke` failing on CI/CD (#237) The sigp/lighthouse docker image was upgraded from version `v7.1.0` to `v8.0.0-rc.0` on `2025-09-29`. Since the image isn't anchored to a version, this update gets pulled in, and it seems to have breaking changes with our previous setup. This change sets the version of the docker image used specifically to `v7.1.0` so that the previous behavior we're used to is seen. Additionally, when `TestSmoke` is running, it initially **MUST** download the images for the docker containers that wer not built in the `Build Devnet` job. This delays the launch and running of the DevNet by quite a bit. Fix this delay by adding a `docker compose pull` setp to `Build Devnet` * Fix CI after rebasing celo-14 (#243) * fix fast tests * fix go version in dockerfile and ce in streamer_test.go * update prepare-allocs.sh * fix prepare-allocs.sh * try to fix l1-geth docker * fix op-stack dockerfile * fix op-stack dockerfile * try to fix l1-geth dockerfile * fix config read * try to fix l1-geth dockerfile * try to fix challenger gamer * TN5 withdrawal devnet test (#226) * Add smoke test for devnet. * Add test to ensure L2 funds can be withdrawn to L1. --------- Co-authored-by: Theodore Schnepper <tschnepper@espressosys.com> * Rename DevNet to E2eDevnet (#239) * Rename DevNet to E2eDevnet * Remove duplicate name * Fix low gasLimit in L1 genesis (#241) * update (#244) * Streaming streamer (#235) * Fix pcr0 extraction in docker compose script and correctly shutdown op-proposer-tee (#246) * fix pcr0 extraction * stopping op-proposer-tee in script * revert to old pcr0 extraction as the new pattern breaks the old pattern * try to fix pcr0 extraction * upload op-proposer-tee * Update metrics (#242) * Update log level (#247) * Decouple Espresso L1 & OP L1 (#248) * Add back forgotten config when rebase celo-sync-14 * Fix prepare-allocs after rebasing celo-sync-14 (#250) * remove doulbe init() * try to fix prepare-allocs.sh * enable more workflow * Skip TestChallengerGame and TestWithdrawal (#251) * Trigger docker-images workflow * uncomment dasel put * skip TestChallengeGame * skip TestWithdrawal * mise: Define fake install sources for disabled tools (ethereum-optimism#18109) (#254) Co-authored-by: Adrian Sutton <adrian@oplabs.co> (cherry picked from commit de37992) * Change the logging level for the safe L2 number (#252) * Update log level * Reduce logging level (cherry picked from commit dd97e4c) * Respect espresso.fetch-api flag (#253) (cherry picked from commit 6d00dcb) * Readd devnet tests to CI. (#257) * Readd devnet tests to CI. *Fix for batcher restart test in CI. (cherry picked from commit f37ea91) * Add a log debouncer to op-service.log package (#259) (cherry picked from commit 788d28f) * Add netcat-openbsd to Dockerfile (#262) * Update log level * Add duplicate command from Terraform (cherry picked from commit b775430) * Add a devnet cleanup script (#261) * Update log level * Add cleanup script * Remove unnecessary commands (cherry picked from commit dc40b6c) * Enable circleCI after rebase14 (#265) * Fix .circleci/config.yaml and lint and most of circleCI tests after rebasing celo-sync-14 * Skip tests (in TEST_PKGS) that need auth or celo-specific rpc specified --------- Co-authored-by: Artemii Gerasimovich <artemii@espressosys.com> * Add origin height to Espresso streamer (#255) * Clean up more loggings (#266) * Reduce logs * Remove one more log * Skip attestation verification (#263) * Update espresso-tee-contracts submodule to sishan/skip-attestation-verification * Skip attestation verification to reduce gas costs * Reduce L1 gas limit from 45M to 16M * Update snapshots for registerSignerWithoutAttestationVerification * Ignore lib/automate submodule directory * fix CI * Update espresso-tee-contracts submodule Remove onlyOwner modifier from registerSignerWithoutAttestationVerification * keep large gasLimit * circleci: Enable workflow on all branches via API trigger Allow CircleCI main workflow to run on any branch when triggered via API, not just webhook triggers. This enables go-lint and go-tests to run on feature branches. * Regenerate semver-lock.json after rebase The initCodeHash for BatchAuthenticator needed to be regenerated after rebasing onto celo-integration-rebase-14.1. --------- Co-authored-by: EC2 Default User <ec2-user@ip-172-31-47-181.us-east-2.compute.internal> * Fix test TestChangeBatchInboxOwner (#264) * Fix the test TestChangeBatchInboxOwner. * Ensure the owner of the Batch Authenticator contract is initialized. * Update README and relevant scripts (#269) * Remove unused metrics (#273) * Rename (#275) * Improve image versioning and repo consistency (#276) * Add githooks and env * update path * test hooks * change wording * include docker compose * Create enclave ami for enclave test (#277) * create enclave ami * use new ami and restore the ci workflow and update enclave prepare ami script * new AMI * update readme * Add devnet smoke test with TEE (#268) * Add devnet smoke test with TEE * Remove unnecessary extra timeout * Add consts, remove incorrect setting * Use consts * Add missed file * Blockscout running inside the local devnet (#281) * Extend the local devnet with blockscout. * Pinpoint versions of blockscout images. * Blockscout fetching blocks from caff node. * Build deployer image in CI * Upate CI utils * Saner 'confirmed' logging * Don't error out on light client issues * fix the name of deployer factory address * Add a workaround for query service lag in real-world networks * Generate more metadata * More faithful compiler output in verifier * Don't fall below hotshot origin height * Remove cache buster to speed up docker image builds * Adjust channel duration in devnet * Jump ahead when origin is too low * Add log line to matching Espresso txn to L2 block * Fix semver lock * Fix snapshot lock * Support environment variables for channel parameters * Enable EigenDaProxy & MEMSTORE (#274) * Enable EigenDaProxy & MEMSTORE * longer eigenda-proxy start period * enable eigenda at the op-node level * Don't copy artifacts to batcher image (#290) * Refactor: replace MultiNode majority rule with SingleNode client and skip deprecated test. * refactor: remove majority rule and switch to single Espresso client * Skip deprecated TestEnforceMajorityRule (deprecated under SingleNode) * Fallback Inbox contract changes (#278) * Implement changes in the Batch Inbox / Batch Authenticator contracts to support a TEE and non TEE batcher. * Add some unit tests for the Batch Authenticator and Batch Inbox contracts. * Remove the failing Circle CI tests * OP succint support (#287) Deploy OP Succint contracts on L1. Spins up op-succinct-challenger and op-succint-proposer services. Adjust the devnet test `TestChallengeGame`. Deletes the Demo we made to Celo Labs as we can now spin up a devnet with Blockscout. * Update error handling (#289) * Update error handling * Fix typo Co-authored-by: Phil <philippe@espressosys.com> --------- Co-authored-by: Phil <philippe@espressosys.com> * Document configuration of all services (#291) * Add readme for config * Insert image, add more description * Support Sepolia Devnet with TEE (#288) * update enclave-entrypoint.bash to correctly deal with external url * preserve host name for external url * Skip IsURLAvailable TCP check when using HTTP proxy * skip VerifyCertTransaction for now * reuse socat so that it can work for internal url * comment and skip TestE2eDevnetWithInvalidAttestation * OP Succinct: Making changes to the derivation pipeline (#293) * Document how to make changes to the kona repository and propagate them. * Reference new docker images for the op-succinct proposer and challenger. * Fix op-succinct dependencies diagram. (#297) * Simplify checks in the derivation pipeline (#296) * Remove the superfluous check about the batcher address as now the Batch Inbox contract verifies the sender is legitimate. * Removed nonexist logs (#298) * Add support for ZK attestation service (#294) * Add support for ZK attestation service * check attestation service url is not nil * upgrade espresso tee verifier contracts * fix contracts * fix merge * fix tests * bring back deploy aws nitro * add support for mock contract * add support for attestation verifier service * fix tee tests * use higher version of github runner * fix tee args * fix tee args * add healthcheck to attestation verifier zk * increase timeout * Invalid attestation test passing * small fixes * fix TestE2eDevnetWithUnattestedBatcherKey * fix health check * fix devnet test * use 127.0.0.1 * fix regex * debug * fix proof generation * debug * fix url * fix url * remove debug logs * resolve based on comments * address comments * update github runner enclave * fix based on suggestions * cleanup logs * Enable AltDA Espresso E2E using EigenDA Docker proxy (#295) * Integrate EigenDA via Docker proxy for AltDA Espresso E2E tests * Scope EigenDA lifecycle to the test and ensure clean startup/teardown * Extract EigenDA Docker port and image into constants * Downgrade Dasel (#303) * Make attestation service url optional * fix dasel * fix dasel * update dockerfiles * make attestation service required again * Make the withdraw devnet test pass again (#301) * Withdraw test passing again on devnet. * Faster CI * Deposit into L1 before requesting the withdrawal on L2. * Add migration related things to readme (#302) --------- Co-authored-by: Philippe Camacho <philippe@espressosys.com> * Reducing logging when outputting the batch (#304) * update logging for the batch * clean up * Document code sync procedure (#308) * Add code sync procedure * Update links * Fix format * Rename files * Update batchAuthenticator according to audit report (#309) * update batchAuthenticator according to audit report * gen bindings and fix fast-tests * Port ForcedTxs test into devnet test suite (#306) * Simplify the test as we cannot in practice reduce the window size. --------- Co-authored-by: Philippe Camacho <philippe@espressosys.com> * Reorder checks of isValidBatchTx in derivation pipeline (#310) * remove warning on every failed tx * reorder the checks * Add fallback mechanism test (#305) * Fallback mechanism test * Update op-e2e/system/e2esys/setup.go Co-authored-by: Phil <philippe@espressosys.com> --------- Co-authored-by: Philippe Camacho <philippe@espressosys.com> * Philippe/fix withdraw flakiness (#312) * Address flakiness. * Simplify the code * Fix CI --------- Co-authored-by: Keyao Shen <shenkeyao@gmail.com> * Use unified run-enclave.sh script for op-batcher-tee (#299) * update single run-enclave.sh * remove BATCHER_PRIVATE_KEY * update run-enclave.sh * Test fallback mechanism on devnet (#313) * Recovery from fallback batcher (#315) * Fallback recovery * Add caff node * Suggestions * Make ZK Verifier Optional for E2E Testing (#321) * Make Attestation Verifier Service optional When the Attestation Verifier Service was added to the integration it fundamentally modified the testing experience, requiring external environment variables to be populated in order to run the tests. Additionally, these environment variable requirements were not documented in the README_ESPRESSO.md file for reference. This change modifies the Attestation Verifier Service setup for the E2E testing environment to make it opt-in instead of being forced to be enabled. Additionally, the Verifier URL is no longer required to run the Batcher. This is a double-edged sword, however, as it means that we could potentially deploy the service without the configuration, and we would potentially be lacking the registered attestation. This may be resolvable with a slight modification to the service configuration, that we would ultimately disable for the E2E testing environment. * Fix misspelling Fix linting error that has caught a misspelling of the work 'Network'. * Modify configuration address to be required from CLI With the change of making the Espresso Attestation Service optional we removed the CLI configuration check that occurs on launch, so that the E2E tests can still be run. This has an unfortunate side-effect of allowing the Batcher to be launched in a state where it is unable to operate as intended due to user error. The only indication being a `WARN` log entry to inform him/her of his/her mistake. This sort of approach is generally discouraged, yet we still need to be able to bypass this check for testing purposes. As a result the `EspressoAttestationService` value has been modified from being a simple `string` to being an interface whose value is inspectable and not allowed to be empty by default. This allows for the test configurations to overwrite this behavior, and allow an optional value in the cases where it is needed. This should preserve the prior behavior of erroring on launch when the parameter is not configured or specified, and should also preserve the new behavior where it is explicitly disabled in tests. * Fix some nil references The EspressoAttestationService configuration value being an interface makes it a `nil`lable value by default. Care needs to be taken when accessing this value an referencing it. This change adds some additional care in referencing the value stored within. * Fix nil access error The `l1Client` being created assumes that the `sys` returned from the call is non-nil before checking the error. This is not guaranteed, and is most likley not ever the case. As a result there is a potential for an error do to attempting an access on a `nil` value. By moving the `l1Client` declaration after the error check, we avoid the potential for this issue. * Apply linting and formatting changes * Fix e2e tests - populate default EspressoAttestationService With the modification of the EspressoAttestationService to an interface instead of an individual value, we need to ensure that the default way of launching the Espresso E2E DevNet results in the value being populated with an empty allowed value. This still allows for extension and override, without requiring the value to be specified, which is our intention. This was missed when adding the capability originally. * Cleanup code practices We have duplicated code that makes the maintenance burden more difficult than it needs to be. In many of these scenarios the code that is duplicated differs by only a single line. Instead of making the system more flexible, we ended up duplicating code paths. This increases the maintenance burden by needing to ensure that these code paths match in every case where they do not differ, yet they are independent of each other. This is not a great approach. Additionally, we end up with multiple starting points for something that should not need them. We also end up storing a configuration that is unnecessary to store. This incurs conditional checks where some are not needed, and ends up making the approach be more confusing than it needs to be. This change aims to replace these approaches with one that adheres to the functional option approach and preserves the existing behavior. * Revert EspressoAttestionService to a `string` As it so happens we rely on the `CLIConfig` for `Espresso`, and the `Batcher` to be serializable. By utilizing an `interface`, we run into trouble doing this. Due this constraint, the `interface` constraint is not feasible. This change reverts the value back to a `string`, which should result in a smaller overall change. It also opts for a private configuration value that is inspectable by the `Verify` check, but not directly configurable. We expose a method to allow for it to be configured, so it can only occur within code within the code base itself. We should only invoke this via Testing where we need the value to be optional. This achieves the same result but in a different way. | NOTE: There may be a better approach to this as well, isntead of having this be a separate field, we could do something akin to sql.NullString, where we encode this value as a Marshable `struct`. The acess pattern becomes different, but we could directly encode the empty allowance into the struct itself. * Add Espresso Attestation Verifier Service to Enclave Test The Enclave tests are currently failing in CI. It is dying due to an error stemming from the lack of the EspressoAttestationService being configured. It is likely that this is required for the Enclave tests specifically. As a result, we need to add and enable it for the enclave tests. * Modify LaunchBatcherInEnclave option The LaunchBatcherInEnclave essentially launches the batcher externally within an enclave. This option actually relies on the Espresso Attestation Verifier Service to be running. This is due to the Espresso Attestation Service only being optional inside of a test environment. When launched externally, the Batcher is no longer considered to be in a "test environment", or configurable for testing. As a result, its configuration **MUST** be something that can actually be resolvable from a CLI launch. Since the Espresso Attestation Verifier Service check is only disabled within the testing environment, this means it **MUST** be enabled in the enclave. For convenience, this option has been added automatically as a part of the LaunchBatcherInEnclave option, since it depends on it. This will minimize accidentl misconfigurations. * Tee support for EigenDA (#319) * add eigenda_proxy_url to op-batcher-tee * fix the url to post to eigenDA * not hardcoding EIGENDA_PROXY_PORT * fix the block height config * Add Batcher Fallback: Channel Not Closed Test (#314) * Add test to check end of channel fallback Asana task: https://app.asana.com/1/1208976916964769/project/1209976130071762/task/1211892212379885?focus=true We need a test to check the fallback Batcher behavior in the event that the Espresso Batcher is able to submit a partial Channel that is im progress. The specific scenario we want to test for is one concerning a multi-frame channel that has had at least part of the full channel submitted to the L1 by the Espresso Batcher, then no more. After which we swap to the Fallback Batcher, and we should be able to pick up the missed / incomplete channel, and complete the transactions. * Rename helper function to match naming pattern * Fix lint issue with not checking error result of wait.For * Commit work in progress multi frame channel efforts * Adjust settings to successfully trigger multi-frame channels After a mob programming session @quentinl was able to help identify a a specific combiniation of parameters to successfully and consistently trigger multi-frames within the Batcher. This condition is a necessary precusor to the test being attempted. This commit updates the test with the information necessary to trigger this condition and sets the necessary test criteria that we are aiming to achieve. * Perform some code cleanup This change does a few things: - Address linting issue causing CI failure - Adjusts some golang forloop usage to be more modern - Adjust function call signatures to remove unused variables * Fix bug tracking unsuccessful frames in test In the `TxManagerIntercept` there is a bug that appends the successful frames to the unsuccessful ones. While this bug isn't great in the information that it taints, it doesn't actually have the large of an impact on the test as a whole, as the resulting failure condition would be triggered regardless. This bug does affect the accurate tracking of failed frames which could be valuable information for inspection. * Update espresso/environment/e2e_helpers.go Co-authored-by: Phil <philippe@espressosys.com> * Replace Disable Batcher setting references There are a number of places in our testing setup where we are explicitly preventing the Batcher from starting on launch. Instead of rewriting this same option every time we want to use it, we should reference a built in option that we can reference continually. This allows for non-repeated code and improved documentation as to the point and purpose of this option. * Refactor custom wait in test There's a condition being waited on in the switch to fallback batcher test. This wait is useful, and can be reused between tests. But the wait itself is somewhat hiding it's intention by being inline defined within the test itself. We should pull this wait out so it can be easily used, and its intention / purpose can be more easily documented. * Cleanup code reuse in frame decoding When decoding frame information for one of the Batcher fallback tests, there are similar code paths taken that result in most of the code being reused. We should clean up this code reuse so that we don't repeat ourselves in order to avoid diverging logic. Additionally, it allows us to reduce the amount of code needing to be maintained, and more clearly document the intention of the code, and the consistency with how we perform this frame decoding process. * Relocate deferred stop calls The Stop calls should occur as close to the launch of the environment as possible. As a result, any deferred calls to Stop for the system or the Espresso Dev Node should occur as close to their occurence as possible. * Modify Initial L2Verif wait to be longer With the specific Frame and Channel settings being specified in the `TestFallbackMechanismIntegrationTestChannelNotClosed` test, the initial startup check for the L2 Verifier is failing. This is due to our settings requiring the Verifier process to take a bit longer than normal. In general, we want to give it more time, but the time frame for the failure is hard-coded in the `wait` function being utilized. While we **could** add a simple `time.Sleep`, and this would work, this is generally a bad appraoch as it just adds an unchecked delay. Instead, we opt to utilize a simple `retry` for up to `n` times. In this case, we only need to wait up to `3x` the normal time, so ensure that we perform at least `3` times. * Fix failure in Batcher Fallback test The TestFallbackMechanismIntegrationTestChannelNotClosed test fails locally without stopping, in spite of the overall time limit being specified on the test. After some troubleshooting and debugging, We were able to chase down the cause to be due to the `RunSimpleMultiTransactions`. It's unclear as to why this was causing the process to hang for as long as it was. It seemed to not be handling timeout errors well for some reason. Either way, we fority this helper by setting an explicit time limit on it, and referncing the context whenever we're performing channel operations. This should allow the channel operations themselves not to block and hang the test. After this modification we were able to determine that this process was failing due to insufficient gas being provided. For some reason when running the transactions through this mechanism, we require even more gas than we're normally need. This seems a bit odd, perhaps it has to do with the differences in the transaction construction. In any case, we up the gas being provided so that this becomes a non-issue. * Fix linting issues * Update espresso/environment/tx_helpers.go Co-authored-by: Phil <philippe@espressosys.com> * Correct failure vs success in Send The triggered conditions for failures and successes are backwards in the `Send` method of `TxMangerIntercept`. Their specific frame markers should be switched. * Update espresso/environment/14_batcher_fallback_test.go Co-authored-by: Phil <philippe@espressosys.com> --------- Co-authored-by: Phil <philippe@espressosys.com> * Move diagram files (#326) * Update Succinct image versions, update diagram (#329) * Inactive Batcher Shouldn't Post (#316) * Check if the batcher is active before publishing to L1/DA * fix readme lint * more lint fixes * check batcher contract * Fix endless warning * add batch authenticator address to rollup config * handle contract undeployed error * attempt test in CI * add test to CI * Revert "add test to CI" This reverts commit 2a9678a. * add test to CI * remove jg/ from branches * attempt to clean up and make the test more reliable * fix ci error WaitUntilSafe undefined * revert 07a82bf * Fix `anvil_setBalance` not found error * Simplify isActive check * add batcher-active-publish-only to devnet tests justfile * - simplify test, one less batcher switch - increase timeouts for devnet test * Cleaned up the code, raise tx waiting time to 60s * Brought back original timeouts * started fallback batcher up + lint fix docker compose file * Ensure that in Espresso mode the batch authenticator address is set. * Removing all changes to driver.go and the tests are still passing. --------- Co-authored-by: Philippe Camacho <philippe@espressosys.com> * Removes PreRegisteredBatcher code (#327) * Remove pre authenticated batcher * fix test * Update Succinct images * Streamer namespace range 14.2 (#334) * Support namespace range endpoint (cherry picked from commit a73f7b6) * fix buils (cherry picked from commit e46909b) * update docker image (cherry picked from commit 0774898) * fix streamer tests (cherry picked from commit f752aa2) * fix streamer tests (cherry picked from commit 168426e) * fix tests (cherry picked from commit b942c28) * fix tests (cherry picked from commit b96622c) * use docker instead of cargo to generate allocs.json (cherry picked from commit efee3ac) * fix readme * address comments * remove fetch api * Enable and test Transparent proxy upgradability and batcher address update (#337) * Enable upgradability * Fix fmt * Fix file name * Fix tests * Clean up tests * Force clean build * Add temp artifact verification * More verification for artifact verification * Fix build command * Fix artifact * Fix artifact * Fix script * Fix and simplify the script * Fix proxyAdmin as well * Add back verification workflow * Fix more workflows * Restore version * Use EspressoTEEVerifierMock * Fix TeeType conversion * Fix fmt * Fix enum conflict * Fix version in test * Fix byte requirement * Add error * Enable batcher address update * Fix typo and remove redundant tests * Fix owner * Fix test build * transfer owner * Fix more tests * Fix devnet test * Fix unused param * Fix ec2 test * Fix enclave test * Fix fmt * Fix circleCI * Fix fmt again * Cleanup * Move events and errors to interface * Fix build * Update proxy admin permission * Description for TestBatcherSwitching (#335) * add description for TestBatcherSwitching * Update espresso/environment/14_batcher_fallback_test.go Co-authored-by: Keyao Shen <shenkeyao@gmail.com> * Update espresso/environment/14_batcher_fallback_test.go Co-authored-by: Keyao Shen <shenkeyao@gmail.com> * Update espresso/environment/14_batcher_fallback_test.go Co-authored-by: Keyao Shen <shenkeyao@gmail.com> * Update espresso/environment/14_batcher_fallback_test.go Co-authored-by: Keyao Shen <shenkeyao@gmail.com> * Update espresso/environment/14_batcher_fallback_test.go Co-authored-by: Phil <philippe@espressosys.com> * Update espresso/environment/14_batcher_fallback_test.go Co-authored-by: Phil <philippe@espressosys.com> --------- Co-authored-by: Keyao Shen <shenkeyao@gmail.com> Co-authored-by: Phil <philippe@espressosys.com> * Add cmd to shutdown all docker containers with TEE (#332) * cmd to shutdown all services * Small change to trigger CI --------- Co-authored-by: Keyao Shen <shenkeyao@gmail.com> * Guardians rebased (#345) Co-authored-by: OpenCode <noreply@opencode.ai> * Audit Document (#339) Co-authored-by: Philippe Camacho <philippe@espressosys.com> * Security Analysis (#342) --------- Co-authored-by: Jean Gal <jgatbkk@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Keyao Shen <shenkeyao@gmail.com> * Fix and improve steps in the code sync doc (#344) * Update doc * Update kona default branch and fix links * Fix typo * Typos --------- Co-authored-by: Philippe Camacho <philippe@espressosys.com> * Fix op-deployer build * Fix go mod and duplicate flag * Fix go-ffi * Fix prepare-allocs * Fix duplicate attribute in pipeline * Fix test slice * Fix builder version * Set timeout for docker-images CI * Fix devnet tests * Add missing file * Fix go version * Fix go version * Add missing address * Use generic way to generate slice * Fix go version for build-op CIs * Fix dockerfile * Continue devnet version fix * Fix dockerfile * More dockerfile fix * More dockerfile fix * Simplify changes * More dockerfile fix * More dockerfile fix * Add missing event type * Fix op-node * Fix op-batcher * Fix batcher TEE and proposer * Remove unnecessary changes * Address Gemini comment * Restore rootClaim fix * Remove duplicate flag * Fix devnet build * Fix go version, add timeout * Update moved crate * Fix the build after rebase (#352) * Fix op-deployer build * Fix go mod and duplicate flag * Fix go-ffi * Fix prepare-allocs * Fix duplicate attribute in pipeline * Fix test slice * Fix builder version * Set timeout for docker-images CI * Fix devnet tests * Add missing file * Fix go version * Fix go version * Add missing address * Use generic way to generate slice * Fix go version for build-op CIs * Fix dockerfile * Continue devnet version fix * Fix dockerfile * More dockerfile fix * More dockerfile fix * Simplify changes * More dockerfile fix * More dockerfile fix * Add missing event type * Fix op-node * Fix op-batcher * Fix batcher TEE and proposer * Remove unnecessary changes * Address Gemini comment * Restore rootClaim fix * Remove duplicate flag * Fix devnet build * Fix go version, add timeout * Remove duplicate import * Fix go module flakiness * Fix refactored types and functions * Set light client * Add timeouts for devnet tests * Investigate test failure * Fix integration tests 0 * Fix fallback batcher test * Fix duplicate devnet running issue * Specify artifact names * Fix fmt * Fix challenge game test * Try fix batcher restart test * Fix fallback test * Fix test build * Remove duplicate builds * Fix parsing * Fix duplicate l1-geth-image * Increase timeout * Fix fallback * Fix batcher restart test * Fix devnet tests 3 and 4 * Fix contracts * Fix fmt * More devnet tests * More contract tests * Update version for contract tests * Fix fmt * Fix foundry * Fix CI for devnet tests * Ignore warning * Fix remaining contract tests * Fix script * Fix EOA path * Fix devnet test command * more yaml fix * Fix docker compose spinup * Remove blockscout * Move blockscount to monitoring profile * Free space * More docker fix * Fix more * Fix more * Add investigation log * Fix beacon * Fix timeout * Fix docker compose dir * Fix path * More sequencer fixes * Fix sequencer * More CI fix * Revert devnet test fixes * Restore more devnet files * Add back image fix * Restore streamer * Restore a devnet fix * Restore ec2 test fix * Restore l1 geth fix * Fix throttle * Restrict throttle fix scope * Remove isActiveBatcher * Remove unnecessary changes * Restore fallback path fix * Restore fmt fixes * Ignore cache error * Revert foundry version and fmt fixes * remove fmt check * fix: mise install (#366) * Add back necessary contract files * Fix contract workflow * Address comments for espresso/docker * Update espresso/scripts/run-tests-github-actions.sh Co-authored-by: Theodore Schnepper <Ayiga@MSN.com> * Remove use of output file * Remove path from contract names * Restore more files * Restore more files * Revert IproxyAdmin changes * Remove unneeded IProxyAdmin uses * Remove more files * Add back toml * Add lock * Replace build * Fix build timeout * Fix duplicate build * Fix more * Match Celo's changes * Restore contract files * Update batcher fallback test * Improvement the comment * Remove dead code * More conflict fixes * Fix test 11 * Address gemini comments * Fix test 8 * Restore gotestsum * Undo unnecessary changes * Remove helper functions * Add a missing file * Improve sha256 installation * Fix Dockerfile build * Restore artifactsfs * Fix Go version --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Theodore Schnepper <Ayiga@MSN.com> Co-authored-by: Philippe Camacho <philippe@espressosys.com> Co-authored-by: Sishan Long <dlsubjam@gmail.com> Co-authored-by: Jeb Bearer <jeb.bearer@gmail.com> Co-authored-by: Artemii Gerasimovich <artemii@espressosys.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Theodore Schnepper <tschnepper@espressosys.com> Co-authored-by: Adrian Sutton <adrian@oplabs.co> Co-authored-by: EC2 Default User <ec2-user@ip-172-31-47-181.us-east-2.compute.internal> Co-authored-by: Sneh Koul <snehkoul1999@gmail.com> Co-authored-by: Jean Gal <45081726+jjeangal@users.noreply.github.com> Co-authored-by: miguelCyclone <miguelhongo90@gmail.com> Co-authored-by: Sneh Koul <35871990+Sneh1999@users.noreply.github.com> Co-authored-by: OpenCode <noreply@opencode.ai> Co-authored-by: Jean Gal <jgatbkk@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #<ISSUE_NUMBER>
This PR:
This PR does not:
Key places to review: