Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

op-program: Use PebbleDB for DiskKV (Reapply with fixes) #11709

Merged
merged 8 commits into from
Sep 4, 2024
Merged

Conversation

ajsutton
Copy link
Contributor

@ajsutton ajsutton commented Sep 2, 2024

Description

Reapplies #11705 with fixes for the test failures.

  • Updates op-challenger to only open preimage source when actually reading preimages. Avoids concurrent access to pebbledb (and ensures the DB is closed again instead of leaking).
  • Handles interrupt signal in op-program to allow for a clean shutdown. Otherwise the preimage database doesn't actually write the stored values to disk and op-challenger can't load them afterwards (as is required to calculate things like kzg proofs).

@ajsutton ajsutton requested a review from a team as a code owner September 2, 2024 04:43
@ajsutton ajsutton force-pushed the aj/reopen branch 3 times, most recently from a550114 to 24f0fc7 Compare September 2, 2024 23:48
@ajsutton ajsutton changed the title Only open preimage source when actually needing preimages. op-program: Use PebbleDB for DiskKV (Reapply with fixes) Sep 2, 2024
@ajsutton ajsutton requested review from Inphi and clabby and removed request for axelKingsley September 2, 2024 23:51
clabby and others added 5 commits September 4, 2024 10:32
* feat(op-program): Use `PebbleDB` for `DiskKV`

* close db

* fix `testFaultProofProgramScenario` tests

* switch to snappy compression

cockroachdb/pebble#3434

* fix tempdir

* update compat release

* defer k/v until preimage server and hinter have both exited
Avoids concurrent access to the pebbledb.
Ensures the preimage kv store is closed cleanly even when the process is sent an interrupt signal.
Enables non-blocking IO so that read calls return io.EOF when a file handle is closed rather than blocking forever.
@ajsutton ajsutton added this pull request to the merge queue Sep 4, 2024
Merged via the queue into develop with commit e80d23b Sep 4, 2024
59 checks passed
@ajsutton ajsutton deleted the aj/reopen branch September 4, 2024 03:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants