Skip to content

v1.1.1
Compare
Choose a tag to compare
@sduchesneau sduchesneau released this 05 May 14:20
v1.1.1
da937b7

Highlights

This release introduces a new RPC protocol and the old one has been removed. The new RPC protocol is in a new Protobuf package sf.substreams.rpc.v2 and it drastically changes how chain re-orgs are signaled to the user. Here the highlights of this release:

  • Getting rid of undo payload during re-org
  • substreams gui Improvements
  • Substreams integration testing
  • Substreams Protobuf definitions updated

Getting rid of undo payload during re-org

Previously, the GRPC endpoint sf.substreams.v1.Stream/Blocks would send a payload with the corresponding "step", NEW or UNDO.

Unfortunately, this led to some cases where the payload could not be deterministically generated for old blocks that had been forked out, resulting in a stalling request, a failure, or in some worst cases, incomplete data.

The new design, under sf.substreams.rpc.v2.Stream/Blocks, takes care of these situations by removing the 'step' component and using these two messages types:

  • sf.substreams.rpc.v2.BlockScopedData when chain progresses, with the payload
  • sf.substreams.rpc.v2.BlockUndoSignal during a reorg, with the last valid block number + block hash

The client now has the burden of keeping the necessary means of performing the undo actions (ex: a map of previous values for each block). The BlockScopedData message now includes the final_block_height to let you know when this "undo data" can be discarded.

With these changes, a substreams server can even handle a cursor for a block that it has never seen, provided that it is a valid cursor, by signaling the client to revert up to the last known final block, trading efficiency for resilience in these extreme cases.

substreams gui Improvements

  • Added key 'f' shortcut for changing display encoding of bytes value (hex, pruned string, base64)
  • Added jq search mode (hit / twice). Filters the output with the jq expression, and applies the search to match all blocks.
  • Added search history (with up/down), similar to less.
  • Running a search now applies it to all blocks, and highlights the matching ones in the blocks bar (in red).
  • Added O and P, to jump to prev/next block with matching search results.
  • Added module search with m, to quickly switch from module to module.

Substreams integration testing

Added a basic Substreams testing framework that validates module outputs against expected values.
The testing framework currently runs on substreams run command, where you can specify the following flags:

  • test-file Points to a file that contains your test specs
  • test-verbose Enables verbose mode while testing.

The test file, specifies the expected output for a given substreams module at a given block.

Substreams Protobuf definitions updated

We changed the Substreams Protobuf definitions making a major overhaul of the RPC communication. This is a breaking change for those consuming Substreams through gRPC.

Note The is no breaking changes for Substreams developers regarding your Rust code, Substreams manifest and Substreams package.

  • Removed the Request and Response messages (and related) from sf.substreams.v1, they have been moved to sf.substreams.rpc.v2. You will need to update your usage if you were consuming Substreams through gRPC.
  • The new Request excludes fields and usages that were already deprecated, like using multiple module_outputs.
  • The Response now contains a single module output
  • In development mode, the additional modules output can be inspected under debug_map_outputs and debug_store_outputs.
Separating Tier1 vs Tier2 gRPC protocol (for Substreams server operators)

Now that the Blocks request has been moved from sf.substreams.v1 to sf.substreams.rpc.v2, the communication between a substreams instance acting
as tier1 and a tier2 instance that performs the background processing has also been reworked, and put under sf.substreams.internal.v2.Stream/ProcessRange. It has also been stripped of parameters that were not used for that level of communication (ex: cursor, logs...)

Fixed

  • The final_blocks_only: true on the Request was not honored on the server. It now correctly sends only blocks that are final/irreversible (according to Firehose rules).

  • Prevent substreams panic when requested module has unknown value for "type"

Added

  • The substreams run command now has flag --final-blocks-only
Assets 6
Loading