v2.3.6

• BlockFetcher: added support for WithdrawalsRoot, BlobGasUsed, BlobExcessGas and ParentBeaconRoot fields when fetching blocks from RPC (for example, to get those values for Optimism)
• Substreams: add support for substreams-tier2-max-concurrent-requests flag to limit the number of concurrent requests to tier2
• Adding traceID for RPCCalls

v2.3.5

Substreams

Warning

This release deprecates the "RPC Cache (for eth_calls)" feature of substreams: It has been turned off by default and will not be supported in future releases.
The RPC cache was a not-well-known feature that cached all eth_calls responses by default and loaded them on each request.
It is being deprecated because it has a negative impact on global performance.
If you want to cache your eth_call responses, you should do it in a specialized proxy instead of having substreams manage this.
Until the feature is completely removed, you can keep the previous behavior by setting the --substreams-rpc-cache-store-url flag to a non-empty value (its previous default value was {data-dir}/rpc-cache)

• Performance: prevent reprocessing jobs when there is only a mapper in production mode and everything is already cached
• Performance: prevent "UpdateStats" from running too often and stalling other operations when running with a high parallel jobs count
• Performance: fixed bug in scheduler ramp-up function sometimes waiting before raising the number of workers
• Added the output module's hash to the "incoming request" log
• Substreams RPC: add --substreams-rpc-gas-limit flag to allow overriding default of 50M. Arbitrum chains behave better with a value of 0 to avoid intrinsic gas too low (supplied gas 50000000) errors

Reader node

• The reader-node-bootstrap-url gained the ability to be bootstrapped from a bash script.

  If the bootstrap URL is of the form bash:///<path/to/script>?<parameters>, the bash script at
  <path/to/script> will be executed. The script is going to receive in environment variables the resolved
  reader node variables in the form of READER_NODE_<VARIABLE_NAME>. The fully resolved node arguments
  (from reader-node-arguments) are passed as args to the bash script. The query parameters accepted are:

  • arg=<value> | Pass as extra argument to the script, prepended to the list of resolved node arguments
  • env=<key>%!d(MISSING)<value> | Pass as extra environment variable as <key>=<value> with key being upper-cased (multiple(s) allowed)
  • env_<key>=<value> | Pass as extra environment variable as <key>=<value> with key being upper-cased (multiple(s) allowed)
  • cwd=<path> | Change the working directory to <path> before running the script
  • interpreter=<path> | Use <path> as the interpreter to run the script
  • interpreter_arg=<arg> | Pass <interpreter_arg> as arguments to the interpreter before the script path (multiple(s) allowed)

  [!NOTE]
  The bash:/// script support is currently experimental and might change in upcoming releases, the behavior changes will be
  clearly documented here.

v2.3.4

• Fix JSON decoding in the client tools (firehose-client, print merged-blocks, etc.).

v2.3.3

Known issues

• The block decoding to JSON is broken in the CLI tools (firehose-client, print merged-blocks, etc.). Use version v2.3.1 for those tools

Hotfix

• Fix block poller panic on v2.3.2

v2.3.2

Auth and metering

• Add missing metering events for sf.firehose.v2.Fetch/Block responses.
• Changed default polling interval in 'continuous authentication' from 10s to 60s, added 'interval' query param to URL.

Substreams

• Fixed bug in scheduler ramp-up function sometimes waiting before raising the number of workers
• Fixed load-balancing from tier1 to tier2 when using dns:/// (round-robin policy was not set correctly)
• Added trace_id in grpc authentication calls
• Bumped connect-go library to new "connectrpc.com/connect" location

v2.3.1

Operators

• Firehose blocks that were produced using the RPC Poller will have to be extracted again to fix the Transaction Status and the potential missing receipt (ex: arb-one pre-nitro, Avalanche, Optimism ...)

Fixes

• Fix race condition in RPC Poller which would cause some missing transaction receipts
• Fix conversion of transaction status from RPC Poller: failed transactions would show up as "status unknown" in firehose blocks.

Added

• Added the support the FORCE_FINALITY_AFTER_BLOCKS environment variable: setting it to a value like '200' will make the 'reader' mark blocks as final after a maximum of 200 block confirmations, even if the chain implements finality via a beacon that lags behind.

v2.3.0

• Reduce logging and logging "payload".

• Tools printing Firehose Block model to JSON now have --proto-paths take higher precedence over well-known types and even the chain itself, the order is --proto-paths > chain > well-known (so well-known is lookup last).

• The tools print one-block now works correctly on blocks generated by omni-chain firecore binary.

• The various health endpoint now sets Content-Type: application/json header prior sending back their response to the client.

• The firehose, substreams-tier1 and substream-tier2 health endpoint now respects the common-system-shutdown-signal-delay configuration value meaning that the health endpoint will return false now if SIGINT has been received but we are still in the shutdown unready period defined by the config value. If you use some sort of load balancer, you should make sure they are configured to use the health endpoint and you should common-system-shutdown-signal-delay to something like 15s.

• Changed reader logger back to reader-node to fit with the app's name which is reader-node.

• Fix tools compare-blocks that would fail on new format.

• Fix substreams to correctly delete .partial files when serving a request that is not on a boundary

v2.2.2

The Cancun hard fork happened on Goerli and after further review, we decided to change the Protobuf definition for the new BlockHeader, Transaction and TransactionReceipt fields that are related to blob transaction.

We made explicit that those fields are optional in the Protobuf definition which will render them in your language of choice using the appropriate "null" mechanism. For example on Golang, those fields are generated as BlobGasUsed *uint64 and ExcessBlobGas *uint64 which will make it clear that those fields are not populated at all.

The affected fields are:

• BlockHeader.blob_gas_used, now optional uint64.
• BlockHeader.excess_blob_gas, now optional uint64.
• TransactionTrace.blob_gas, now optional uint64.
• TransactionTrace.blob_gas_fee_cap, now optional BigInt.
• TransactionReceipt.blob_gas_used, now optional uint64.
• TransactionReceipt.blob_gas_price, now optional BigInt.

This is technically a breaking change for those that could have consumed those fields already but we think he impact is so minimal that it's better to make the change right now.

Operators

You will need to reprocess a small Goerli range. You should update to new version to produce the newer version and the reprocess from block 10377700 up to when you upgraded to v2.2.2.

The block 10377700 was chosen since it is the block at the time of the first release we did supporting Cancun where we introduced those new field. If you know when you deploy either v2.2.0 or v2.2.1, you should reprocess from that point.

An alternative to reprocessing is updating your blocks by having a StreamingFast API Token and using fireeth tools download-from-firehose goerli.eth.streamingfast.io:443 -a SUBSTREAMS_API_TOKEN 10377700:<recent block rounded to 100s> <destination>.

Note

You should download the blocks to a temporary destination and copy over to your production destination once you have them all.

You can reach to us on Discord if you need help on something.

v2.2.1

• Updated the documentation for some of the upcoming new Cancun hard-fork fields:
  • TransactionTrace.blob_gas
  • TransactionReceipt.blob_gas_used
  • TransactionReceipt.blob_gas_price

v2.2.0

Important

Operators running Goerli chain will need to upgrade to this version, with this geth node release: https://github.com/streamingfast/go-ethereum/releases/tag/geth-v1.13.10-fh2.4

Support for Dencun fork (Goerli: Jan 17th)

• Added support for EIP-4844 (upcoming with activation of Dencun fork), through instrumented go-ethereum nodes with version fh2.4. This adds new fields in the Ethereum Block model, fields that will be non-empty when the Ethereum network your pulling have EIP-4844 activated. The fields in questions are:
  • Block.system_calls
  • BlockHeader.blob_gas_used
  • BlockHeader.excess_blob_gas
  • BlockHeader.parent_beacon_root
  • TransactionTrace.blob_gas
  • TransactionTrace.blob_gas_fee_cap
  • TransactionTrace.blob_hashes
  • TransactionReceipt.BlobGasUsed
  • TransactionReceipt.BlobGasPrice
  • A new TransactionTrace.Type value TRX_TYPE_BLOB

Substreams server (bumped to v1.3.1)

• Fixed error-passing between tier2 and tier1 (tier1 will not retry sending requests that fail deterministicly to tier2)
• Tier1 will now schedule a single job on tier2, quickly ramping up to the requested number of workers after 4 seconds of delay, to catch early exceptions
• "store became too big" is now considered a deterministic error and returns code "InvalidArgument"

Misc

• Added tools poller generic-evm subcommand. It is identical to optimism/arb-one in feature at the moment and should work for most evm chains.