Add Real-World Util Blob Example

[holgerd77]

Hi @paulmillr maybe you can help with this one too, respectively I wonder if there is a blob-read bug still in micro-eth-signer.

So I am trying to do a real-world blob round trip, by downloading this blob from Blobscan (under the "Google" button) and then storing this as ascii hex with:

hexdump -ve '1/1 "%.2x"' 016992e52290e394778729dd607b14b3fc43d4e4d8241260218708cb8e250e0c.bin > blob.txt

File with vi looks solid and correct:

I have then used the following code from this PR to read in this blob, respectively after first trying to read the full one I switched to read chunks of it (see the substring(0, 100) in the example code).

Doing this with 0, 32 works:

But going "full blob" or even going up to 0, 64 breaks the thing with:

/node_modules/micro-eth-signer/esm/kzg.js:39
        throw new Error('parseScalar: invalid field element');
              ^

Error: parseScalar: invalid field element

Is this really correct to go through parseScalar here? Shouldn't a blob just be arbitraty data?

Note that I am testing on micro-eth-signer 15.0.0, the related logic doesn't seem to have changed though, checked in the code.

[paulmillr]

That blob works at verify, seems like the example is invalid.

Copy-past from website to file:

This is with '0x' (262146 bytes). Code works with both.

# sha256 blob.txt
SHA256 (blob.txt) = b5cf2485712c37a318dc771f4ec8e20ea75a4e3d7607b3f23e2c7e2c3bb2e08d

This hash of version without '0x' (262144 bytes):

SHA256 (blob.txt) = e320325cdac9a57057e3e8902c6b71d0131afa2ac4ae140f1c238d364cafe5db

Old version:

        "node_modules/micro-eth-signer": {
            "version": "0.15.0",
            "resolved": "https://registry.npmjs.org/micro-eth-signer/-/micro-eth-signer-0.15.0.tgz",
            "integrity": "sha512-VkKK698Odm0ef40ERC9FdcG6BHBL9vWhy+7xkLL1M0O3bcRBLi7FP7mr+4SdiPaP823Q+r6c8JdUtWdESSf06A==",
            "license": "MIT",
            "dependencies": {
                "@noble/curves": "~1.9.1",
                "@noble/hashes": "~1.8.0",
                "micro-packed": "~0.7.3"
            }
        },

//         "micro-eth-signer": "^0.15.0"
import * as fs from 'node:fs';
import { KZG } from 'micro-eth-signer/kzg.js';
import { trustedSetup as s_fast } from '@paulmillr/trusted-setups/fast-kzg.js';
import { deepStrictEqual } from 'node:assert';
const kzg = new KZG(s_fast);
const blob = fs.readFileSync('./blob.txt', 'utf8');

/*
OK [
  9950896364063828479760105497824655790015207806091689101821226327781578341321n,
  5073668346629795774871283667712309276490414982699714158417373496810743118145n,
  25176228008882631450863129405107679141280536219895767651782763491732366126953n,
  2611873799884172396458779361450711640617140137603788323278223043506227032276n,
  27008833669916875399255315324821569880893552449183082959694602115932697760336n,
  22338810404566275086580161916721569668795387632298891059325211760519627059809n,
  23774835051371149139915222215553252928398022756962564187347204236001807219301n,
  16508100157248019383180095226
*/
console.log('OK', kzg.parseBlob(blob));
// passes!
deepStrictEqual(
  kzg.verifyBlobProof(
    blob,
    '0x873889d4a2ebc9b85b6e644f00475e74fe9123c930dd57002621952e4d8f2fff7329b78eb51d08746280939f5c45f611',
    '0x8397af82a8aca743b2bd3041ec76b467783de03903ee704403de4a41a28ce6c1e2fb66b195c0a0151b6285ad523e6fc8'
  ),
  true
);

New version

//         "micro-eth-signer": "^0.17.3"
import * as fs from 'node:fs';
import { KZG } from 'micro-eth-signer/advanced/kzg.js';
import { trustedSetup as s_fast } from '@paulmillr/trusted-setups/fast-kzg.js';
import { deepStrictEqual } from 'node:assert';

const kzg = new KZG(s_fast);
const blob = fs.readFileSync('./blob.txt', 'utf8');
console.log('OK', kzg.parseBlob(blob));
deepStrictEqual(
  kzg.verifyBlobProof(
    blob,
    '0x873889d4a2ebc9b85b6e644f00475e74fe9123c930dd57002621952e4d8f2fff7329b78eb51d08746280939f5c45f611',
    '0x8397af82a8aca743b2bd3041ec76b467783de03903ee704403de4a41a28ce6c1e2fb66b195c0a0151b6285ad523e6fc8'
  ),
  true
);

Example

Seems like an issue with code in the example:

export function getBlob(data: Uint8Array): PrefixedHexString {
  const blob = new Uint8Array(BLOB_SIZE);
  for (let i = 0; i < FIELD_ELEMENTS_PER_BLOB; i++) {
    const chunk = new Uint8Array(32);
    chunk.set(data.subarray(i * 31, (i + 1) * 31), 0);
    blob.set(chunk, i * 32);
  }

  return bytesToHex(blob);
}

This is incorrect: chunk.set(data.subarray(i * 31, (i + 1) * 31), 0), stride as 31 bytes, but reads 32.

Working example code:

import * as fs from 'fs';
import {
  type PrefixedHexString,
  blobsToCellProofs,
  blobsToProofs,
  computeVersionedHash,
  hexToBytes,
  bytesToHex,
} from '@ethereumjs/util';
import { trustedSetup } from '@paulmillr/trusted-setups/fast-peerdas.js';
import { KZG as microEthKZG } from 'micro-eth-signer/kzg.js';
const BYTES_PER_FIELD_ELEMENT = 32; // EIP-4844
const FIELD_ELEMENTS_PER_BLOB = 4096; // EIP-4844
const BLOB_SIZE = BYTES_PER_FIELD_ELEMENT * FIELD_ELEMENTS_PER_BLOB;

const MAX_BLOBS_PER_TX = 6; // EIP-7691: Blob throughput increase, Pectra HF
const MAX_BLOB_BYTES_PER_TX = BLOB_SIZE * MAX_BLOBS_PER_TX - 1;
export function getBlob(data: Uint8Array): PrefixedHexString {
  const blob = new Uint8Array(BLOB_SIZE);
  for (let i = 0; i < FIELD_ELEMENTS_PER_BLOB; i++) {
    const chunk = new Uint8Array(32);
    chunk.set(data.subarray(i * 32, (i + 1) * 32), 0);
    blob.set(chunk, i * 32);
  }

  return bytesToHex(blob);
}

const kzg = new microEthKZG(trustedSetup);

// Use with node ./examples/blobs.ts <file path>
const filePath = process.argv[2];
let blobData: string = fs.readFileSync(filePath, 'ascii');
console.log(blobData);
console.log(blobData.length);
//blobData = blobData.substring(0, 100);

const blobs = [getBlob(hexToBytes(`0x${blobData}`))];

//const blobData = 'hello'
//const blobs = getBlobs(blobData)

console.log('Created the following blobs:');
//console.log(blobs)

const commitment = kzg.blobToKzgCommitment(blobs[0]);

const blobCommitmentVersion = 0x01;
const versionedHash = computeVersionedHash(commitment as PrefixedHexString, blobCommitmentVersion);

// EIP-4844 only
const blobProof = blobsToProofs(kzg, blobs, [commitment as PrefixedHexString]);
const cellProofs = blobsToCellProofs(kzg, blobs);

console.log(`Commitment                  : ${commitment}`);
console.log(`Versioned hash              : ${versionedHash}`);
console.log(`Blob proof (EIP-4844)       : ${blobProof}`);
console.log(`First cell proof (EIP-7594) : ${cellProofs[0]}`);
console.log(`Num cell proofs (EIP-7594)  : ${cellProofs.length}`);

That example outputs:

Created the following blobs:
Commitment                  : 0x873889d4a2ebc9b85b6e644f00475e74fe9123c930dd57002621952e4d8f2fff7329b78eb51d08746280939f5c45f611
Versioned hash              : 0x016992e52290e394778729dd607b14b3fc43d4e4d8241260218708cb8e250e0c
Blob proof (EIP-4844)       : 0x8397af82a8aca743b2bd3041ec76b467783de03903ee704403de4a41a28ce6c1e2fb66b195c0a0151b6285ad523e6fc8
First cell proof (EIP-7594) : 0x8cd15d019d9d81f68b38f00980c2a24d96f3cc636147c2e912fbc5ad657f4132106f31ffcc67beb66aa4979936c24625
Num cell proofs (EIP-7594)  : 128

Looks same as that website (versioned hash/proof/commitment same).

[holgerd77]

@paulmillr

Wow! 🤯

That works! I would have never expected that we (you!) find a bug in the production code with this real-world setup!

But: that's great! Thanks so much! Need to clean this a bit up still, today too tired.

Then we can also take this into the next releases.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.56%. Comparing base (7eead0b) to head (c85b4b9).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

Flag	Coverage Δ
block	88.44% <ø> (ø)
blockchain	88.85% <ø> (ø)
common	93.38% <ø> (ø)
evm	62.01% <ø> (ø)
mpt	90.11% <ø> (ø)
statemanager	78.10% <ø> (ø)
static	91.35% <ø> (ø)
tx	88.01% <ø> (ø)
util	84.94% <100.00%> (ø)
vm	65.08% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[paulmillr]

• getBlob function is correct, it takes arbitrary data and converts it into 'eth blob' (which is array of scalar elements). This is not specc-ed, so it's up to application to choose a way to encode stuff (it may be encoded differently). It simply splits arbitrary data into 31 byte chunks then creates 32 byte elements from them. Since it is LE, that causes padding which makes everything valid scalar element.
• On the blockchain there are already encoded blobs (32 byte scalar elements), so usage of 'getBlob' here is just incorrect (thats already blob, not arbitrary data!).
• "Fixed" getBlob is simple no-op, it splits by 32 byte chunks and then concats them back.

[paulmillr]

To reiterate:

1. getBlob(hexToBytes(0x${blobData}))] should be changed to remove getBlob call. The it will be hexToBytes. Although meth accepts hex, which means hexToBytes is also not necessary.
2. Change inside getBlob which changes 31 to 32 should be reverted back to 31.

[ScottyPoi]

Reverted the change to getBlob and fixed the example

[github-actions]

📦 Bundle Size Analysis

Package	Size (gzip)	Δ
binarytree	30.1 KB	⚪ ±0%
block	45.4 KB	⚪ ±0%
blockchain	52.4 KB	⚪ ±0%
client	181.7 KB	⚪ ±0%
common	44.9 KB	⚪ ±0%
devp2p	76.4 KB	⚪ ±0%
e2store	30.3 KB	⚪ ±0%
ethash	8.4 KB	⚪ ±0%
evm	201.4 KB	⚪ ±0%
genesis	519.5 KB	⚪ ±0%
mpt	55.1 KB	⚪ ±0%
rlp	7.2 KB	⚪ ±0%
statemanager	57.2 KB	⚪ ±0%
testdata	60.2 KB	⚪ ±0%
tx	87.0 KB	⚪ ±0%
util	61.8 KB	🔴 +0.0 KB (+0.02%)
vm	61.3 KB	⚪ ±0%
wallet	14.0 KB	⚪ ±0%

_{Generated by bundle-size workflow}

[holgerd77]

Implicitly approved by Scotty fixing the example (thanks! 🙏), will merge.