BlsMultiThreadWorkerPool error - Value is not a valid number: Infinity

[nflaig]

Caught the following error during Kurtosis testing

Nov-28 14:24:39.365[chain]           error: BlsMultiThreadWorkerPool error - Value is not a valid number: Infinity
TypeError: Value is not a valid number: Infinity
    at /usr/app/node_modules/prom-client/lib/histogram.js:237:10
    at Histogram.observeWithoutExemplar (/usr/app/node_modules/prom-client/lib/histogram.js:74:54)
    at Timeout.BlsMultiThreadWorkerPool.runJob [as _onTimeout] (file:///usr/app/packages/beacon-node/src/chain/bls/multithread/index.ts:490:49)

I don't think it's specific to the configuration I used but might be isolated to small network only as it's liekly due to dividing by 0 here

lodestar/packages/beacon-node/src/chain/bls/multithread/index.ts

Line 490 in 48dea55

this.metrics?.blsThreadPool.timePerSigSet.observe(workerJobTimeSec / startedSigSets);

[nflaig]

This causes the batch validation to fail on kurtosis but does not happen all the time

Nov-28 17:29:02.353[chain]           error: BlsMultiThreadWorkerPool error - Value is not a valid number: Infinity
TypeError: Value is not a valid number: Infinity
    at /usr/app/node_modules/prom-client/lib/histogram.js:237:10
    at Histogram.observeWithoutExemplar (/usr/app/node_modules/prom-client/lib/histogram.js:74:54)
    at Timeout.BlsMultiThreadWorkerPool.runJob [as _onTimeout] (file:///usr/app/packages/beacon-node/src/chain/bls/multithread/index.ts:490:49)
Nov-28 17:29:02.354[network]         debug: Gossip batch validation beacon_attestation threw an error - Value is not a valid number: Infinity
TypeError: Value is not a valid number: Infinity
    at /usr/app/node_modules/prom-client/lib/histogram.js:237:10
    at Histogram.observeWithoutExemplar (/usr/app/node_modules/prom-client/lib/histogram.js:74:54)
    at Timeout.BlsMultiThreadWorkerPool.runJob [as _onTimeout] (file:///usr/app/packages/beacon-node/src/chain/bls/multithread/index.ts:490:49)

I used the following config

participants:
  - el_type: geth
    el_image: ethpandaops/geth:lightclient-prague-devnet-4-37035c5
    cl_type: lodestar
    cl_image: nflaig/lodestar:single-attestation7
    vc_type: lodestar
    vc_image: nflaig/lodestar:single-attestation7
    count: 4
network_params:
  electra_fork_epoch: 1
  genesis_delay: 60
  num_validator_keys_per_node: 64
  seconds_per_slot: 12
additional_services:
  - dora
  - apache
  - assertoor
dora_params:
  image: "ethpandaops/dora:master"
snooper_enabled: true

[nflaig]

Looks like this started happening on devnet-5 branch and is related to single attestation changes and it only happens post-electra

[nflaig]

Resolved by #7266