Skip to content

feat(builder): add metrics to builder client - #9848

Merged
nflaig merged 6 commits into
ChainSafe:unstablefrom
markolazic01:feat/builder-metrics
Aug 19, 2026
Merged

nflaig merged 6 commits into
ChainSafe:unstablefrom
markolazic01:feat/builder-metrics

Conversation

@markolazic01

@markolazic01 markolazic01 commented Aug 18, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Metrics setup.

Description

Additions and notes:

  • Basic metrics setup contains wiring + gauges for builder balance and status.
  • Chose bc_ as a metric prefix, avoiding collision with lodestar_builder_ in beacon node metrics, and aligned with vc_ on validator.
  • Chose port 5065 for metrics server, analog to 5064 in validator.
  • No monitoring service until we determine if it's needed.
  • Rest api client metrics.

Verified with Kurtosis dev environment setup.

AI Assistance Disclosure

Used Claude to check the repo for existing patterns and audit changes. All changes authored manually.

@markolazic01
markolazic01 marked this pull request as ready for review August 18, 2026 22:59
@markolazic01
markolazic01 requested a review from a team as a code owner August 18, 2026 22:59

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

馃挕 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 44d7e415ad

鈩癸笍 About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 馃憤.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread packages/builder/src/metrics.ts Outdated
Comment thread packages/builder/src/metrics.ts Outdated
Comment thread packages/builder/src/metrics.ts Outdated

@nflaig nflaig left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@markolazic01

Copy link
Copy Markdown
Contributor Author

checked the failings, should be fixed with a re-run

@nflaig

nflaig commented Aug 19, 2026

Copy link
Copy Markdown
Member

@lodekeeper can you investigate why unit tests are flaky https://github.com/ChainSafe/lodestar/actions/runs/32237943058/job/96025727362?pr=9848, was there a regression we introduced recently? it seems quite common in the last few days for those to run into a timeout

@nflaig
nflaig merged commit bd3a76e into ChainSafe:unstable Aug 19, 2026
34 of 38 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Lodestar Team Coordination Aug 19, 2026
@lodekeeper

Copy link
Copy Markdown
Contributor

@nflaig I checked the linked job and nearby failures.

The linked unit failure is attempt 2 job 96025727362 on head 0851a1a3; it failed only in packages/state-transition/test/unit/rewards/blockRewards.test.ts > Normal case with the 90s test timeout. The same run's later attempt passed Unit Tests on the same head (96050420029).

The pattern is not #9848-specific: recent unrelated branches hit the same family (packages/state-transition/test/unit/sanityCheck.test.ts phase0/altair perf-state checks and blockRewards.test.ts). Those tests use the 250k-validator perf-state fixture and are already close to the edge even when green: recent successful CI jobs show blockRewards around ~79-110s/file and sanityCheck around ~174-227s/file. I don't see evidence this was introduced by #9848 or by a recent code change in that area.

The remaining red on current #9848 is the separate e2e bucket: runDevCmd.test.ts cleanup hook times out stopping lodestar dev (Hook timed out in 10000ms), with voluntaryExitRemoteSigner also showing setup/teardown timeout fallout. Older green e2e jobs pass runDevCmd around ~10-13s while the hook timeout is 10s, so that also looks like threshold/runner variance rather than the builder metrics diff.

Likely hardening: isolate or raise timeout for the perf-state unit tests (or reduce the fixture/cache cost), and make the child-process e2e cleanup use a bounded kill fallback / longer hook timeout. I haven't found a deterministic recent regression to blame yet.

@wemeetagain

Copy link
Copy Markdown
Member

馃帀 This PR is included in v1.47.0 馃帀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

4 participants