Skip to content

feat: add Tempo as an additional service to collect Lighthouse tracing data#1150

Merged
barnabasbusa merged 5 commits intoethpandaops:mainfrom
jimmygchen:add-tempo-service
Aug 19, 2025
Merged

feat: add Tempo as an additional service to collect Lighthouse tracing data#1150
barnabasbusa merged 5 commits intoethpandaops:mainfrom
jimmygchen:add-tempo-service

Conversation

@jimmygchen
Copy link
Contributor

@jimmygchen jimmygchen commented Aug 12, 2025 •
edited
Loading

This PR adds tempo as an additional service, and adds the collector url to lighthouse startup parameters, so that lighthouse nodes can push traces to Tempo tracing backend when it's enabled.

When tempo is enabled, Lighthouse traces can be observed in the Grafana Explore tab, and select Search query type:

image

I have a separate branch identical to this one but with the kurtosis.yml file modified to my fork, so it can be run with:

kurtosis run --enclave local-testnet github.com/jimmygchen/ethereum-package@tempo-test --args-file ./network_params.yaml

Note: this PR is coded by Claude and it's working - I'm using it but I likely won't have capacity in the short term to polish it further, feel free to leave the PR open or take over.

Another current limitation is that the service name is hard coded to lighthouse so we can't distinguish between different lighthouse nodes. We could add a flag to lighthouse to configure the tracing service name.

@jimmygchen jimmygchen changed the title Add Tempo as an additional service to collect Lighthouse tracing data feat : add Tempo as an additional service to collect Lighthouse tracing data Aug 12, 2025
@jimmygchen jimmygchen changed the title feat : add Tempo as an additional service to collect Lighthouse tracing data feat: add Tempo as an additional service to collect Lighthouse tracing data Aug 12, 2025
@barnabasbusa
Copy link
Collaborator

barnabasbusa commented Aug 13, 2025

Can you run kurtosis lint --format and see if you can solve the mix-with-tools.yaml issue?

@jimmygchen
Copy link
Contributor Author

jimmygchen commented Aug 14, 2025

@barnabasbusa done, thanks!
A little note here - the tempo service is started before participant network is started because the IP needs to be available before launching lighthouse. I'm not sure if there's other better ways to do this.

@barnabasbusa
Copy link
Collaborator

barnabasbusa commented Aug 14, 2025

you can use the service name instead of ip, within the same docker network dns will work.

@jimmygchen
Copy link
Contributor Author

jimmygchen commented Aug 19, 2025 •
edited
Loading

I've changed it to use service name instead.

If this feature ends up being accepted, the next thing that would be useful is to configure metric generation for traces - that would give us metrics for timing / count of spans. I didn't attempt to do it in this PR because it would require changes to the prometheus package to enable remote write API, e.g.:

https://github.com/jimmygchen/lighthouse-metrics/blob/9f2af74c7f8daad8cacc8144216d8acb411b3ad4/docker-compose.yaml#L19

Here's a working example of the metrics generator tempo config, that generates metrics and push to prometheus:
https://github.com/jimmygchen/lighthouse-metrics/blob/9f2af74c7f8daad8cacc8144216d8acb411b3ad4/tempo.yaml#L19-L40

metrics_generator.processor.span_metrics.dimensions would be a useful param to add to override params.

@jimmygchen
Copy link
Contributor Author

jimmygchen commented Aug 19, 2025

CI should now be passing!

@barnabasbusa barnabasbusa merged commit ba328bb into ethpandaops:main Aug 19, 2025
14 checks passed
@EthPandaOpsBot EthPandaOpsBot mentioned this pull request Aug 19, 2025
Merged
barnabasbusa pushed a commit that referenced this pull request Jan 5, 2026
@EthPandaOpsBot
chore(main): release 6.0.0 (#1013)
b992b41 
🤖 I have created a release *beep* *boop*
---


##
[6.0.0](5.0.1...6.0.0)
(2026-01-05)


### ⚠ BREAKING CHANGES

* geth genesis flag, osaka time passthrough for public networks
([#1229](#1229))
* remove mev_flood
([#1091](#1091))

### Features

* Add 'trace' to enabled JSON-RPC modules
([#1235](#1235))
([ea241af](ea241af))
* add `depends_on` to store service files for parallel
([#1208](#1208))
([98206c8](98206c8))
* add `env` parameter to Blockscout configuration for custom environment
variables
([#1262](#1262))
([627619c](627619c))
* add api to dora
([#1120](#1120))
([9dbde5a](9dbde5a))
* add bpo
([#1016](#1016))
([37082b2](37082b2))
* add chainspec support for fusaka-devnet-2
([#1055](#1055))
([0e18733](0e18733))
* add checkpointz
([#1254](#1254))
([c9d72f9](c9d72f9))
* Add cl_devices parameter for mounting host devices to CL containers
([#1251](#1251))
([ab9e55f](ab9e55f))
* add client-language label to ethereum service containers
([#1074](#1074))
([6955763](6955763))
* add custody group and getBlobsV2 metrics on PeerDAS dashboard
([#982](#982))
([e43e569](e43e569))
* add debug port for ethjs
([#1044](#1044))
([459a931](459a931))
* add disk usage as a metric for ethereum-metrics-exporter
([#1026](#1026))
([8e793a5](8e793a5))
* add el genesis files to dora config for extended blob gas display
([#1180](#1180))
([2fd2b41](2fd2b41))
* add engine snooper urls to dora config for block execution time
tracking
([#1083](#1083))
([7ffb9d5](7ffb9d5))
* Add eRPC integration
([#1223](#1223))
([16b72b7](16b72b7))
* add ethrex execution client
([#1131](#1131))
([82e5a71](82e5a71))
* add extra labels
([#1030](#1030))
([12447f8](12447f8))
* add genesis_time as a possible network_param config value
([#1216](#1216))
([b11ce6f](b11ce6f))
* Add Geth dashboard to the grafana module
([#1166](#1166))
([47c90f6](47c90f6))
* Add Kurtosis config for BALs devnet 0
([#1197](#1197))
([9a93b33](9a93b33))
* add log level to ethrex
([#1269](#1269))
([57af564](57af564))
* add maxblobspertx
([#1063](#1063))
([1944080](1944080))
* add mempool-bridge service integration
([#1222](#1222))
([92d2239](92d2239))
* add mev-fulu test
([#1014](#1014))
([67a76e9](67a76e9))
* add MIN_EPOCHS_FOR_DATA_COLUMN_SIDECARS_REQUESTS as config option
([#1057](#1057))
([97afe9c](97afe9c))
* add more groups to spamoor clients
([#1066](#1066))
([c9ed485](c9ed485))
* add multiple bn nodes per vc
([#1189](#1189))
([7727330](7727330))
* add name and nameoverride func to clients page spamoor
([#1068](#1068))
([e3abf47](e3abf47))
* add new timing parameters
([#1168](#1168))
([1d524c8](1d524c8))
* add nginx file server implementation
([#1065](#1065))
([fd76bba](fd76bba))
* add node index label
([#1086](#1086))
([5aa0d44](5aa0d44))
* add node selectors and tolerations to run_sh
([#1167](#1167))
([c4e0c89](c4e0c89))
* add OTLP collector URL to Lighthouse validator client
([#1252](#1252))
([f507360](f507360))
* add peercount support for nimbusel
([#1092](#1092))
([12409e4](12409e4))
* add per participant checkpoint sync enabled flag
([#1243](#1243))
([2101448](2101448))
* add public ports for mev
([#1023](#1023))
([5d89274](5d89274))
* add public ports for other tools
([#1025](#1025))
([fa9d05e](fa9d05e))
* add sanity check for lack of supernodes
([#1145](#1145))
([70dd011](70dd011))
* add sanity check for perfect peerdas
([#1217](#1217))
([6c51752](6c51752))
* add skip_start
([#1253](#1253))
([338bb88](338bb88))
* Add support for dummy EL in kurtosis config
([#1276](#1276))
([b8007fd](b8007fd))
* add support for extra mounts for CL, EL, and VC clients
([#1136](#1136))
([d385265](d385265))
* add support for MIN_EPOCHS_FOR_BLOCK_REQUESTS
([#1211](#1211))
([17ad84a](17ad84a))
* add support for separate bootnode with bootnodoor
([#1238](#1238))
([f8f4de6](f8f4de6))
* Add support for the helix relay
([#1237](#1237))
([e17cb60](e17cb60))
* add telemetry service name flag to Lighthouse
([#1160](#1160))
([2f61b9c](2f61b9c))
* add Tempo as an additional service to collect Lighthouse tracing data
([#1150](#1150))
([ba328bb](ba328bb))
* add tolerations
([#1137](#1137))
([a4b52da](a4b52da))
* add tx snooper
([#1043](#1043))
([34e1151](34e1151))
* add validator balance
([#1032](#1032))
([3601346](3601346))
* add validator ranges for devnets
([#1176](#1176))
([3fb5084](3fb5084))
* add validator summary dora
([#1177](#1177))
([f289914](f289914))
* **ai:** Add docs
([#1061](#1061))
([1bf0893](1bf0893))
* allow passing custom env vars to the genesis generator
([#1227](#1227))
([a43368e](a43368e))
* allow specifying additional mnemonics
([#1267](#1267))
([dad4ea3](dad4ea3))
* automatically generate a 2/3 ratio for target/max blobs
([#1156](#1156))
([2d1aa15](2d1aa15))
* bump egg (fulu genesis support)
([#1140](#1140))
([601df3b](601df3b))
* configure Blockscout to index from shadowfork block height
([#1221](#1221))
([a1347fe](a1347fe))
* default to ethpandaops/client:devnet images
([#1097](#1097))
([fa4f99a](fa4f99a))
* enable `custom_preset` in checkpointz config
([#1259](#1259))
([8e9913b](8e9913b))
* enable extra env vars to be set during runtime mev, enable pprof by
default
([#1012](#1012))
([94a7f22](94a7f22))
* enable mass das guardian scans in dora
([#1125](#1125))
([0671925](0671925))
* enable prom and grafana to be ran separatly
([#1028](#1028))
([500c3f0](500c3f0))
* enable rpc proxy in dora
([#1212](#1212))
([4de44ce](4de44ce))
* enable tty for prysm
([#1076](#1076))
([1ae1826](1ae1826))
* enable validator block on sentry
([#1224](#1224))
([0f61746](0f61746))
* Extra Files for `*_extra_mounts` support
([#1144](#1144))
([1b889f6](1b889f6))
* feature flag for DisableFinalizedRootCheck
([#1228](#1228))
([c51f183](c51f183))
* fine grained control with public ip addresses per service
([#1111](#1111))
([3f60fa8](3f60fa8))
* make default node a supernode
([#1230](#1230))
([802c045](802c045))
* remove maxBlobsPerTx
([#1113](#1113))
([9f40d0a](9f40d0a))
* remove mev_flood
([#1091](#1091))
([2d3b170](2d3b170))
* rename eip7732 to gloas
([#1157](#1157))
([f0c5522](f0c5522))
* set fulu fork epoch at genesis
([#1261](#1261))
([6ae2474](6ae2474))
* sps setting Qu0b/nethermind sps
([#1225](#1225))
([969a707](969a707))
* Support el_storage_type flag
([#1257](#1257))
([2eb1e85](2eb1e85))
* use dns names instead of ip addresses for services
([#1194](#1194))
([f360a51](f360a51))


### Bug Fixes

* add fulu fork version for mev-boost-relay
([#1088](#1088))
([953ec57](953ec57))
* add input option for blobber
([#1072](#1072))
([293286d](293286d))
* assertoor image for fulu support
([#1240](#1240))
([b0f4fab](b0f4fab))
* besu sync snap if non kurtosis
([#1034](#1034))
([6752218](6752218))
* **blockscout:** make frontend available in kubernetes
([#1033](#1033))
([d3ae571](d3ae571))
* bump egg,fix minimal preset
([#1165](#1165))
([0f877c6](0f877c6))
* bump ethereum-genesis-generator to fix issues with large additional
contracts
([#1019](#1019))
([cb644af](cb644af))
* change default images
([#1099](#1099))
([ba92830](ba92830))
* change lh supernode flag
([#1186](#1186))
([f64ff38](f64ff38))
* change nimbus supernode flag
([#1275](#1275))
([094b3f3](094b3f3))
* cl node discovery on k8s
([#1162](#1162))
([5643dfd](5643dfd))
* cleanup besu
([#1139](#1139))
([5001427](5001427))
* commit boost cb-config
([#1233](#1233))
([87f3e03](87f3e03))
* commit boost integration
([#1204](#1204))
([69e60b3](69e60b3))
* default to empty blob schedule if non defined
([#1115](#1115))
([35c298d](35c298d))
* default to pandaops ethrex image for arm/amd
([#1249](#1249))
([b03a571](b03a571))
* disable page cache in dora
([#1079](#1079))
([1e51446](1e51446))
* dora,assertoor pull through cache
([#1059](#1059))
([69c965f](69c965f))
* downgrade teku from latest to master
([#1155](#1155))
([996c2a1](996c2a1))
* el/cl/vc index calculation bug, due to parallel execution
([#1121](#1121))
([fc4e65e](fc4e65e))
* enable submission pages in dora
([#1031](#1031))
([33e3f7b](33e3f7b))
* ensure proper bpo scheduling
([#1266](#1266))
([57120bf](57120bf))
* erigon db size alloc
([#1096](#1096))
([777d37e](777d37e))
* failed to start network: ethereum-package execution error: Evaluation
error: key osaka_time not in dict
([#1218](#1218))
([ae74385](ae74385))
* fix checkpointz params override
([#1258](#1258))
([9518b72](9518b72))
* geth genesis flag, osaka time passthrough for public networks
([#1229](#1229))
([d58cab7](d58cab7))
* geth peering bug
([#1133](#1133))
([bc62c0c](bc62c0c))
* gloas minimal config values
([#1250](#1250))
([2ec3a94](2ec3a94))
* helix logging type
([#1279](#1279))
([756bfdd](756bfdd))
* **lighthouse:** allow genesis sync when checkpoint sync isn't enabled
([#1192](#1192))
([4053331](4053331))
* make sure builder cl is supernode
([#1188](#1188))
([dfef921](dfef921))
* mev rbuilder remove unused config param
([#1248](#1248))
([3838a5f](3838a5f))
* minimal builds should use latest unstable branches
([#1174](#1174))
([d6d6d5f](d6d6d5f))
* minimal spec
([#1037](#1037))
([2372550](2372550))
* missing dns_name
([#1274](#1274))
([3238be5](3238be5))
* move bootnodoor to additional_services
([#1264](#1264))
([bf40917](bf40917))
* nethermind chainspec, default genesis gas to 60M
([#1039](#1039))
([b839e61](b839e61))
* network params default images
([#1213](#1213))
([33a0db2](33a0db2))
* nimbus checkpoint syncing
([#1181](#1181))
([d464295](d464295))
* only add --target-peers=0 only when the network is kurtosis
([#1119](#1119))
([572cbfc](572cbfc))
* only append blob schedule, if defined
([#1022](#1022))
([43db03a](43db03a))
* override bpo1,2
([#1196](#1196))
([836cbb8](836cbb8))
* pass gas limit to ethrex if network gas_limit was specified
([#1232](#1232))
([39ac09b](39ac09b))
* prysm gzip encoding bug
([#1112](#1112))
([9f5fc45](9f5fc45))
* pull kurtosis images in kurtosis install
([#1048](#1048))
([a00b6dd](a00b6dd))
* rbuilder parallel safe sorting
([#1046](#1046))
([ec5895d](ec5895d))
* readme for additional services
([#1270](#1270))
([ba855e0](ba855e0))
* readme/CI jobs
([#1263](#1263))
([ca6b7d2](ca6b7d2))
* remove default basefee fraction
([#1143](#1143))
([d29e0bf](d29e0bf))
* remove graffiti
([#1082](#1082))
([ee4fff4](ee4fff4))
* remove unused env
([#1153](#1153))
([ea73a95](ea73a95))
* revert prometheus branch
([#1024](#1024))
([1559386](1559386))
* sanity check for all subfields
([#1130](#1130))
([3d2c71c](3d2c71c))
* service ports
([#1021](#1021))
([e83a1ad](e83a1ad))
* set deploy_client_group for mev related uniswap spammer
([#1195](#1195))
([09d09b0](09d09b0))
* set miner gasprice for geth if running kt'
([#1027](#1027))
([161fc14](161fc14))
* sf for erigon/geth post fulu
([#1183](#1183))
([e964e30](e964e30))
* shadowfork enclave edits
([#1070](#1070))
([63689ec](63689ec))
* shadowfork latest bug
([#1045](#1045))
([197cdf8](197cdf8))
* shadowfork upstream to eth-clients
([#1047](#1047))
([7c11a34](7c11a34))
* single-node lighthouse startup issue
([#1073](#1073))
([6d29b3a](6d29b3a))
* some tests
([#1190](#1190))
([fca81b3](fca81b3))
* specify devnet size for persistent flag
([#1054](#1054))
([b4c398c](b4c398c))
* update custom-network to --network
([#1159](#1159))
([1f57a7b](1f57a7b))
* Update mainnet yaml
([#1069](#1069))
([288919b](288919b))
* Update sf to osaka
([#1105](#1105))
([59579bb](59579bb))
* update some tests
([#1122](#1122))
([9488046](9488046))
* use default dora image for fulu networks
([#1128](#1128))
([b1f4e5c](b1f4e5c))
* use reth-rbuilder image as the default mev_builder_image in
network_params.yaml
([#1077](#1077))
([f07f3b6](f07f3b6))
* use self hosted runners
([#1100](#1100))
([2fc4a3c](2fc4a3c))
* use ubuntu-latest
([#1078](#1078))
([d209af4](d209af4))
* validator client compatibility update
([#1114](#1114))
([b826cc9](b826cc9))
* yeet unused mev builder
([#1056](#1056))
([40767fe](40767fe))
* yeet-7907
([#1116](#1116))
([93c6630](93c6630))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@barnabasbusa barnabasbusa barnabasbusa approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jimmygchen @barnabasbusa