[6] graph/db: use batch fetching to improve ForEachNode* performance by ellemouton · Pull Request #10123 · lightningnetwork/lnd

ellemouton · 2025-08-01T15:42:58Z

The graph store performance improvement saga continues....

In this PR, the performance of the SQLStore ForEachNodeCacheable and ForEachNodeCached are improved.

For both methods, previously we would iterate through all nodes and fetch one node's info (feature bits, tlv data, addresses) at a time. Now, both are updated to fetch this data in batches. (ie, for a set of nodes).

Further more, the ForEachNodeCached previously would fetch the channels for each node (one node at a time), this is updated so that we fetch channel data for a set/page of nodes at a time.

Benchmark improvements.

ForEachNodeCacheable

Here is the benchstat report. The old & new versions were run 10 times each. Note that the - indicates the speed up (ie, it can be read as "percentage speed reduction")

name                                      old time/op  new time/op  delta

ForEachNodeCacheable-native-sqlite-10     184ms ± 2%   145ms ±10%  -21.45%  (p=0.000 n=10+10)
ForEachNodeCacheable-native-postgres-10   697ms ± 8%    51ms ± 4%  -92.68%  (p=0.000 n=9+10)

ForEachNodeCached

For this one, the old version was so slow that I decided just to run it once for sqlite since the performance gains are so huge:
Basically, it used to take 276s to iterate through all nodes&channels for a mainnet node, and now it takes 10s.

name                                  old time/op  new time/op  delta
ForEachNodeCached-native-sqlite-10    276s ± 0%     10s ± 0%   ~     (p=1.000 n=1+1)

Part of #9795
Depends on #10121

gemini-code-assist

Summary of Changes

Hello @ellemouton, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

I've continued the performance improvement efforts for the graph store by implementing batch fetching for node and channel data within the SQLStore. This significantly boosts the performance of ForEachNodeCacheable and ForEachNodeCached methods, moving from single-item fetches to efficient batched queries, especially benefiting large graph traversals.

Highlights

Dramatic Performance Gains: ForEachNodeCacheable now runs 21.45% faster on SQLite and 92.68% faster on PostgreSQL. ForEachNodeCached saw an even more substantial improvement, reducing execution time from 276 seconds to just 10 seconds on a mainnet SQLite node.
Batch Fetching Implementation: Introduced a new ListChannelsForNodeIDs SQL query and integrated batch loading logic for node features and channel policies across several ForEach methods, moving away from N+1 query patterns.
Refactored Data Structures: Updated sqlGraphNodeTx to carry the SQLStoreConfig for better context, enabling more flexible and efficient data retrieval.
Enhanced Benchmarking: Added dedicated benchmark tests for the improved ForEachNodeCacheable and ForEachNodeCached methods to validate and track performance.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces significant performance improvements to ForEachNodeCacheable and ForEachNodeCached by refactoring them to use batch database queries instead of fetching data for each node individually. This is achieved through paginated queries and batch-loading related data like node features and channel information. The changes are well-structured, and the new logic appears correct. I have one minor suggestion regarding adherence to the coding style guide.

graph/db/sql_store.go

bhandras

LGTM, moar gainz 💪

bhandras · 2025-08-04T06:46:09Z

graph/db/sql_store.go

+		fv := lnwire.EmptyFeatureVector()
+		if features, exists := featureBits[dbNode.ID]; exists {
+			for _, bit := range features {
+				fv.Set(lnwire.FeatureBit(bit))


Not in the scope of this PR just taking note: right now there's no validation if the feature bit is valid (ie within uint16).

yeah - however, i do think this is something we'd want to check at write time and not read time just so that any existing bad feature bits dont stop us from iterating through our graph

guggero

LGTM 🎉

graph/db/benchmark_test.go

So that we can properly measure upcoming performance improvements.

In this commit, we update the SQLStore's ForEachNodeCacheable and ForEachNodeCached methods to use batch collection for node feature bits. This results in the following performance gains: ``` name old time/op new time/op delta ForEachNodeCacheable-native-sqlite-10 184ms ± 2% 145ms ±10% -21.45% (p=0.000 n=10+10) ForEachNodeCacheable-native-postgres-10 697ms ± 8% 51ms ± 4% -92.68% (p=0.000 n=9+10) ```

Add a ListChannelsForNodeIDs query which will let us fetch all channels belonging to a set of nodes.

Previously, ForEachNodeCached would batch fetch node _feature_ data but would still fetch the channel set of each node in a node-by-node fashion which is not ideal. So this commit updates this method to make use of the new sqldb.ExecuteCollectAndBatchWithSharedDataQuery helper. It lets us batch load channel data for a range of node IDs. This _greatly_ improves the performance of the method.

Update the forEachNodeChannel helper to batch fetch channel data.

ellemouton self-assigned this Aug 1, 2025

ellemouton added the no-changelog label Aug 1, 2025

gemini-code-assist bot reviewed Aug 1, 2025

View reviewed changes

graph/db/sql_store.go Show resolved Hide resolved

ellemouton force-pushed the graphPerf7 branch from f1782fd to da86865 Compare August 3, 2025 14:07

ellemouton requested review from bhandras and guggero August 3, 2025 14:11

ellemouton mentioned this pull request Aug 3, 2025

[7] graph/db+autopilot: improve efficiency of autopilot methods that use the ForEachNode/ForEachChannel pattern #10127

Merged

bhandras approved these changes Aug 4, 2025

View reviewed changes

guggero approved these changes Aug 4, 2025

View reviewed changes

graph/db/benchmark_test.go Show resolved Hide resolved

ellemouton force-pushed the elle-graphPerf6 branch from 80cef58 to ae13158 Compare August 5, 2025 04:47

ellemouton force-pushed the graphPerf7 branch from da86865 to d0e10ef Compare August 5, 2025 04:51

ellemouton added 5 commits August 5, 2025 08:00

graph/db: cover more DB methods in benchmark test

6ac19b4

So that we can properly measure upcoming performance improvements.

sqlc: add ListChannelsForNodeIDs batch query

3b60d33

Add a ListChannelsForNodeIDs query which will let us fetch all channels belonging to a set of nodes.

graph/db: batch fetch channel data in forEachNodeChannel

0850bf4

Update the forEachNodeChannel helper to batch fetch channel data.

ellemouton changed the base branch from elle-graphPerf6 to master August 5, 2025 06:00

ellemouton force-pushed the graphPerf7 branch from d0e10ef to 0850bf4 Compare August 5, 2025 06:01

ellemouton merged commit d9ff94a into lightningnetwork:master Aug 5, 2025
39 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[6] graph/db: use batch fetching to improve ForEachNode* performance#10123

[6] graph/db: use batch fetching to improve ForEachNode* performance#10123
ellemouton merged 5 commits intolightningnetwork:masterfrom
ellemouton:graphPerf7

ellemouton commented Aug 1, 2025 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

bhandras left a comment

Uh oh!

bhandras Aug 4, 2025

Uh oh!

ellemouton Aug 5, 2025

Uh oh!

guggero left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ellemouton commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark improvements.

ForEachNodeCacheable

ForEachNodeCached

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

bhandras left a comment

Choose a reason for hiding this comment

Uh oh!

bhandras Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

ellemouton Aug 5, 2025

Choose a reason for hiding this comment

Uh oh!

guggero left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ellemouton commented Aug 1, 2025 •

edited

Loading