invoices: migrate KV invoices to native SQL for users of KV SQL backends by bhandras · Pull Request #8831 · lightningnetwork/lnd

bhandras · 2024-06-12T10:35:27Z

Change Description

This pull request adds the migration of old key-value (KV) invoices to the new native SQL schema when the --db.use-native-sql flag is set, unless the --db.skip-sql-invoice-migration flag is also specified.

Please note that since we currently do not support running on mixed database backends for users of bbolt or etcd, an additional step is required to migrate their KV database to SQL first. For more context, please see lightninglabs/lndinit#21.

coderabbitai · 2024-06-12T10:35:32Z

Important

Review skipped

Auto reviews are limited to specific labels.

🏷️ Labels to auto review (1)

llm-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

invoices/sql_migration.go

Abdulkbk · 2024-12-17T12:51:22Z

sqldb/no_sqlite.go

+	return s.BaseDB
+}
+
+// ApplyAllMigrations applices both the SQLC and custom in-code migrations to


spelling: applices -> applies, and 2 other instances where this function was implemented

Thank you! Fixed.

invoices/sql_store.go

Abdulkbk · 2024-12-17T14:50:08Z

invoices/sql_store.go

+	params := sqlc.InsertInvoiceParams{
+		Hash:       paymentHash[:],
+		AmountMsat: int64(invoice.Terms.Value),
+		CltvDelta: sqldb.SQLInt32(


Just noticed the comment: // Note: BOLT12 invoices don't have a final cltv delta. is no longer there

Abdulkbk · 2024-12-17T16:42:43Z

invoices/sql_migration.go

+	// which is a monotonically increasing uint32.
+	invoiceBucket = []byte("invoices")
+
+	// paymentHashIndexBucket is the name of the sub-bucket within the


The comment references paymentHashIndexBucket, while the variable is actually named invoiceIndexBucket. Is this discrepancy intentional?

ziggie1984

LGTM 🫶, massive effort in this PR, and also shout-out to Elle's great ideas here.

Had some minor comments but nothing blocking.

Would be great to open some follow-up issues for the stuff we want to move into another PR.

ziggie1984 · 2025-01-09T11:31:06Z

sqldb/sqlc/migrations/000005_migration_tracker.up.sql

+-- only run once. It tracks a global database version that encompasses both
+-- schema migrations handled by golang-migrate and custom in-code migrations
+-- for more complex data conversions that cannot be expressed in pure SQL.
+CREATE TABLE IF NOT EXISTS migration_tracker (


does this mean, this migration_tracker table is always greater or equal to the schema migration table of the sql table ? Because the in-code migration will lead to the version being greater but the version in this table cannot be lower than the version that the schema migration correct ?

Correct! There's a possibility that we could just rely in Golang Migrate in the future once this PR has landed (if ever): golang-migrate/migrate#932

ziggie1984 · 2025-01-09T14:45:34Z

sqldb/migrations.go

+// version. If a migration includes a non-nil MigrationFn, it is executed after
+// the SQL schema has been migrated to the corresponding schema version.


Does this still hold true if for every code migration will be done within a separate migration so that we do not have schema-migration and normal version in the same version ?

Yes, a migration can be of three types: "schema-only", "schema and in-code" or "in-code-only". If a migration function is defined, the schema version acts as a prerequisite for the migration. This means the schema is always updated to the required version first.

ziggie1984 · 2025-01-09T15:00:27Z

sqldb/migrations.go

+type MigrationExecutor interface {
+	// CurrentSchemaVersion returns the current schema version of the
+	// database.
+	CurrentSchemaVersion() (int, error)


haven't seen a code part where this function is used ?

Indeed this is not used anymore. Great catch! Removed.

ziggie1984 · 2025-01-09T15:01:56Z

sqldb/migrations.go

+
+	// ExecuteMigrations runs migrations for the database, depending on the
+	// target given, either all migrations or up to a given version.
+	ExecuteMigrations(target MigrationTarget) error


Nit: Maybe add here in the comment that this function handles the schema and in-code migration, and maybe recommend that migrations schema or in-code should be done separately ?

ziggie1984 · 2025-01-09T17:55:49Z

sqldb/migrations.go

+	migrationConfig = []MigrationConfig{
+		{
+			Name:          "000001_invoices",
+			Version:       1,


does the schema version always correspond to the filename here, meaning that would we start with 000002 in the firt place the database version would start at 2 ?

Yes the schema version should always correspond to the filename. This is only used to ensure clean ordering.

ziggie1984 · 2025-01-12T23:36:05Z

invoices/sql_migration.go

+
+	offset := uint64(0)
+	t0 := time.Now()
+	// Create the hash index which we will use to look up invoice


Nit NewLine above.

ziggie1984 · 2025-01-12T23:39:01Z

invoices/testdata/channel.db

would it be difficult to inject in at runtime @bhandras

ziggie1984 · 2025-01-12T23:45:34Z

config_builder.go

+		// disabled, the regular native SQL store migrations will still
+		// run. If the database version is already above this custom
+		// migration's version (6), it will be skipped permanently,
+		// regardless of the flag.


Hmm so if the user disables this, and we add a new migration in the future (version 7) he will not be able to migrate the invoice ?

The idea is that users won't disable the invoice migration unless the it fails for them. In that case they can continue using their node and report a bug to us and they should be able to migrate later on.

ziggie1984 · 2025-01-12T23:51:14Z

itest/lnd_misc_test.go

-// testNativeSQLNoMigration tests that nodes that have invoices would not start
-// up with native SQL enabled, as we don't currently support migration of KV
-// invoices to the new SQL schema.
-func testNativeSQLNoMigration(ht *lntest.HarnessTest) {


Create an issue for this because this will be done in a followu up ?

ziggie1984 · 2025-01-12T23:52:42Z

invoices/sql_store.go


 	if len(htlcs) > 0 {
 		invoice.Htlcs = htlcs
-		var amountPaid lnwire.MilliSatoshi


Open an issue to investigate this behavior ?

Discussed offline, we'll open the issue soon.

ziggie1984 · 2025-01-13T09:24:04Z

invoices/sql_store.go

 		invoice.Htlcs = htlcs
-		var amountPaid lnwire.MilliSatoshi
-		for _, htlc := range htlcs {
-			if htlc.State == HtlcStateSettled {


I think we just need to adopt it and also add it if the htlc is in the accepted state, rather then removing it?

lnd/invoices/update_invoice.go

Lines 370 to 379 in 1b0f41d

invoiceStateReady := accepted || settled

if !invoiceIsAMP {

// Update the running amount paid to this invoice. We

// don't include accepted htlcs when the invoice is

// still open.

if invoice.State != ContractOpen &&

invoiceStateReady {

amtPaid += htlc.Amt

or maybe remove it because it might be error pruning and we haven't tested every case.

tested it locally it does work with this change:

for _, htlc := range htlcs { if htlc.State == HtlcStateSettled || htlc.State == HtlcStateAccepted { amountPaid += htlc.Amt } } invoice.AmtPaid = amountPaid

Tested it with go test -v -run=TestMigrationWithChannelDB -tags=kvdb_sqlite and i think the only correct fix given your db is to use the stored amount paid value. Also discussed this offline.

lightninglabs-deploy · 2025-01-20T10:39:38Z

@Roasbeef: review reminder
@aakselrod: review reminder
@bhandras, remember to re-request review from reviewers when ready

This commit adds the migration_tracker table which we'll use to track if a custom migration has already been done.

This commit introduces support for custom, in-code migrations, allowing a specific Go function to be executed at a designated database version during sqlc migrations. If the current database version surpasses the specified version, the migration will be skipped.

This commit separates the execution of SQL and in-code migrations from their construction. This change is necessary because, currently, the SQL schema is migrated during the construction phase in the lncfg package. However, migrations are typically executed when individual stores are constructed within the configuration builder.

Previously we intentially did not set settled_at and settle_index when inserting a new invoice as those fields are set when we settle an invoice through the usual invoice update. As migration requires that we set these nullable fields, we can safely add them.

Certain invoices may not have a deterministic payment hash. For such invoices we still store the payment hashes in our KV database, but we do not have a sufficient index to retrieve them. This PR adds such index to the SQL database that will be used during migration to retrieve payment hashes.

…a hash The current sqlc GetInvoice query experiences incremental slowdowns during the migration of large invoice databases, primarily due to its complex predicate set. For this specific use case, a streamlined GetInvoiceByHash function provides a more efficient solution, maintaining near-constant lookup times even with extensive table sizes.

This commit runs the invoice migration if the user has a KV SQL backend configured.

Previously we'd recalculate the paid amount by summing amounts of settled HTLCs. This approach while correct would stop the SQL migration process as some KV invoices may have incorrectly stored paid amounts.

Previously, we applied replacements to our schema definitions to make them compatible with both SQLite and Postgres backends, as the files were not fully compatible with either. With this change, the only replacement required for SQLite has been moved to the generator script. This adjustment ensures compatibility by enabling auto-incrementing primary keys that are treated as 64-bit integers by sqlc.

guggero · 2025-01-23T11:48:19Z

The "Check commits" CI step succeeded locally, probably just too many commits for GitHub.

ellemouton · 2025-02-01T08:15:28Z

reminder that we can remove this replace now:

lnd/go.mod

Line 210 in e403243

    
           // Temporary replace until https://github.com/lightningnetwork/lnd/pull/8831 is

bhandras · 2025-02-01T09:55:44Z

reminder that we can remove this replace now:

lnd/go.mod

Line 210 in e403243

// Temporary replace until https://github.com/lightningnetwork/lnd/pull/8831 is

Thanks for the reminder, ptal #9469

bhandras self-assigned this Jun 12, 2024

bhandras added database Related to the database/storage of LND migration labels Jun 12, 2024

bhandras added this to the 0.19.0 milestone Jun 12, 2024

bhandras force-pushed the sql-invoice-migration branch 3 times, most recently from 6682b50 to 338e1f0 Compare June 12, 2024 15:30

bhandras force-pushed the sql-invoice-migration branch 3 times, most recently from d2a329f to 6379a8b Compare June 21, 2024 17:37

bhandras mentioned this pull request Jul 31, 2024

mod: bump kvdb to v1.4.10 #8959

Merged

bhandras force-pushed the sql-invoice-migration branch 2 times, most recently from 5fe92e2 to a7bf598 Compare August 14, 2024 09:38

aakselrod reviewed Sep 5, 2024

View reviewed changes

invoices/sql_migration.go Show resolved Hide resolved

bhandras force-pushed the sql-invoice-migration branch 5 times, most recently from b6f0ac8 to b983851 Compare September 17, 2024 14:42

bhandras changed the title ~~[wip] invoices: migrate KV invoices to native SQL~~ invoices: migrate KV invoices to native SQL for users of KV SQL backends Sep 17, 2024

bhandras force-pushed the sql-invoice-migration branch from b983851 to 706b444 Compare September 17, 2024 14:51

bhandras marked this pull request as ready for review September 17, 2024 14:51

bhandras force-pushed the sql-invoice-migration branch 7 times, most recently from 96f0cbe to bfe4ad5 Compare September 19, 2024 15:08

Abdulkbk reviewed Dec 17, 2024

View reviewed changes

ziggie1984 approved these changes Jan 13, 2025

View reviewed changes

ziggie1984 reviewed Jan 13, 2025

View reviewed changes

bhandras mentioned this pull request Jan 22, 2025

[feature]: Prevent Switching from Native SQL to KV SQL After Invoice Migration #9434

Closed

ziggie1984 mentioned this pull request Jan 22, 2025

[bug]: Invoice Amount Paid Field also account for canceled/accepted HTLCs #9436

Open

bhandras added 21 commits January 23, 2025 09:10

mod: temporarily replace sqldb with local version

6803945

sqldb: add table to track custom SQL migrations

9acd06d

This commit adds the migration_tracker table which we'll use to track if a custom migration has already been done.

multi: add call to directly insert an AMP sub-invoice

115f96c

sqldb: add a temporary index to store KV invoice hash to ID mapping

b7d7439

sqldb: remove unused preimage query parameter

d65b630

invoices: extract method to create invoice insertion params

be18f55

invoices: add migration code for a single invoice

708bed5

invoices: add migration code that runs a full invoice DB SQL migration

b92f57e

lnd: run invoice migration on startup

8d20e2a

This commit runs the invoice migration if the user has a KV SQL backend configured.

itest: add integration test for invoice migration

a29f243

itest: remove obsolete itest

0839d4b

invoices+sql: use the stored AmtPaid value instead of recalculating

5e3ef3e

Previously we'd recalculate the paid amount by summing amounts of settled HTLCs. This approach while correct would stop the SQL migration process as some KV invoices may have incorrectly stored paid amounts.

invoices: allow migration test to work on kv sqlite channeldb

ea98933

invoices: raise the number of allowed clients for the Postgres fixture

97c025f

docs: update release notes for 0.19.0

b1a462d

		// version. If a migration includes a non-nil MigrationFn, it is executed after
		// the SQL schema has been migrated to the corresponding schema version.

	invoiceStateReady := accepted \|\| settled

	if !invoiceIsAMP {
	// Update the running amount paid to this invoice. We
	// don't include accepted htlcs when the invoice is
	// still open.
	if invoice.State != ContractOpen &&
	invoiceStateReady {

	amtPaid += htlc.Amt

Conversation

bhandras commented Jun 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Description

Uh oh!

coderabbitai bot commented Jun 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ziggie1984 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ziggie1984 Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lightninglabs-deploy commented Jan 20, 2025

Uh oh!

guggero commented Jan 23, 2025

Uh oh!

ellemouton commented Feb 1, 2025

Uh oh!

bhandras commented Feb 1, 2025

bhandras commented Jun 12, 2024 •

edited

Loading

coderabbitai bot commented Jun 12, 2024 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)

ziggie1984 Jan 13, 2025 •

edited

Loading