Improve block connection logging #2559

Sharmalm · 2023-09-07T06:23:16Z

this pr solves #2348.

TheBlueMatt · 2023-09-07T16:59:22Z

handle_channel_ready is a message processing function, and unrelated to block connection, #2348 is about ensuring we can trace what happened in the block_connected/transactions_confirmed logic looking at just the logs.

codecov-commenter · 2023-09-09T08:41:13Z

Codecov Report

Attention: 249 lines in your changes are missing coverage. Please review.

Comparison is base (f3616e6) 90.57% compared to head (c8f1fe6) 89.04%.
Report is 292 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2559      +/-   ##
==========================================
- Coverage   90.57%   89.04%   -1.53%     
==========================================
  Files         110      112       +2     
  Lines       57557    86947   +29390     
  Branches    57557    86947   +29390     
==========================================
+ Hits        52130    77419   +25289     
- Misses       5427     7296    +1869     
- Partials        0     2232    +2232

Files	Coverage Δ
lightning-invoice/src/utils.rs	`94.80% <100.00%> (-2.87%)`	⬇️
lightning-persister/src/test_utils.rs	`100.00% <100.00%> (ø)`
lightning/src/chain/channelmonitor.rs	`84.91% <100.00%> (-9.68%)`	⬇️
lightning/src/ln/chanmon_update_fail_tests.rs	`97.69% <ø> (-1.03%)`	⬇️
lightning/src/ln/channel.rs	`88.35% <100.00%> (-1.49%)`	⬇️
lightning/src/ln/channel_id.rs	`97.14% <ø> (-2.86%)`	⬇️
lightning/src/ln/features.rs	`87.58% <100.00%> (-10.48%)`	⬇️
lightning/src/ln/functional_test_utils.rs	`91.38% <100.00%> (+2.39%)`	⬆️
lightning/src/ln/functional_tests.rs	`97.28% <100.00%> (-0.90%)`	⬇️
lightning/src/ln/inbound_payment.rs	`91.35% <100.00%> (-0.43%)`	⬇️
... and 30 more

... and 71 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt

Instead of merging the upstream code into your PR, please rebase onto upstream if there is a conflict. If there's not actually a conflict, please don't bother.

TheBlueMatt · 2023-09-14T18:46:33Z

lightning/src/ln/channelmanager.rs

+		for tx in txdata.iter() {
+			let (index, transaction) = tx;
+			let txid = transaction.txid();
+			log_trace!(self.logger, "Transaction id {} confirmed in block {}", txid, block_hash);


I dont think we should log every transaction, as many users actually connect all transactions in a block. Instead, we should look for transactions we care about, and maybe only actually add logging in ChannelMonitor where we can do that.

TheBlueMatt

We already filter block data at the top of transactions_confirmed, so you can print for everything in txn_matched

Sharmalm · 2023-09-19T18:38:21Z

We already filter block data at the top of transactions_confirmed, so you can print for everything in txn_matched

I have updated the changes,please review

TheBlueMatt · 2023-09-26T20:31:34Z

Instead of merging the upstream code into your PR, please rebase onto upstream. Merging the upstream branch via git merge/git pull makes review very difficult as its not clear what the actual head commit is.

TheBlueMatt · 2023-09-26T20:32:36Z

The issue also mentions "maybe also log when we add things to the filter." which I think we should absolutely do here.

lightning/src/chain/channelmonitor.rs

TheBlueMatt · 2023-09-26T20:35:15Z

Can you also log what's going on at the top of best_block_updated?

Sharmalm · 2023-09-27T09:29:16Z

Instead of merging the upstream code into your PR, please rebase onto upstream. Merging the upstream branch via git merge/git pull makes review very difficult as its not clear what the actual head commit is.

I will keep in mind from my next PR.

Sharmalm · 2023-09-30T08:41:08Z

I have updated the changes please review :)

jbesraa · 2023-10-01T12:33:32Z

Thanks for working on this @Sharmalm
You should merge the 7 commits into a single commit, that would also fix the remaining ci check I guess.
Look here #2511 (review) for more info regarding git conventions

There are several conditions that can be violated which indicate a stale monitor. This logs each that doesn't hold.

- Add fuzz test for `NetAddress` `from_str` function

We upstream the `KVStore` interface trait from LDK Node, which will replace `KVStorePersister` in the coming commits. Besides persistence, `KVStore` implementations will also offer to `list` keys present in a given `namespace` and `read` the stored values.

We add a utility function needed by upcoming `KVStore` implementation tests.

We upstream the `FilesystemStore` implementation, which is backwards compatible with `lightning-persister::FilesystemPersister`.

This replaces the `FilesystemPersister::read_channelmonitors` method, as we can now implement a single utility for all `KVStore`s.

Firstly, we switch our BP over to use `FilesystemStore`, which also gives us test coverage and ensures the compatibility. Then, we remove the superseded `KVStorePersister` trait and the `FilesystemPersister` code.

We re-add benchmarking for `FilesystemStore` now that we switched over to it.

When an invoice is requested but either receives an error or never receives a response, surface an event to indicate to the user that the corresponding future payment has failed.

When a BOLT 12 invoice has been requested, in order to guarantee invoice payment idempotency the future payment needs to be tracked. Add an AwaitingInvoice variant to PendingOutboundPayment such that only requested invoices are paid only once. Timeout after a few timer ticks if a request has not been received.

When a BOLT 12 invoice has been received, a payment attempt is made and any errors result in abandoning the PendingOutboundPayment. This results in generating at PaymentFailed event, which has a PaymentHash. Thus, when receiving an invoice, transition from AwaitingInvoice to a new InvoiceReceived state, the latter of which contains a PaymentHash such the abandon_payment helper can still be used.

Consolidate the creation and insertion of onion_session_privs to the PendingOutboundPayment::Retryable arm. In an upcoming commit, this method will be reused for an initial BOLT 12 invoice payment. However, onion_session_privs are created using a different helper.

It will be used for initial attempts at paying BOLT 12 invoices, so rename it something that covers both that and retries. Support paying BOLT 12 invoices Add a send_payment_for_bolt12_invoice method to OutboundPayments for initiating payment of a BOLT 12 invoice. This will be called from an OffersMessageHandler, after which any retries are handled using the Retryable logic. pub(crate) visibility for offers/test_utils.rs The test utilities for Offers are needed for testing message handling in ChannelManager and OutboundPayments. Add tests for send_payment_for_bolt12_invoice Use u32 instead of usize in Retry::Attempts An upcoming commit requires serializing Retry, so use a type with a fixed byte length. Otherwise, using eight bytes to serialize a usize would fail to read on 32-bit machines. Configure BOLT 12 invoice payment retry strategy Replace a constant three retry attempts for BOLT 12 invoice payments with a retry strategy specified when creating a pending outbound payment. This is configured by users in a later commit when constructing an InvoiceRequest or a Refund. Rename SocketAddress from NetAddress

Sharmalm · 2023-10-01T13:36:42Z

Thanks for working on this @Sharmalm You should merge the 7 commits into a single commit, that would also fix the remaining ci check I guess. Look here #2511 (review) for more info regarding git conventions

Thanks for all support and guidance from mentors.

Sharmalm · 2023-10-01T13:47:22Z

@jbesraa is rebasing correct?

jbesraa · 2023-10-01T15:08:33Z

unfortunately its not, you should endup with a single commit in this pr.
when using git you can use "merge" or "rebase" to integrate changes, when one use "merge" the commit history is not clean enough. I suggest to read the differences between them as it looks you understand the "merge" way, so that should make it easier to understand the "rebase" way. its also helpful to look into merged PRs to get a sense of the git conventions used in the project.

Sharmalm · 2023-10-01T15:36:41Z

Ok So , should I rebase all the above commits so that there should be one commit left in PR

jbesraa · 2023-10-01T16:09:17Z

yes, this might be heplful https://stackoverflow.com/questions/17577409/git-remove-merge-commit-from-history. and this https://stackoverflow.com/questions/16741724/does-pulling-from-a-remote-give-the-branches-a-different-history could be a good read to understand merge vs rebase

Sharmalm · 2023-10-02T09:54:06Z

Hello , I think i messed up with branches and rebasing , so i will close this PR and make a new PR .
Thanks :)

logging channel_id of connected txids

1cece1b

Merge branch 'lightningdevkit:main' into main

b3c824f

Sharmalm added 2 commits September 9, 2023 16:00

logging confirmed txid

b2236a6

Merge remote-tracking branch 'refs/remotes/origin/main'

7272dc5

TheBlueMatt reviewed Sep 14, 2023

View reviewed changes

logging in channel monitor

743f548

TheBlueMatt reviewed Sep 17, 2023

View reviewed changes

logging all filtered txids

10ea10e

TheBlueMatt reviewed Sep 26, 2023

View reviewed changes

lightning/src/chain/channelmonitor.rs Outdated Show resolved Hide resolved

adding txid to filter and logging best_block_update

3477eea

benthecarman and others added 11 commits October 1, 2023 18:26

Handle if funding output is in a coinbase transaction

8f1e8b6

Support creating coinbase funding transactions in tests

3b30a80

Add test for coinbase funding transactions

87ac1bf

Add constructor to RouteParameters

fe0a7fc

Have get_route take RouteParameters

203a847

Have Route hold RouteParameters

976d077

Include overpaid value in Route::get_total_fees

539d60d

Improve Route::get_total_amount docs

9963e16

Log each condition that was violated for a stale monitor

40d901f

There are several conditions that can be violated which indicate a stale monitor. This logs each that doesn't hold.

Fix misspelling, s/and/but/.

b874ccf

logging channel_id of connected txids

eb593db

Sharmalm and others added 23 commits October 1, 2023 18:26

logging confirmed txid

62afba6

Add RFC4648 base32 encode and decode functions

e052324

Move zbase32 implementation to base32 file

84e3ba6

Fix indent in message_signing.rs

b401464

Implement from_str trait for NetAddress

4d31af3

- Add fuzz test for `NetAddress` `from_str` function

Add KVStore interface trait

ef14292

We upstream the `KVStore` interface trait from LDK Node, which will replace `KVStorePersister` in the coming commits. Besides persistence, `KVStore` implementations will also offer to `list` keys present in a given `namespace` and `read` the stored values.

Update lightning-persister crate

e82f427

Add test_utils

5591d34

We add a utility function needed by upcoming `KVStore` implementation tests.

Add FilesystemStore

e9321a5

We upstream the `FilesystemStore` implementation, which is backwards compatible with `lightning-persister::FilesystemPersister`.

Add read_channel_monitors utility

485cc59

This replaces the `FilesystemPersister::read_channelmonitors` method, as we can now implement a single utility for all `KVStore`s.

Migrate to KVStore/FilesystemStore

a7ce0e3

Firstly, we switch our BP over to use `FilesystemStore`, which also gives us test coverage and ensures the compatibility. Then, we remove the superseded `KVStorePersister` trait and the `FilesystemPersister` code.

Migrate FilesystemPersister tests to FilesystemStore

20433de

Add benchmarking for FilesystemStore

bc27771

We re-add benchmarking for `FilesystemStore` now that we switched over to it.

Add TestStore implementation of KVStore

d731a35

Add an InvoiceRequestFailed event

446e078

When an invoice is requested but either receives an error or never receives a response, surface an event to indicate to the user that the corresponding future payment has failed.

Move IDEMPOTENCY_TIMEOUT_TICKS to where it is used

1f2d0e7

Test for removing stale AwaitingInvoice payments

9c4739a

Test removing abandoned AwaitingInvoice payments

1ce341b

Merge branch 'main' of https://github.com/Sharmalm/rust-lightning

c8f1fe6

Sharmalm closed this Oct 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve block connection logging #2559

Improve block connection logging #2559

Sharmalm commented Sep 7, 2023

TheBlueMatt commented Sep 7, 2023

codecov-commenter commented Sep 9, 2023 •

edited

Loading

TheBlueMatt left a comment

TheBlueMatt Sep 14, 2023

Sharmalm Sep 17, 2023

TheBlueMatt left a comment

Sharmalm commented Sep 19, 2023 •

edited

Loading

TheBlueMatt commented Sep 26, 2023

TheBlueMatt commented Sep 26, 2023

TheBlueMatt commented Sep 26, 2023

Sharmalm commented Sep 27, 2023

Sharmalm commented Sep 30, 2023

jbesraa commented Oct 1, 2023

Sharmalm commented Oct 1, 2023

Sharmalm commented Oct 1, 2023

jbesraa commented Oct 1, 2023

Sharmalm commented Oct 1, 2023 •

edited

Loading

jbesraa commented Oct 1, 2023

Sharmalm commented Oct 2, 2023

Improve block connection logging #2559

Improve block connection logging #2559

Conversation

Sharmalm commented Sep 7, 2023

TheBlueMatt commented Sep 7, 2023

codecov-commenter commented Sep 9, 2023 • edited Loading

Codecov Report

TheBlueMatt left a comment

Choose a reason for hiding this comment

TheBlueMatt Sep 14, 2023

Choose a reason for hiding this comment

Sharmalm Sep 17, 2023

Choose a reason for hiding this comment

TheBlueMatt left a comment

Choose a reason for hiding this comment

Sharmalm commented Sep 19, 2023 • edited Loading

TheBlueMatt commented Sep 26, 2023

TheBlueMatt commented Sep 26, 2023

TheBlueMatt commented Sep 26, 2023

Sharmalm commented Sep 27, 2023

Sharmalm commented Sep 30, 2023

jbesraa commented Oct 1, 2023

Sharmalm commented Oct 1, 2023

Sharmalm commented Oct 1, 2023

jbesraa commented Oct 1, 2023

Sharmalm commented Oct 1, 2023 • edited Loading

jbesraa commented Oct 1, 2023

Sharmalm commented Oct 2, 2023

codecov-commenter commented Sep 9, 2023 •

edited

Loading

Sharmalm commented Sep 19, 2023 •

edited

Loading

Sharmalm commented Oct 1, 2023 •

edited

Loading