Implement detailed logging for XCM failures #8724

raymondkfcheung · 2025-06-02T14:53:23Z

This PR enhances diagnostics across XCM-related components by implementing more detailed logging, especially for failures. The primary aim is to improve visibility into XCM errors (e.g., within map_err blocks), enabling faster troubleshooting for developers and node operators.

Key enhancements include:

Logging specific error conditions (e.g., BadVersion, BadLocation, execution errors).
Including essential context in error logs (e.g., message hashes, origin, destination, relevant parameters).
Standardising log targets for easier analysis (e.g., xcm::module::function_name).
Clarifying log messages.

This work continues #7003 and partially addresses #6119.

Integration

Downstream projects using XCM functionalities should see improved diagnostic logs without needing direct integration changes, as this primarily enhances internal logging. Node operators will find new, more detailed XCM error logs, aiding in monitoring and troubleshooting. Log parsing scripts might need updates for new log formats.

Review Notes

This PR enhances logging throughout XCM-related code paths, particularly in error handling (e.g., map_err blocks), to improve failure visibility.

Reviewers are encouraged to assess:

Clarity & Actionability: Are logs clear, concise, and actionable for failures?
Context: Do error logs capture essential context (message hash, origin, destination, error specifics, parameters) appropriately for their level?
Log Levels: Is the chosen log level (e.g., error, warn, debug, trace) suitable? Are any level adjustments appropriate?
Targets: Are target: fields (e.g., xcm::module::function) consistent and specific?
Error Accuracy: Do logs accurately reflect the handled error condition?
Impact: Logging should minimally affect core logic and performance; favour simple logging of existing/easily derived data.

raymondkfcheung · 2025-06-02T15:17:07Z

/cmd prdoc --audience runtime_dev --bump patch

…time_dev --bump patch'

acatangiu

Please change all the log::error! to log::debug! - error logs are reserved for errors relevant to node operator - these types of runtime errors should be coming up only when debugging (otherwise validators get spammed with error logs for on-chain stuff that has no relevance to them).

The rest is good!

cumulus/pallets/xcmp-queue/src/lib.rs

cumulus/parachains/common/src/xcm_config.rs

polkadot/xcm/pallet-xcm/src/lib.rs

bkontur · 2025-06-03T07:47:28Z

cumulus/pallets/xcmp-queue/src/lib.rs

 			_ => {
 				// Anything else is unhandled. This includes a message that is not meant for us.
 				// We need to make sure that dest/msg is not consumed here.
+				log::error!(target: LOG_TARGET, "Failed to validate XCM destination: unexpected location {d:?}");


definitely not error here, because we use XcmpQueue in XcmRouter tuple and for example on AssetHubs, we have other routers after XcmpQueue for Ethereum or Polkadot/Kusama bridge routers.
So basically, we would log this error for every other location than (1, [Parachain(id)]) => {

Suggested change

log::error!(target: LOG_TARGET, "Failed to validate XCM destination: unexpected location {d:?}");

log::trace!(target: LOG_TARGET, "Failed to validate XCM destination: unexpected location {d:?}");

Shall I use debug or trace on xcmp-queue/src/lib.rs?

this tuple SendXcm routing is expected not to pass depending on the d, so definitely trace,
maybe questionable if really need a log here, but if so, then trace

Agreed, removed logs.

cumulus/pallets/xcmp-queue/src/lib.rs

bkontur · 2025-06-04T10:03:42Z

cumulus/pallets/xcmp-queue/src/lib.rs

 			_ => {
 				// Anything else is unhandled. This includes a message that is not meant for us.
 				// We need to make sure that dest/msg is not consumed here.
+				log::debug!(target: LOG_TARGET, "Failed to validate XCM destination: unexpected location {d:?}");


Didn't we want to remove this log?

#8724 (comment)

bkontur · 2025-06-04T10:04:56Z

cumulus/parachains/common/src/xcm_config.rs

 		let asset_amount = BalanceConverter::to_asset_balance(amount, asset_id)
-			.map_err(|_| XcmError::TooExpensive)?;
+			.map_err(|error| {
+				log::debug!(target: "xcm::charge_weight_in_fungibles", "AssetFeeAsExistentialDepositMultiplier Cannot convert to valid balance (possibly below ED): {error:?}");


Suggested change

log::debug!(target: "xcm::charge_weight_in_fungibles", "AssetFeeAsExistentialDepositMultiplier Cannot convert to valid balance (possibly below ED): {error:?}");

log::debug!(target: "xcm::charge_weight_in_fungibles", "AssetFeeAsExistentialDepositMultiplier cannot convert to valid balance (possibly below ED): {error:?}");

bkontur · 2025-06-04T10:08:21Z

polkadot/xcm/xcm-builder/src/currency_adapter.rs

-			.map_err(|e| XcmError::FailedToTransactAsset(e.into()))?;
+		let _ = Currency::withdraw(&who, amount, WithdrawReasons::TRANSFER, AllowDeath).map_err(
+			|error| {
+				tracing::debug!(target: "xcm::currency_adapter", ?error, "Failed to withdraw asset");


Can we add here &who, amount to the log?

bkontur · 2025-06-04T10:08:47Z

polkadot/xcm/xcm-builder/src/currency_adapter.rs

-		Currency::transfer(&from, &to, amount, AllowDeath)
-			.map_err(|e| XcmError::FailedToTransactAsset(e.into()))?;
+		Currency::transfer(&from, &to, amount, AllowDeath).map_err(|error| {
+			tracing::debug!(target: "xcm::currency_adapter", ?error, "Failed to transfer asset");


Also here, I would add &from, &to, amount to the log

bkontur · 2025-06-04T10:09:37Z

polkadot/xcm/xcm-builder/src/fungible_adapter.rs

-			.map_err(|error| XcmError::FailedToTransactAsset(error.into()))?;
+		Fungible::transfer(&source, &dest, amount, Expendable).map_err(|error| {
+			tracing::debug!(
+				target: "xcm::fungible_adapter", ?error,


Also here: &source, &dest, amount - please check also other logs, if possible to add without clone

bkontur · 2025-06-04T10:11:06Z

prdoc/pr_8724.prdoc

+    This PR improves diagnostics in XCM-related code by adding detailed error logging, especially within map_err paths. It includes clearer messages, standardized log targets, and richer context to aid runtime developers and node operators in debugging and monitoring.
+crates:
+- name: cumulus-pallet-xcmp-queue
+  bump: minor


can be patch according to https://github.com/paritytech/polkadot-sdk/actions/runs/15417460762/job/43383686510?pr=8724#step:11:103

Suggested change

bump: minor

bump: patch

* master: omni-node: fix `benchmark pallet` to work with `--runtime` (#8594) Handle and suppress "New unknown `FromSwarm` libp2p event" warning (#8731) Implement detailed logging for XCM failures (#8724) [pallet-revive] contract's nonce starts at 1 (#8734) sync/fix: Clear gap sync on known imported blocks (#8445) [PoP] Add personhood tracking pallets (#8164) client/net: Use litep2p as the default network backend (#8461) Unflake `returns_status_for_pruned_blocks` (#8709) [AHM] Report the weights of epmb pallet to expose kusama and polkadot weights (#8704) Remove all XCM dependencies from `pallet-revive` (#8584) Docker master image tag fix (#8711) Record ed as part of the storage deposit (#8718) [pallet-revive] update dry-run logic (#8662) feat: add collator peer ID to ParachainInherentData (#8708) Nest errors in pallet-xcm (#7730) pallet-assets ERC20 precompile (#8554) Broker: Introduce min price + adjust renewals to lower market. (#8630) [AHM] Staking async fixes for XCM and election planning (#8422) Staking (EPMB): Add defensive error handling to voter snapshot creation and solution verification (#8687)

This PR enhances diagnostics across XCM-related components by implementing more detailed logging, especially for failures. The primary aim is to improve visibility into XCM errors (e.g., within `map_err` blocks), enabling faster troubleshooting for developers and node operators. Key enhancements include: * Logging specific error conditions (e.g., `BadVersion`, `BadLocation`, execution errors). * Including essential context in error logs (e.g., message hashes, origin, destination, relevant parameters). * Standardising log targets for easier analysis (e.g., `xcm::module::function_name`). * Clarifying log messages. This work continues #7003 and partially addresses #6119. ## Integration Downstream projects using XCM functionalities should see improved diagnostic logs without needing direct integration changes, as this primarily enhances internal logging. Node operators will find new, more detailed XCM error logs, aiding in monitoring and troubleshooting. Log parsing scripts might need updates for new log formats. ## Review Notes This PR enhances logging throughout XCM-related code paths, particularly in error handling (e.g., `map_err` blocks), to improve failure visibility. Reviewers are encouraged to assess: * **Clarity & Actionability:** Are logs clear, concise, and actionable for failures? * **Context:** Do error logs capture essential context (message hash, origin, destination, error specifics, parameters) appropriately for their level? * **Log Levels:** Is the chosen log level (e.g., `error`, `warn`, `debug`, `trace`) suitable? Are any level adjustments appropriate? * **Targets:** Are `target:` fields (e.g., `xcm::module::function`) consistent and specific? * **Error Accuracy:** Do logs accurately reflect the handled error condition? * **Impact:** Logging should minimally affect core logic and performance; favour simple logging of existing/easily derived data. --------- Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

This PR replaces `log` with `tracing` instrumentation on XCM-related modules to significantly improve debugging capabilities for XCM execution flows. Continues #8724 and partially addresses #6119 by providing structured logging throughout XCM components, making it easier to diagnose execution failures, fee calculation errors, and routing issues. ## Key Features - **Consistent targets**: All components use predictable `xcm::*` log targets - **Structured fields**: Uses `?variable` syntax for automatic Debug formatting - **Zero runtime impact**: No behavioural changes, only observability improvements

This PR enhances diagnostics across XCM-related components by implementing more detailed logging, especially for failures. The primary aim is to improve visibility into XCM errors (e.g., within `map_err` blocks), enabling faster troubleshooting for developers and node operators. Key enhancements include: * Logging specific error conditions (e.g., `BadVersion`, `BadLocation`, execution errors). * Including essential context in error logs (e.g., message hashes, origin, destination, relevant parameters). * Standardising log targets for easier analysis (e.g., `xcm::module::function_name`). * Clarifying log messages. This work continues #7003 and partially addresses #6119. ## Integration Downstream projects using XCM functionalities should see improved diagnostic logs without needing direct integration changes, as this primarily enhances internal logging. Node operators will find new, more detailed XCM error logs, aiding in monitoring and troubleshooting. Log parsing scripts might need updates for new log formats. ## Review Notes This PR enhances logging throughout XCM-related code paths, particularly in error handling (e.g., `map_err` blocks), to improve failure visibility. Reviewers are encouraged to assess: * **Clarity & Actionability:** Are logs clear, concise, and actionable for failures? * **Context:** Do error logs capture essential context (message hash, origin, destination, error specifics, parameters) appropriately for their level? * **Log Levels:** Is the chosen log level (e.g., `error`, `warn`, `debug`, `trace`) suitable? Are any level adjustments appropriate? * **Targets:** Are `target:` fields (e.g., `xcm::module::function`) consistent and specific? * **Error Accuracy:** Do logs accurately reflect the handled error condition? * **Impact:** Logging should minimally affect core logic and performance; favour simple logging of existing/easily derived data. --------- Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

This PR replaces `log` with `tracing` instrumentation on XCM-related modules to significantly improve debugging capabilities for XCM execution flows. Continues #8724 and partially addresses #6119 by providing structured logging throughout XCM components, making it easier to diagnose execution failures, fee calculation errors, and routing issues. ## Key Features - **Consistent targets**: All components use predictable `xcm::*` log targets - **Structured fields**: Uses `?variable` syntax for automatic Debug formatting - **Zero runtime impact**: No behavioural changes, only observability improvements

raymondkfcheung added 5 commits May 30, 2025 16:21

Add debug logs

a37ce6f

Add debug logs

f2bc0c4

Merge branch 'master' into ray-add-err-logs

9fae961

Add debug logs

df7c14e

Merge branch 'master' into ray-add-err-logs

22ed3f5

raymondkfcheung requested a review from a team as a code owner June 2, 2025 14:53

raymondkfcheung changed the title ~~Ray add err logs~~ Implement detailed logging for XCM failures Jun 2, 2025

raymondkfcheung self-assigned this Jun 2, 2025

raymondkfcheung added the T6-XCM This PR/Issue is related to XCM. label Jun 2, 2025

raymondkfcheung added this to Bridges + XCM Jun 2, 2025

raymondkfcheung moved this to In Progress in Bridges + XCM Jun 2, 2025

raymondkfcheung marked this pull request as draft June 2, 2025 15:18

github-actions bot and others added 6 commits June 2, 2025 15:20

Update from github-actions[bot] running command 'prdoc --audience run…

8cc9a7c

…time_dev --bump patch'

Update debug logs

1fe50c8

Update PRDoc

5da8217

Update PRDoc

aaf1770

Merge branch 'master' into ray-add-err-logs

fffaba6

Update PRDoc

0479c85

raymondkfcheung marked this pull request as ready for review June 2, 2025 16:01

acatangiu approved these changes Jun 2, 2025

View reviewed changes

cumulus/pallets/xcmp-queue/src/lib.rs Outdated Show resolved Hide resolved

acatangiu approved these changes Jun 2, 2025

View reviewed changes

cumulus/parachains/common/src/xcm_config.rs Outdated Show resolved Hide resolved

polkadot/xcm/pallet-xcm/src/lib.rs Outdated Show resolved Hide resolved

bkontur reviewed Jun 3, 2025

View reviewed changes

cumulus/pallets/xcmp-queue/src/lib.rs Outdated Show resolved Hide resolved

raymondkfcheung moved this from In Progress to In-Review in Bridges + XCM Jun 3, 2025

raymondkfcheung added 5 commits June 3, 2025 09:19

Merge branch 'master' into ray-add-err-logs

0586daf

Change to debug

4fde280

Add error

91dc578

Change to debug

da6e4da

Fix Debug

45a6f89

raymondkfcheung added 4 commits June 3, 2025 12:53

Update PRDoc

52206fe

Merge branch 'master' into ray-add-err-logs

7499f1b

Fix fmt

6ee98df

Merge branch 'paritytech:master' into ray-add-err-logs

9717549

raymondkfcheung mentioned this pull request Jun 3, 2025

Replace log with tracing on XCM-related modules #8732

Merged

raymondkfcheung enabled auto-merge June 4, 2025 08:37

bkontur reviewed Jun 4, 2025

View reviewed changes

bkontur approved these changes Jun 4, 2025

View reviewed changes

raymondkfcheung added this pull request to the merge queue Jun 4, 2025

bkontur removed this pull request from the merge queue due to a manual request Jun 4, 2025

raymondkfcheung added 3 commits June 4, 2025 12:58

Merge branch 'master' into ray-add-err-logs

15ab385

Address the comments

891822c

Update to tracing

b818238

raymondkfcheung requested a review from bkontur June 4, 2025 12:52

bkontur enabled auto-merge June 4, 2025 13:03

raymondkfcheung disabled auto-merge June 4, 2025 13:17

raymondkfcheung added 2 commits June 4, 2025 14:18

Update PRDoc

6a14404

Merge branch 'master' into ray-add-err-logs

4eb34d6

raymondkfcheung enabled auto-merge June 4, 2025 13:21

raymondkfcheung added this pull request to the merge queue Jun 4, 2025

Merged via the queue into paritytech:master with commit 873cd45 Jun 4, 2025
244 checks passed

raymondkfcheung deleted the ray-add-err-logs branch June 4, 2025 14:58

github-project-automation bot moved this from In-Review to Done in Bridges + XCM Jun 4, 2025

github-actions bot mentioned this pull request Oct 24, 2025

Update polkadot-sdk from stable2503 to stable2506 moondance-labs/tanssi#1340

Closed

	log::error!(target: LOG_TARGET, "Failed to validate XCM destination: unexpected location {d:?}");
	log::trace!(target: LOG_TARGET, "Failed to validate XCM destination: unexpected location {d:?}");

	log::debug!(target: "xcm::charge_weight_in_fungibles", "AssetFeeAsExistentialDepositMultiplier Cannot convert to valid balance (possibly below ED): {error:?}");
	log::debug!(target: "xcm::charge_weight_in_fungibles", "AssetFeeAsExistentialDepositMultiplier cannot convert to valid balance (possibly below ED): {error:?}");

Implement detailed logging for XCM failures #8724

Implement detailed logging for XCM failures #8724

Uh oh!

Conversation

raymondkfcheung commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration

Review Notes

Uh oh!

raymondkfcheung commented Jun 2, 2025

Uh oh!

acatangiu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bkontur Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

raymondkfcheung commented Jun 2, 2025 •

edited

Loading

bkontur Jun 3, 2025 •

edited

Loading