Skip to content

Conversation

@yrong
Copy link
Contributor

@yrong yrong commented May 9, 2025

The TokenIdOf convert is XCM version-agnostic, meaning we will get the same token ID for both V5 and legacy V4 asset.

However, the extra check is unnecessary, as theConvertAssetId::convert(&token_id).ok_or(InvalidAsset)?; alone is sufficient to verify whether the token is registered.

Comment on lines -177 to -178
let expected_asset_id = ConvertAssetId::convert(&token_id).ok_or(InvalidAsset)?;
ensure!(asset_id == expected_asset_id, InvalidAsset);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@franciscoaguirre are there perhaps other ways to check asset ID equivalence, across different XCM versions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I assume the check here is unnecessary, as the conversion alone is sufficient to verify whether the token is registered.

@yrong yrong changed the title Remove asset location check for compatibility Snowbridge: Remove asset location check for compatibility May 9, 2025
@yrong yrong marked this pull request as ready for review May 12, 2025 12:10
@paritytech-review-bot paritytech-review-bot bot requested a review from a team May 12, 2025 12:11
@acatangiu acatangiu added T15-bridges This PR/Issue is related to bridges. A4-backport-stable2503 Pull request must be backported to the stable2503 release branch labels May 12, 2025
@vgeddes
Copy link
Contributor

vgeddes commented May 12, 2025

The TokenIdOf convert is XCM version-agnostic, meaning we will get the same token ID for both V5 and legacy V4 asset

Strictly speaking, TokenIdOf can actually return different token ids for a location encoded in V4 and in V5, as newer versions of XCM can add or remove various enum variants for NetworkId, and the scale-encoded network id is used as input into the token id generator: https://github.com/vgeddes/polkadot-sdk/blob/2567a5ac3a22fe669279277d32561a918a07ae58/bridges/snowbridge/primitives/core/src/location.rs#L67

Comment on lines 414 to 416
let token_id = TokenIdOf::convert_location(&asset_id).ok_or(InvalidAsset)?;

let expected_asset_id = ConvertAssetId::convert(&token_id).ok_or(InvalidAsset)?;

ensure!(asset_id == expected_asset_id, InvalidAsset);
ConvertAssetId::convert(&token_id).ok_or(InvalidAsset)?;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the location has already been registered as a PNA and a foreign token id has been allocated in storage for it, why not just use ConvertAssetId::convert_back to retrieve the stored id?

Recreating the token id deterministically using TokenIdOf::convert_location is dangerous for the reason I mentioned in #8473 (comment)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as newer versions of XCM can add or remove various enum variants for NetworkId

I assume removal is not allowed, or that the codec index should not chage. That's why I created PR#6503

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not just use ConvertAssetId::convert_back to retrieve the stored id?

As mentioned the token registered (in V4) is not the same as the token that was transferred (in V5). However, this reminds me that NativeToForeignId storage is actually unused, so I performed some cleanup accordingly.

dabf3bf

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned the token registered (in V4) is not the same as the token that was transferred (in V5).

Can you give an example scenario so that we can make the discussion more concrete?

Copy link
Contributor Author

@yrong yrong May 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the confusion earlier. Yes, technically we can use convert_back to obtain the tokenID from the location. But in my opinion, it’s unnecessary — the tokenID is already stored on Ethereum and must be permanent. This means that TokenIdOf::convert_location(&location) should remain stable across different XCM versions.

In my view, changes like those in this PR#5390 are not good practice. Adding new fields or enums is fine, but replacing or removing existing ones should not be allowed. This should be a core principle for all upgrades — whether on Substrate or Ethereum.

To ensure the tokenID remains stable, I’ve added tests in fellow-runtime.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my view, changes like those in this #5390 are not good practice. Adding new fields or enums is fine, but replacing or removing existing ones should not be allowed. This should be a core principle for all upgrades — whether on Substrate or Ethereum.

Well XCM is explicitly versioned. Parity's contract to us developers is that V4::NetworkId and V5::NetworkId are allowed to ABI-incompatible, as long as they are infallibly convertible to each other. Accordingly, there is no guarantee that #5390 won't again happen again for V6.

In light of this, I think using convert_back is more secure and robust.

Deterministic id generation is mathematically neat, but I feel there too many compatibility problems for me to feel safe using it here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is no guarantee that #5390 won't again happen again for V6.

That's why I added the tests as mentioned in this comment - the tests will fail before any issues go alive to production.

Actually, #5390 is not relevant to PNA transfers, since the codec index for the NetworkId of the Polkadot enum remains unchanged in V5. However, it will break ENA registration on AH, which is another incompatability issue we want to avoid - hence the need for PR#6503.

So I'd suggest avoiding incompatibility issues if possible.

@yrong
Copy link
Contributor Author

yrong commented May 13, 2025

By the way, in a recent commit, I changed the storage to use VersionedLocation.

IMHO, it’s always safer to use VersionedLocation in a storage context. Although this requires a runtime upgrade to migrate from a V4 location to VersionedLocation::V4 (and pinged as V4), no further migration is needed - as long as the conversion (from V4 to V5 or V6) works correctly.

@acatangiu @bkontur Please let me know what you think.

Comment on lines 239 to 234
StorageMap<_, Blake2_128Concat, xcm::v5::Location, TokenId, OptionQuery>;
StorageMap<_, Blake2_128Concat, TokenId, VersionedLocation, OptionQuery>;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, in a recent commit, I changed the storage to use VersionedLocation.

I am slightly leaning against this.

We support a sliding window of 3 XCM versions: [lastest, latest - 2]. So over time you still need migrations to migrate from e.g. VersionedLocation::V3() storage entries to VersionedLocation::V6() when we upgrade to XCMv6.

You get the advantage of somewhat smaller migrations (only oldest entries need migrating), but
you get the disadvantage of mixed storage holding various versions.

Neither option is truly better IMO, but I prefer consistency across pallets and storing base/unversioned types.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

disadvantage of mixed storage holding various versions.

IMHO, this may not be too problematic for this specific use case, as long as the derived TokenId is (and is intended to be) XCM version-agnostic.

Copy link
Contributor Author

@yrong yrong May 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a sliding window of 3 XCM versions: [lastest, latest - 2]

That's good. As mentioned, the current storage version is V4. Assuming we migrate from Location(V4) to VersionLocation::V5, that means we don't need to worry about a runtime migration until XCM V8.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer not to use VersionedLocation in storage either. I want highly normalized, consistent data in storage.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted recent updates.

Comment on lines 537 to 538
fn convert_back(_: &Location) -> Option<TokenId> {
None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why use MaybeEquivalence if you don't provide convert_back functionality?

Can you use TryConvert/MaybeConvert instead?

Copy link
Contributor Author

@yrong yrong May 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To keep this PR focused, I've opend a separate PR for the cleanup and refactoring work, which isn't high priority.

Copy link
Contributor

@vgeddes vgeddes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yrong must this be backported to stable2503? If so, lets focus on the immediate issue at hand - removal of the asset location check.

@yrong
Copy link
Contributor Author

yrong commented May 16, 2025

must this be backported to stable2503?

Based on the tests demonstrated in polkadot-fellows/runtimes#730 (comment), both the cleanup in this PR and the migration in polkadot-fellows/runtimes#730 are technically not required.

However, I still prefer to merge these PRs and include the runtime migration to reduce inconsistencies and ensure compatibility with future versions.

@acatangiu Please let me know what you think.

@yrong yrong requested a review from vgeddes May 16, 2025 02:48
@yrong yrong force-pushed the remove-asset-location-check branch from 62946bb to 697f4ed Compare May 16, 2025 03:22
@vgeddes
Copy link
Contributor

vgeddes commented May 20, 2025

must this be backported to stable2503?

Based on the tests demonstrated in polkadot-fellows/runtimes#730 (comment), both the cleanup in this PR and the migration in polkadot-fellows/runtimes#730 are technically not required.

However, I still prefer to merge these PRs and include the runtime migration to reduce inconsistencies and ensure compatibility with future versions.

acatangiu Please let me know what you think.

I agree that we should merge these prs to reduce inconsistencies.

@acatangiu
Copy link
Contributor

you should include yrong#19 here as well then merge and backport as patch to 2503

@yrong
Copy link
Contributor Author

yrong commented May 22, 2025

@vgeddes Are you okay with my comment? If so, I'll go ahead and merge yrong#19 , which essentially sticks to deterministic ID generation.

@vgeddes
Copy link
Contributor

vgeddes commented May 22, 2025

@vgeddes Are you okay with my comment? If so, I'll go ahead and merge yrong#19 , which essentially sticks to deterministic ID generation.

Yes, I think I am satisfied now. I see that #6503 adds V4<->V5 compatibility for NetworkId.

@acatangiu
Copy link
Contributor

It seems the CI requires me to mark this as a major API change to snowbridge-pallet-system.
@acatangiu This PR only removes the unused NativeToForeignId storage, which IMO is not a major breaking change. Do I still need to handle this failure?

CI does pure semver check on the crate - removing a public item from the crate is theoretically a major change since there might be some downstream user of the crate using that item.

However, we need to be pragmatic and think about actual crate usage as well. In this case, since snowbridge is the only user of this crate, you can use any type of bump since you are also handling the integration and know what is actually used and isn't.

Since you want to backport this to 2503, all bumps need to be patch or none - we cannot backport any minor or major bumps since those are by definition incompatible bumps and cargo will reject them downstream.

Also, for these situations where we override the automated semver recommendation, you should add validate: false under the bump.

Comment on lines 30 to 35
- name: bridge-hub-westend-runtime
bump: patch
validate: false
- name: bridge-hub-westend-integration-tests
bump: patch
validate: false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
- name: bridge-hub-westend-runtime
bump: patch
validate: false
- name: bridge-hub-westend-integration-tests
bump: patch
validate: false
- name: bridge-hub-westend-runtime
bump: none
validate: false
- name: bridge-hub-westend-integration-tests
bump: none
validate: false

test runtimes are not published so can be bump: none - as a general rule, if it has westend in the name, we don't publish.

@acatangiu acatangiu enabled auto-merge May 27, 2025 07:51
@acatangiu acatangiu added this pull request to the merge queue May 27, 2025
Merged via the queue into paritytech:master with commit 64aed49 May 27, 2025
183 of 184 checks passed
paritytech-release-backport-bot bot pushed a commit that referenced this pull request May 27, 2025
The `TokenIdOf`
[convert](https://github.com/paritytech/polkadot-sdk/blob/4b83d24f4bc96a7b17964be94b178dd7b8f873b5/bridges/snowbridge/primitives/core/src/location.rs#L40)
is XCM version-agnostic, meaning we will get the same token ID for both
V5 and legacy V4 asset.

However, the extra check is unnecessary, as
the`ConvertAssetId::convert(&token_id).ok_or(InvalidAsset)?;` alone is
sufficient to verify whether the token is registered.

(cherry picked from commit 64aed49)
@paritytech-release-backport-bot

Successfully created backport PR for stable2503:

ordian added a commit that referenced this pull request May 27, 2025
* master: (99 commits)
  Snowbridge: Remove asset location check for compatibility (#8473)
  add poke_deposit extrinsic to pallet-bounties (#8382)
  litep2p/peerset: Reject non-reserved peers in the reserved-only mode (#8650)
  Charge deposit based on key length (#8648)
  [pallet-revive] make subscription task panic on error (#8587)
  tx/metrics: Add metrics for the RPC v2 `transactionWatch_v1_submitAndWatch` (#8345)
  Bridges: Fix - Improve try-state for pallet-xcm-bridge-hub (#8615)
  Introduce CreateBare, deprecated CreateInherent (#7597)
  Use hashbrown hashmap/hashset in validation context (#8606)
  ci: rm gitlab config (#8622)
  🔪 flaky and Zombienet tests (#8600)
  cumulus: adjust unincluded segment size metric buckets (#8617)
  Benchmark storage access on block validation (#8069)
  Revert 7934 es/remove tj changes (#8611)
  collator-protocol: add more collation observability (#8230)
  `fatxpool`: add fallback for ready at light (#8533)
  txpool: fix tx removal from unlocks set (#8500)
  XCMP weight metering: account for the MQ page position (#8344)
  fix epmb solution duplicate issue + add remote mining apparatus to epm (#8585)
  Fix generated address returned by Substrate RPC runtime call (#8504)
  ...
EgorPopelyaev added a commit that referenced this pull request Jun 2, 2025
Backport #8473 into `stable2503` from yrong.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

---------

Co-authored-by: Ron <[email protected]>
Co-authored-by: Egor_P <[email protected]>
pgherveou pushed a commit that referenced this pull request Jun 11, 2025
The `TokenIdOf`
[convert](https://github.com/paritytech/polkadot-sdk/blob/4b83d24f4bc96a7b17964be94b178dd7b8f873b5/bridges/snowbridge/primitives/core/src/location.rs#L40)
is XCM version-agnostic, meaning we will get the same token ID for both
V5 and legacy V4 asset.

However, the extra check is unnecessary, as
the`ConvertAssetId::convert(&token_id).ok_or(InvalidAsset)?;` alone is
sufficient to verify whether the token is registered.
alstjd0921 pushed a commit to bifrost-platform/polkadot-sdk that referenced this pull request Aug 14, 2025
Backport paritytech#8473 into `stable2503` from yrong.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

---------

Co-authored-by: Ron <[email protected]>
Co-authored-by: Egor_P <[email protected]>
@girazoki
Copy link
Contributor

girazoki commented Sep 26, 2025

@acatangiu I think this change makes the system be less flexible to errors to win only one less read when doing the conversions. I think the conversion from location to token-id should happen always through whatever is dictated in pallet-ethereum-system and NOT by tokenIdOf (cc @bkontur )

I am going to put a few examples in which things can go wrong:

  • you somehow update any of the implementations that are part of tokenIdOf (e.g. describeFamily, DescribeGlobalPrefix, DescribeGlobalPrefix).

  • you somehow update the implementation of HashedDescription

  • you somehow update your token location (e.g., pallet-assets references now with a different junction the assetId)

In any of those cases the hash of the token-location will not be the same anymore. meaning that you lose the opportunity to transfer the tokens to ethereum.

I understand this is coming from the fact that all conversions from token location to token-id where done with tokenIdOf, but I truly believe this is a mistake. Instead, I would keep using tokenIdOf but only when you register the token in EthereumSystem, and after that I would:

  • maitnain both ForeignToNative and NativeToForeign mappings
  • use EthereumSystem to convert from location to token-id and viceversa

I think this is less prone to errors and adds much more flexibility than the current scenario.

I am sorry I did not realize about this change before. I could have provided earlier feedback

@acatangiu
Copy link
Contributor

I am going to put a few examples in which things can go wrong:
you somehow update any of the implementations that are part of tokenIdOf (e.g. describeFamily, DescribeGlobalPrefix, DescribeGlobalPrefix).
you somehow update the implementation of HashedDescription
you somehow update your token location (e.g., pallet-assets references now with a different junction the assetId)

all of the above are deeply destructive, breaking changes :) if any of them is attempted maaany things will break, not just this ethereum token id conversion (think thousands of derived accounts, derived locations, and all the hardcoded stuff off-chain)

having said that, I am not against making tokenIdOf more flexible - I am happy to review a PR where you propose some concrete change

@yrong
Copy link
Contributor Author

yrong commented Sep 26, 2025

In any of those cases the hash of the token-location will not be the same anymore. meaning that you lose the opportunity to transfer the tokens to ethereum.

The Token ID (in V4) ↔ Address mapping is already persisted on Ethereum. Since V5 is fully compatible with V4, this should be fine.

If any of the XCM primitives used to generate the ID were to change, as you mentioned, we could consider performing a storage migration in both the System Pallet and on the Ethereum side to ensure both ID formats are recognized and map to the same token address.

That said, I strongly recommend avoiding this if possible, for the reasons Adrian mentioned.

@girazoki Besides that, I’m also curious — did you encounter any specific issues, or are there changes you want to make that the current implementation is preventing?

@girazoki
Copy link
Contributor

girazoki commented Sep 29, 2025

I am going to put a few examples in which things can go wrong:
you somehow update any of the implementations that are part of tokenIdOf (e.g. describeFamily, DescribeGlobalPrefix, DescribeGlobalPrefix).
you somehow update the implementation of HashedDescription
you somehow update your token location (e.g., pallet-assets references now with a different junction the assetId)

all of the above are deeply destructive, breaking changes :) if any of them is attempted maaany things will break, not just this ethereum token id conversion (think thousands of derived accounts, derived locations, and all the hardcoded stuff off-chain)

having said that, I am not against making tokenIdOf more flexible - I am happy to review a PR where you propose some concrete change

I agree that those are destructive changes. However they are no the first time that I see them :). the location change for assethub assets already happened in the past, possibly with the system not being mature enough and therefore without too many catastrophic consequences. But my point is that with the current mechanism in place, those destructive changes are out of the table.

In the case of EthereumSystem the flexibility I would believe it to be easy to add, but it would involve bringing the NativeToForeignId storage back with its corresponding migration. In short, my proposal is:

I am happy to implement these changes if you guys agree. The point of them would be to tie the fate of the location to token-id to whatever ethereumSystem dictates, and not to something constant without the possibility of changing it

@girazoki
Copy link
Contributor

girazoki commented Oct 1, 2025

I am going to put a few examples in which things can go wrong:
you somehow update any of the implementations that are part of tokenIdOf (e.g. describeFamily, DescribeGlobalPrefix, DescribeGlobalPrefix).
you somehow update the implementation of HashedDescription
you somehow update your token location (e.g., pallet-assets references now with a different junction the assetId)

all of the above are deeply destructive, breaking changes :) if any of them is attempted maaany things will break, not just this ethereum token id conversion (think thousands of derived accounts, derived locations, and all the hardcoded stuff off-chain)
having said that, I am not against making tokenIdOf more flexible - I am happy to review a PR where you propose some concrete change

I agree that those are destructive changes. However they are no the first time that I see them :). the location change for assethub assets already happened in the past, possibly with the system not being mature enough and therefore without too many catastrophic consequences. But my point is that with the current mechanism in place, those destructive changes are out of the table.

In the case of EthereumSystem the flexibility I would believe it to be easy to add, but it would involve bringing the NativeToForeignId storage back with its corresponding migration. In short, my proposal is:

I am happy to implement these changes if you guys agree. The point of them would be to tie the fate of the location to token-id to whatever ethereumSystem dictates, and not to something constant without the possibility of changing it

@acatangiu @yrong how does this sound? do u want me to open a draft PR (without the migration) so that u guys see what it looks like?

@yrong
Copy link
Contributor Author

yrong commented Oct 1, 2025

how does this sound? do u want me to open a draft PR (without the migration) so that u guys see what it looks like?

Cool. I’m currently on vacation and will be back early next week. I’ll review it then.

alvicsam pushed a commit that referenced this pull request Oct 17, 2025
The `TokenIdOf`
[convert](https://github.com/paritytech/polkadot-sdk/blob/4b83d24f4bc96a7b17964be94b178dd7b8f873b5/bridges/snowbridge/primitives/core/src/location.rs#L40)
is XCM version-agnostic, meaning we will get the same token ID for both
V5 and legacy V4 asset.

However, the extra check is unnecessary, as
the`ConvertAssetId::convert(&token_id).ok_or(InvalidAsset)?;` alone is
sufficient to verify whether the token is registered.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A4-backport-stable2503 Pull request must be backported to the stable2503 release branch T15-bridges This PR/Issue is related to bridges.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants