Strip metadata fix #1774
Strip metadata fix #1774
Conversation
metadata/src/utils/retain.rs
Outdated
| } | ||
| } | ||
|
|
||
| fn collect_types(&mut self, metadata: &Metadata, t: &PortableType) { |
There was a problem hiding this comment.
So this logic is basically recursing through all of the types linked from some type and adding them to the set. But this is the same thing that we do in PortableRegistry::retain here https://github.com/paritytech/scale-info/blob/master/src/portable.rs#L104; is it necessary?
There was a problem hiding this comment.
Yup, retain is called dead last before updating types, but we need to collect types ourselves to determine which outer type variants are safe to drop/which pallets can be stripped or dropped completetely.
There was a problem hiding this comment.
I'm not sure how I feel about this function, because it duplicates the logic in scale_info's retain.
a) On the one hand, we could also remove this function entirely (ie no longer recurse and building this type Ids map when we collect up our pallet/runtime etc types), and instead have a function which has the specific goal of searching through the metadata to look for the runtime types (and in that way, it doesn't need to allocate/collect anything).
b) On the other hand, keeping this function is "easier" because you need to recurse anyway to do the above searching, so maybe you end up with fewer lines of code by just keeping it around.
Personally I'd try (a) and see whether doing so makes things easier to follow or not, but not sure!
metadata/src/utils/retain.rs
Outdated
| || !matches!( | ||
| pallet, | ||
| PalletMetadataInner { | ||
| storage: None, | ||
| call_ty: None, | ||
| event_ty: None, | ||
| error_ty: None, | ||
| constants: map, | ||
| .. | ||
| } if map.is_empty() |
There was a problem hiding this comment.
Does this make any difference? I can't see why a pallet would exist that had nothing meaningful in it, but even if it did, is there any harm in leaving it there?
There was a problem hiding this comment.
Ah I am starting to see why this is kept but not understanding it yet; you are turning things to None above in some cases
| @@ -588,7 +584,7 @@ impl<'a> MetadataHasher<'a> { | |||
|
|
|||
| // Get the hashes of outer enums, considering only `specific_pallets` (if any are set). | |||
| // If any of the typed that represent outer enums are encountered later, hashes from `top_level_enum_hashes` can be substituted. | |||
| let outer_enum_hashes = OuterEnumHashes::new(metadata, self.specific_pallets.as_deref()); | |||
| let outer_enum_hashes = OuterEnumHashes::new(metadata); | |||
There was a problem hiding this comment.
So here we no logner care about the possibility of the enums like RuntimeCall etc being changed, but how can we avoid caring about this?
(either the variants and all of their types still exist, so the hashes are the same, or we have stripped some types and/or variants and then the hashes owuld be different unless we ignore them, wouldn't they?)
There was a problem hiding this comment.
list of pallets/variants types is completely dependant on the given set of pallets for a particular pallet or unpredictable
There was a problem hiding this comment.
I don't understand,
So what happens if Call::Balances(pallet_system::AccountData) variant exist and I only want retain the system pallet, does the Call::Balances get filtered out?
There was a problem hiding this comment.
Aight, I think this was opinionated and removed variants without checking that it was actually referenced by pallets, runtime apis?
We now only rely on TypeSet::retain_metadata(..) to achieve this
metadata/src/utils/retain.rs
Outdated
| fn collect_extrinsic_types(&mut self, extrinsic: &ExtrinsicMetadata) { | ||
| let mut ids = Vec::from([ | ||
| extrinsic.address_ty, | ||
| extrinsic.call_ty, | ||
| extrinsic.signature_ty, | ||
| extrinsic.extra_ty, | ||
| ]); | ||
|
|
||
| for signed in &extrinsic.signed_extensions { | ||
| ids.push(signed.extra_ty); | ||
| ids.push(signed.additional_ty); | ||
| } | ||
| for id in ids { | ||
| self.seen_ids.insert(id); | ||
| } | ||
| } | ||
|
|
||
| if let Some(ty) = pallet.event_ty { | ||
| type_ids.insert(ty); | ||
| /// Collect all type IDs needed to represent the runtime APIs. | ||
| fn collect_runtime_api_types(&mut self, api: &RuntimeApiMetadataInner) { | ||
| for method in api.methods.values() { | ||
| for input in &method.inputs { | ||
| self.seen_ids.insert(input.ty); | ||
| } | ||
| self.seen_ids.insert(method.output_ty); | ||
| } | ||
| } |
There was a problem hiding this comment.
These two functions don't recurse through any sub types. While I don't think anything should, if we did decide we wanted to recurse here then I guess these should, too?
There was a problem hiding this comment.
pushed a change that does traverse, a few more types added to retained set on average.
metadata/src/utils/retain.rs
Outdated
| let mut retained_set = TypeSet { | ||
| seen_ids: type_set | ||
| .seen_ids | ||
| .difference(&retained_set.seen_ids) | ||
| .copied() | ||
| .collect(), | ||
| }; |
There was a problem hiding this comment.
So it looks like you are:
- building a set of types to keep based on the pallets we want to keep (
retained_set) - cloning that set and then adding all of the other types we want to keep (runtime APIs etc) too (
type_set) - taking the difference between those (ie
type_set - retained_set? So just the other types without the pallet ones, which we could get by just not cloning above?). This difference is nowretained_set. - for pallets we want to filter, run
update_filtered_pallet, which roughly seems to keep things in the pallet if they reference types we've collected outside of the pallet stuff, and remove things if not. We then only discard pallets if no types that they directly reference are used elsewhere outside of pallets.
I might be wrong, but I don't really understand the reason for doing this? since you collected the outer enums, I can see that the first pallet we see that references those would actually be kept (which I guess has the effect of keeping the variant too?), but it feels like quite a long winded way to go about avoiding getting rid of variants if the outer enums are used somewhere, and I don't know if there is any other benefit :D
I also see that update_filtered_pallet calls retained_set.remove() each time it checks, so only the first pallet referencing eg RuntimeCall would end up having bits kept I think, and other pallets that reference RuntimeCall would be removed. Is that intentional?
Overall, I feel like we can remove a lot of this logic and rely on scale_info::PortableRegistry::retain for the recursive retaining of IDs we care about (because either that works ok already, or doesn't work and should be fixed). Then the logic can be something like:
- Add all of the type IDs from the pallets etc that we want to keep to a set
- For any return types or storage types in these things we are keeping, do a deep scan to see if we find the
RuntimeCall,RutimeEventorRuntimeErrorouter enums. If we find one, we can't strip any variants out of it - Now, call
PortableRegistry::retainto keep all of the type Ids we need (which will vary depending on whether those enums had variants stripped out of them or not and thus fewer IDs to recurse through).
All of that said though, if there is some value I'm not understanding in the way that we are keeping pallets etc here, then probably I'd need to be walked through it and we'd want to add a comment or two somewhere here to explain the approach and reasoning for it for our future selves :)
There was a problem hiding this comment.
taking the difference between those (ie type_set - retained_set? So just the other types without the pallet ones, which we could get by just not cloning above?). This difference is now retained_set.
The difference is the types present in the type_set and absent in retained_set. retained_set is not interesting in this case as types were already present in retained pallet. But i feel like we are talking about the same thing
but it feels like quite a long winded way to go about avoiding getting rid of variants if the outer enums are used somewhere, and I don't know if there is any other benefit :D
Its more about getting rid of pallets and outer enums, but only if they are not referenced anywhere. Its about stripping types that are not interesting to user without accidentally removing something that is required.
I also see that update_filtered_pallet calls retained_set.remove() each time it checks, so only the first pallet referencing eg RuntimeCall would end up having bits kept I think, and other pallets that reference RuntimeCall would be removed. Is that intentional?
I guess i'm misunderstanding something, shouldn't only one pallet with the type be enough or we can safely ignore those refs to runtime call/runtime error/runtime event and remove pallet based purely on storage/constant types? My intuition was to keep them as they might be referencing inner types inside outerenum variant, ie pallet.event_ty = 365 and metadata.outer_enum.event_ty = 22
All of that said though, if there is some value I'm not understanding in the way that we are keeping pallets etc here, then probably I'd need to be walked through it and we'd want to add a comment or two somewhere here to explain the approach and reasoning for it for our future selves :)
as above: My intuition was to keep them as they might be referencing inner types inside outerenum variant, ie pallet.event_ty = 365 and metadata.outer_enum.event_ty = 22
|
I had a look over and think I'm starting to understand the code more, though I still think that it's more complex than we need (but let's chat tomorrow and look at the code more to confirm). Essentially, So, the approach I'd lean towards with this is more like:
For working out whether some type contains an outer enum, you might end up with a recursive check which is a simplified version of yours, and doesn't need to allocate anything like: fn find_tys_in_id(id: u32, types: &PortableRegistry, unseen: &mut BTreeSet<u32>) {
unseen.remove(id)
let ty = types.resolve(id);
match ty.type_def {
TypeDef::Composite(TypeDefComposite { fields }) => {
for field in fields {
find_tys_in_id(field.ty.id, types, unseen)
}
}
TypeDef::Variant(TypeDefVariant { variants }) => {
for variant in variants {
for field in &variant.fields {
find_tys_in_id(field.ty.id, types, unseen)
}
}
}
TypeDef::Array(TypeDefArray { len: _, type_param })
| TypeDef::Sequence(TypeDefSequence { type_param })
| TypeDef::Compact(TypeDefCompact { type_param }) => {
find_tys_in_id(type_param.id, types, unseen)
}
TypeDef::Tuple(TypeDefTuple { fields }) => {
for field in fields {
find_tys_in_id(field.id, types, unseen)
}
}
TypeDef::Primitive(_) | TypeDef::BitSequence(_) => {
// End of recursion.
}
}
}I def think that the existing code can be made clearer w.r.t what steps it takes, and hopefully this extra check to see if we can strip outer enum types handles the edge case we've faced where we strip variants even though we need the fully types to decode something in Subxt. Note: If the outer enum types are inputs to some runtime API etc, we can still strip them I think. All this means is that users won't be able to express things about pallets that have been stripped when constructing such inputs. Everything should still encode ok. |
ok, so after fiddling this a bit more i think this suggested approach is the way to go. click me, wall of text`pallet_filter` - the pallet we want to keep. `name` - approach to traversal that was taken. `count` - amount of types retained in the registry after stripping.as visible from set below its the Note however that amount of types on average will increase compared to the old approach, e.g. for a dataset of 943 types it will be 644 instead of the 411 in the past. [
Collected {
pallets_filter: "System",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 581,
},
Info {
name: "traverse pallets",
count: 644,
},
Info {
name: "traverse extrinsics",
count: 581,
},
Info {
name: "traverse pallets & extrinsics",
count: 644,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 644,
},
],
},
Collected {
pallets_filter: "Babe",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 575,
},
Info {
name: "traverse pallets",
count: 575,
},
Info {
name: "traverse extrinsics",
count: 575,
},
Info {
name: "traverse pallets & extrinsics",
count: 575,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 640,
},
],
},
Collected {
pallets_filter: "Timestamp",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 562,
},
Info {
name: "traverse pallets",
count: 562,
},
Info {
name: "traverse extrinsics",
count: 562,
},
Info {
name: "traverse pallets & extrinsics",
count: 562,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "Indices",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 565,
},
Info {
name: "traverse pallets",
count: 565,
},
Info {
name: "traverse extrinsics",
count: 565,
},
Info {
name: "traverse pallets & extrinsics",
count: 565,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 629,
},
],
},
Collected {
pallets_filter: "Balances",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 584,
},
Info {
name: "traverse pallets",
count: 584,
},
Info {
name: "traverse extrinsics",
count: 584,
},
Info {
name: "traverse pallets & extrinsics",
count: 584,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 647,
},
],
},
Collected {
pallets_filter: "Parameters",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 570,
},
Info {
name: "traverse pallets",
count: 570,
},
Info {
name: "traverse extrinsics",
count: 570,
},
Info {
name: "traverse pallets & extrinsics",
count: 570,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "TransactionPayment",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "Authorship",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 562,
},
Info {
name: "traverse pallets",
count: 562,
},
Info {
name: "traverse extrinsics",
count: 562,
},
Info {
name: "traverse pallets & extrinsics",
count: 562,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "Offences",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 566,
},
Info {
name: "traverse pallets",
count: 566,
},
Info {
name: "traverse extrinsics",
count: 566,
},
Info {
name: "traverse pallets & extrinsics",
count: 566,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 630,
},
],
},
Collected {
pallets_filter: "Historical",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 563,
},
Info {
name: "traverse pallets",
count: 563,
},
Info {
name: "traverse extrinsics",
count: 563,
},
Info {
name: "traverse pallets & extrinsics",
count: 563,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "Session",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 567,
},
Info {
name: "traverse pallets",
count: 567,
},
Info {
name: "traverse extrinsics",
count: 567,
},
Info {
name: "traverse pallets & extrinsics",
count: 567,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 631,
},
],
},
Collected {
pallets_filter: "Grandpa",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 567,
},
Info {
name: "traverse pallets",
count: 567,
},
Info {
name: "traverse extrinsics",
count: 567,
},
Info {
name: "traverse pallets & extrinsics",
count: 567,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 631,
},
],
},
Collected {
pallets_filter: "AuthorityDiscovery",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 563,
},
Info {
name: "traverse pallets",
count: 563,
},
Info {
name: "traverse extrinsics",
count: 563,
},
Info {
name: "traverse pallets & extrinsics",
count: 563,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "Treasury",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 570,
},
Info {
name: "traverse pallets",
count: 570,
},
Info {
name: "traverse extrinsics",
count: 570,
},
Info {
name: "traverse pallets & extrinsics",
count: 570,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 634,
},
],
},
Collected {
pallets_filter: "ConvictionVoting",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 576,
},
Info {
name: "traverse pallets",
count: 576,
},
Info {
name: "traverse extrinsics",
count: 576,
},
Info {
name: "traverse pallets & extrinsics",
count: 576,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 640,
},
],
},
Collected {
pallets_filter: "Referenda",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 582,
},
Info {
name: "traverse pallets",
count: 582,
},
Info {
name: "traverse extrinsics",
count: 582,
},
Info {
name: "traverse pallets & extrinsics",
count: 582,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 645,
},
],
},
Collected {
pallets_filter: "FellowshipCollective",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 571,
},
Info {
name: "traverse pallets",
count: 571,
},
Info {
name: "traverse extrinsics",
count: 571,
},
Info {
name: "traverse pallets & extrinsics",
count: 571,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 633,
},
],
},
Collected {
pallets_filter: "FellowshipReferenda",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 581,
},
Info {
name: "traverse pallets",
count: 581,
},
Info {
name: "traverse extrinsics",
count: 581,
},
Info {
name: "traverse pallets & extrinsics",
count: 581,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 644,
},
],
},
Collected {
pallets_filter: "Origins",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 562,
},
Info {
name: "traverse pallets",
count: 562,
},
Info {
name: "traverse extrinsics",
count: 562,
},
Info {
name: "traverse pallets & extrinsics",
count: 562,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "Whitelist",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "Claims",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "Utility",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "Identity",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 579,
},
Info {
name: "traverse pallets",
count: 579,
},
Info {
name: "traverse extrinsics",
count: 579,
},
Info {
name: "traverse pallets & extrinsics",
count: 579,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 643,
},
],
},
Collected {
pallets_filter: "Society",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 585,
},
Info {
name: "traverse pallets",
count: 585,
},
Info {
name: "traverse extrinsics",
count: 585,
},
Info {
name: "traverse pallets & extrinsics",
count: 585,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 648,
},
],
},
Collected {
pallets_filter: "Recovery",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 568,
},
Info {
name: "traverse pallets",
count: 568,
},
Info {
name: "traverse extrinsics",
count: 568,
},
Info {
name: "traverse pallets & extrinsics",
count: 568,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 632,
},
],
},
Collected {
pallets_filter: "Vesting",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 567,
},
Info {
name: "traverse pallets",
count: 567,
},
Info {
name: "traverse extrinsics",
count: 567,
},
Info {
name: "traverse pallets & extrinsics",
count: 567,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 631,
},
],
},
Collected {
pallets_filter: "Scheduler",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 569,
},
Info {
name: "traverse pallets",
count: 569,
},
Info {
name: "traverse extrinsics",
count: 569,
},
Info {
name: "traverse pallets & extrinsics",
count: 569,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 633,
},
],
},
Collected {
pallets_filter: "Proxy",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 572,
},
Info {
name: "traverse pallets",
count: 572,
},
Info {
name: "traverse extrinsics",
count: 572,
},
Info {
name: "traverse pallets & extrinsics",
count: 572,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 636,
},
],
},
Collected {
pallets_filter: "Multisig",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 567,
},
Info {
name: "traverse pallets",
count: 567,
},
Info {
name: "traverse extrinsics",
count: 567,
},
Info {
name: "traverse pallets & extrinsics",
count: 567,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 631,
},
],
},
Collected {
pallets_filter: "Preimage",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 573,
},
Info {
name: "traverse pallets",
count: 573,
},
Info {
name: "traverse extrinsics",
count: 573,
},
Info {
name: "traverse pallets & extrinsics",
count: 573,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 637,
},
],
},
Collected {
pallets_filter: "AssetRate",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "Bounties",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 569,
},
Info {
name: "traverse pallets",
count: 569,
},
Info {
name: "traverse extrinsics",
count: 569,
},
Info {
name: "traverse pallets & extrinsics",
count: 569,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 633,
},
],
},
Collected {
pallets_filter: "ChildBounties",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 567,
},
Info {
name: "traverse pallets",
count: 567,
},
Info {
name: "traverse extrinsics",
count: 567,
},
Info {
name: "traverse pallets & extrinsics",
count: 567,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 631,
},
],
},
Collected {
pallets_filter: "Nis",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 576,
},
Info {
name: "traverse pallets",
count: 576,
},
Info {
name: "traverse extrinsics",
count: 576,
},
Info {
name: "traverse pallets & extrinsics",
count: 576,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 640,
},
],
},
Collected {
pallets_filter: "NisCounterpartBalances",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 584,
},
Info {
name: "traverse pallets",
count: 584,
},
Info {
name: "traverse extrinsics",
count: 584,
},
Info {
name: "traverse pallets & extrinsics",
count: 584,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 647,
},
],
},
Collected {
pallets_filter: "ParachainsOrigin",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 562,
},
Info {
name: "traverse pallets",
count: 562,
},
Info {
name: "traverse extrinsics",
count: 562,
},
Info {
name: "traverse pallets & extrinsics",
count: 562,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "Configuration",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 566,
},
Info {
name: "traverse pallets",
count: 566,
},
Info {
name: "traverse extrinsics",
count: 566,
},
Info {
name: "traverse pallets & extrinsics",
count: 566,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 631,
},
],
},
Collected {
pallets_filter: "ParasShared",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 565,
},
Info {
name: "traverse pallets",
count: 565,
},
Info {
name: "traverse extrinsics",
count: 565,
},
Info {
name: "traverse pallets & extrinsics",
count: 565,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 630,
},
],
},
Collected {
pallets_filter: "ParaInclusion",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 566,
},
Info {
name: "traverse pallets",
count: 566,
},
Info {
name: "traverse extrinsics",
count: 566,
},
Info {
name: "traverse pallets & extrinsics",
count: 566,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 630,
},
],
},
Collected {
pallets_filter: "ParaInherent",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 563,
},
Info {
name: "traverse pallets",
count: 563,
},
Info {
name: "traverse extrinsics",
count: 563,
},
Info {
name: "traverse pallets & extrinsics",
count: 563,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "ParaScheduler",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 570,
},
Info {
name: "traverse pallets",
count: 570,
},
Info {
name: "traverse extrinsics",
count: 570,
},
Info {
name: "traverse pallets & extrinsics",
count: 570,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 635,
},
],
},
Collected {
pallets_filter: "Paras",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 575,
},
Info {
name: "traverse pallets",
count: 575,
},
Info {
name: "traverse extrinsics",
count: 575,
},
Info {
name: "traverse pallets & extrinsics",
count: 575,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 639,
},
],
},
Collected {
pallets_filter: "Initializer",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 629,
},
],
},
Collected {
pallets_filter: "Dmp",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 562,
},
Info {
name: "traverse pallets",
count: 562,
},
Info {
name: "traverse extrinsics",
count: 562,
},
Info {
name: "traverse pallets & extrinsics",
count: 562,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "Hrmp",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 569,
},
Info {
name: "traverse pallets",
count: 569,
},
Info {
name: "traverse extrinsics",
count: 569,
},
Info {
name: "traverse pallets & extrinsics",
count: 569,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 633,
},
],
},
Collected {
pallets_filter: "ParaSessionInfo",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 562,
},
Info {
name: "traverse pallets",
count: 562,
},
Info {
name: "traverse extrinsics",
count: 562,
},
Info {
name: "traverse pallets & extrinsics",
count: 562,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "ParasDisputes",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 568,
},
Info {
name: "traverse pallets",
count: 568,
},
Info {
name: "traverse extrinsics",
count: 568,
},
Info {
name: "traverse pallets & extrinsics",
count: 568,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 630,
},
],
},
Collected {
pallets_filter: "ParasSlashing",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 629,
},
],
},
Collected {
pallets_filter: "MessageQueue",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 571,
},
Info {
name: "traverse pallets",
count: 571,
},
Info {
name: "traverse extrinsics",
count: 571,
},
Info {
name: "traverse pallets & extrinsics",
count: 571,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 634,
},
],
},
Collected {
pallets_filter: "OnDemandAssignmentProvider",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 576,
},
Info {
name: "traverse pallets",
count: 576,
},
Info {
name: "traverse extrinsics",
count: 576,
},
Info {
name: "traverse pallets & extrinsics",
count: 576,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 640,
},
],
},
Collected {
pallets_filter: "CoretimeAssignmentProvider",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 573,
},
Info {
name: "traverse pallets",
count: 573,
},
Info {
name: "traverse extrinsics",
count: 573,
},
Info {
name: "traverse pallets & extrinsics",
count: 573,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 638,
},
],
},
Collected {
pallets_filter: "Registrar",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 566,
},
Info {
name: "traverse pallets",
count: 566,
},
Info {
name: "traverse extrinsics",
count: 566,
},
Info {
name: "traverse pallets & extrinsics",
count: 566,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 630,
},
],
},
Collected {
pallets_filter: "Slots",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 567,
},
Info {
name: "traverse pallets",
count: 567,
},
Info {
name: "traverse extrinsics",
count: 567,
},
Info {
name: "traverse pallets & extrinsics",
count: 567,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 631,
},
],
},
Collected {
pallets_filter: "Auctions",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 568,
},
Info {
name: "traverse pallets",
count: 568,
},
Info {
name: "traverse extrinsics",
count: 568,
},
Info {
name: "traverse pallets & extrinsics",
count: 568,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 632,
},
],
},
Collected {
pallets_filter: "Crowdloan",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 567,
},
Info {
name: "traverse pallets",
count: 567,
},
Info {
name: "traverse extrinsics",
count: 567,
},
Info {
name: "traverse pallets & extrinsics",
count: 567,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 631,
},
],
},
Collected {
pallets_filter: "Coretime",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "XcmPallet",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 583,
},
Info {
name: "traverse pallets",
count: 583,
},
Info {
name: "traverse extrinsics",
count: 583,
},
Info {
name: "traverse pallets & extrinsics",
count: 583,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 647,
},
],
},
Collected {
pallets_filter: "Beefy",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 629,
},
],
},
Collected {
pallets_filter: "Mmr",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 562,
},
Info {
name: "traverse pallets",
count: 562,
},
Info {
name: "traverse extrinsics",
count: 562,
},
Info {
name: "traverse pallets & extrinsics",
count: 562,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "MmrLeaf",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 562,
},
Info {
name: "traverse pallets",
count: 562,
},
Info {
name: "traverse extrinsics",
count: 562,
},
Info {
name: "traverse pallets & extrinsics",
count: 562,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "IdentityMigrator",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 563,
},
Info {
name: "traverse pallets",
count: 563,
},
Info {
name: "traverse extrinsics",
count: 563,
},
Info {
name: "traverse pallets & extrinsics",
count: 563,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "ParasSudoWrapper",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 563,
},
Info {
name: "traverse pallets",
count: 563,
},
Info {
name: "traverse extrinsics",
count: 563,
},
Info {
name: "traverse pallets & extrinsics",
count: 563,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
Collected {
pallets_filter: "AssignedSlots",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 565,
},
Info {
name: "traverse pallets",
count: 565,
},
Info {
name: "traverse extrinsics",
count: 565,
},
Info {
name: "traverse pallets & extrinsics",
count: 565,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 629,
},
],
},
Collected {
pallets_filter: "ValidatorManager",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 563,
},
Info {
name: "traverse pallets",
count: 563,
},
Info {
name: "traverse extrinsics",
count: 563,
},
Info {
name: "traverse pallets & extrinsics",
count: 563,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "StateTrieMigration",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 565,
},
Info {
name: "traverse pallets",
count: 565,
},
Info {
name: "traverse extrinsics",
count: 565,
},
Info {
name: "traverse pallets & extrinsics",
count: 565,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "RootTesting",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 563,
},
Info {
name: "traverse pallets",
count: 563,
},
Info {
name: "traverse extrinsics",
count: 563,
},
Info {
name: "traverse pallets & extrinsics",
count: 563,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 627,
},
],
},
Collected {
pallets_filter: "Sudo",
results: [
Info {
name: "no stripping",
count: 943,
},
Info {
name: "no deep traversal",
count: 564,
},
Info {
name: "traverse pallets",
count: 564,
},
Info {
name: "traverse extrinsics",
count: 564,
},
Info {
name: "traverse pallets & extrinsics",
count: 564,
},
Info {
name: "traverse pallets, extrinsics & runtime",
count: 628,
},
],
},
] |
| @@ -214,7 +214,10 @@ impl Metadata { | |||
| MetadataHasher::new(self) | |||
| } | |||
|
|
|||
| /// Filter out any pallets that we don't want to keep, retaining only those that we do. | |||
| /// Filter out any pallets and/or runtime_apis that we don't want to keep, retaining only those that we do. | |||
| /// Note: | |||
There was a problem hiding this comment.
I would rather create a dev-note using // to avoid this to end-up in the API docs
metadata/src/lib.rs
Outdated
| /// Filter out any pallets that we don't want to keep, retaining only those that we do. | ||
| /// Filter out any pallets and/or runtime_apis that we don't want to keep, retaining only those that we do. | ||
| /// Note: | ||
| /// without also filtering out irrelevant runtime_api's in addition to pallets one should not expect significant metadata size reduction. |
There was a problem hiding this comment.
| /// without also filtering out irrelevant runtime_api's in addition to pallets one should not expect significant metadata size reduction. | |
| /// runtime api filtering will not lead to significant metadata size reduction because the return types are kept to ensure that those can be decoded. |
There was a problem hiding this comment.
runtimeApi filtering will lead to significant metadata size reductions.
If a runtimeApi returns an outerEnum type we no longer can strip the enum, because we don't know which variants are used.
metadata/src/lib.rs
Outdated
| /// Filter out any pallets and/or runtime_apis that we don't want to keep, retaining only those that we do. | ||
| /// Note: | ||
| /// without also filtering out irrelevant runtime_api's in addition to pallets one should not expect significant metadata size reduction. | ||
| /// This happens because we will keep the types returned by runtime_api methods to guarantee that we can safely decode them. |
There was a problem hiding this comment.
why do we need to keep the return types if those are not used/trimmed?
There was a problem hiding this comment.
we can never know if they are used fully or partially, therefore we always keep the full type
| match entry.entry_type { | ||
| StorageEntryType::Plain(ty) => { | ||
| type_ids.insert(ty); | ||
| use alloc::collections::{BTreeMap, BTreeSet, VecDeque}; |
There was a problem hiding this comment.
Do you mind document the algorithm we use to filter out pallets/runtime APIs as top-level docs in this module?
Such as the tree/types, outer enums etc are traversed and in which order...
metadata/src/utils/retain.rs
Outdated
| }; | ||
|
|
||
| // If the type was not referenced earlier we can safely strip some of the variants | ||
| if self.seen_ids.insert(id) { |
There was a problem hiding this comment.
| if self.seen_ids.insert(id) { | |
| if self.seen_ids.contains(id) { |
Inserting makes the result of calling collect_variants_in_type depend on the order in which you call it. So I wonder whether there is a need to insert here?
There was a problem hiding this comment.
I'm not sure i'm following, we still need to add the enum type to the set and we don't do a deep traversal for outerenums anyways so insertion order shouldn't matter
There was a problem hiding this comment.
Maybe this is one of the things I find harder to follow; I guess I want to separate the funciton that "determines whether to keep all of the variants" from functions that mutate and add type IDs to some set.
| }; | ||
|
|
||
| #[derive(Clone)] | ||
| struct TypeSet { |
There was a problem hiding this comment.
Personally I'd likely make the functions impl'd on this free standing and just have a. BTreeSet passed about, primarily for the reason that despite this TypeSet struct, there are still places where you manually do type_set.seen_ids.insert(foo) or whatever, so the internal details are leaking anyway :)
There was a problem hiding this comment.
removed those parts to use defined method on the struct. i moved those functions an impl block to not carry the set around :D
| @@ -281,7 +358,6 @@ pub fn retain_metadata<F, G>( | |||
| mod tests { | |||
There was a problem hiding this comment.
Is there any test that we can add which ensures we don't strip outer enum variants when it would lead to breakage (ie they are used somewhere)?
There was a problem hiding this comment.
added a test that references initial issue
metadata/src/utils/validation.rs
Outdated
| // test that the hashes are the same: | ||
| let hash = MetadataHasher::new(&metadata) | ||
| .only_these_pallets(&["First"]) | ||
| .only_these_pallets(&name_set) |
There was a problem hiding this comment.
I guess we can revert this to "First" now to simplify?
There was a problem hiding this comment.
removed the test completetely as the behaviour in OuterEnumHashes relied on the fact that some of the variants in outernums are removed unconditionally.
| { | ||
| let set = "Balances,Timestamp,Contracts,ContractsEvm,System" | ||
| .split(",") | ||
| .collect::<BTreeSet<&str>>(); | ||
| move |s| set.contains(&s) | ||
| }, |
There was a problem hiding this comment.
I'd simplify this to just |s| ["Balances","Timestamp","Contracts","ContractsEVM","System"].contains(s); no need for the set etc really (and with so few items it's prob not even slower anyways, but being a test it's not a big deal)
There was a problem hiding this comment.
i written it like this only because repro command is a string and i was lazy to type
metadata/src/utils/retain.rs
Outdated
|
|
||
| #[test] | ||
| fn issue_1659() { | ||
| let metadata_cache = load_metadata("../artifacts/regressions/1659.scale"); |
There was a problem hiding this comment.
Nit: rename to full_metadata?
There was a problem hiding this comment.
which full_metadata? maybe we can name it 1659.full_metadata.scale?
There was a problem hiding this comment.
Sorry, I wasn't clear at all! I meant to rename metadata_cache to full_metadata :)
metadata/src/utils/retain.rs
Outdated
| } | ||
| } | ||
|
|
||
| #[test] | ||
| fn retain_one_runtime_api() { | ||
| let metadata_cache = load_metadata(); | ||
| let metadata_cache = load_metadata_polkadot(); |
There was a problem hiding this comment.
Nit: I'd call this just metadata too (not really this PR but hey :))
There was a problem hiding this comment.
renamed load_metadata_polkadot to load_metadata and introduced load_metadata_custom to load different files
There was a problem hiding this comment.
Sorry, I wasn't clear at all there! I was referring to the variable metadata_cache; it's not really a cache or anything ,it is just the metadata :)
|
One of the things that was bugging me a bit in this code prior to this PR is that we have separate Something that I find a bit hard to do still in this PR is following exactly the steps that we take to strip types. I think this was also a bit messy before the PR too. Anyway, I got to wondering whether something like the following would be ultimately much clearer and easier to follow, while avoiding duplicating the logc around collect vs update: pub fn retain_metadata<F, G>(
metadata: &mut Metadata,
mut pallets_filter: F,
mut runtime_apis_filter: G,
) where
F: FnMut(&str) -> bool,
G: FnMut(&str) -> bool,
{
// 1. Delete pallets we don't want to keep.
metadata.pallets.retain(|pallet| pallets_filter(&pallet.name));
metadata.pallets_by_index = metadata
.pallets
.values()
.iter()
.enumerate()
.map(|(pos, p)| (p.index, pos))
.collect();
// 2. Delete runtime APIs we don't want to keep.
metadata.apis.retain(|api| runtime_apis_filter(&api.name));
// 3. For each outer enum type, strip it if possible, ie if it is not returned by any
// of the things we're keeping (because if it is, we need to keep all of it so that we
// can still decode values into it).
let outer_enums = metadata.outer_enums();
for outer_enum_ty_id in [outer_enums.call_enum_ty(), outer_enums.error_enum_ty(), outer_enums.event_enum_ty()] {
if !find_type_id_in_return_pos(metadata, outer_enum_ty_id) {
retain_variants_in_enum_type(metadata, &pallets_filter);
// ^ prob clearer renamed to "strip_variants_in_enum_type"
}
}
// 4. Collect all of the type IDs we still want to keep after deleting.
let mut keep_these_type_ids: BTreeSet<u32> = iterate_metadata_types(metadata).map(|id| *id).collect();
// 5. Additionally, subxt depends on the `DispatchError` type existing; we use the same
// logic here that is used when building our `Metadata` to ensure we keep it too.
let dispatch_error_ty = metadata
.types
.types
.iter()
.find(|ty| ty.ty.path.segments == ["sp_runtime", "DispatchError"])
.expect("Metadata must contain sp_runtime::DispatchError");
keep_these_type_ids.insert(dispatch_error_ty.id);
// 5. Strip all of the type IDs we no longer need, based on the above set.
let map_ids = metadata.types.retain(|id| keep_these_type_ids.contains(&id));
// 6. Now, update the type IDs referenced in our metadata to reflect this.
for id in iterate_metadata_types(metadata) {
if let Some(new_id) = map_ids.get(id) {
*id = *new_id;
} else {
panic!("Type id {id} was not retained. This is a bug");
}
}
}
/// return an iterator handing back a mutable reference to each type ID seen in the metadata (not recursively).
/// This will iterate over every type referenced in the metadata outside of `metadata.types`.
fn iterate_metadata_types(metadata: &mut Metadata) -> impl Iterator<Item = &mut u32> {
unimplemented!()
}
/// Look for a type ID anywhere that we can be given back, ie in constants, storage or runtime API return types.
/// This will recurse deeply into those type IDs to find them (but doesn't need to allocate).
fn find_type_id_in_return_pos(metadata: &mut Metadata, type_id: u32) -> bool {
unimplemented!()
}I think this would be something like my "north star" for a readable and hopefullyfairly easy to follow implementation of stripping pallets/runtime APIs. That all said, I think we can aim to get this PR merged and havea go as a follow up step to do something like this. |
| // If any of the typed that represent outer enums are encountered later, hashes from `top_level_enum_hashes` can be substituted. | ||
| let outer_enum_hashes = OuterEnumHashes::new(metadata, self.specific_pallets.as_deref()); | ||
| let outer_enum_hashes = OuterEnumHashes::new(metadata); | ||
|
|
There was a problem hiding this comment.
I'm still a bit confused by this.
Basically, the case I am thinking about is if we want to compare some "full" metadata with some "stripped down" metadata. We want to ensure that the remaining parts of the stripped metadata are in line with this full metadata (which would come form a node at runtime most probably).
To generate the same hash as the stripped metadata, we need to hash only the pallets that were kept around when hashing the full metadata.
To do this, we would previously generate outer_enum_hashes which hashed only the variants for those specific pallets (because we always stripped them). Now, it looks like we always hash every pallet in the full metadata, and so the hash would sometimes be different from the stripped metadata (ie whenever the stripped metadata actually removes variants, it will be different).
I guess this isn't tested yet, since I don't think it should work as it stands (but maybe I'm missing something?)
I think this hashing function would have to also work out whether the variants would be stripped or not based on what we pass to only_these_runtime_apis and only_these_pallets? This would be a good reason to have a separate find_type_id_in_return_pos type function, so that we can lean on it in both places.
There was a problem hiding this comment.
dq: but isn't this testing the stripping logic by using said stripping logic? maybe its more logical to remove validation that uses specific_pallets?
There was a problem hiding this comment.
This is essentially using the striping logic (without actually having to do the heavyweight stuff of removing types/pallets etc) in order that we can compare some stripped metadata that a user might have passed to the Subxt macro with some "full" metadata that you get from a node, to check that the two are compatible.
And so my concern with this bit of code is that it looks like the hash generated for some stripped metadata would be different from the hash gernerated from the same "full" metadata (assuming that the stripped metadata was able to strip variants from one or more outer enums).
There was a problem hiding this comment.
(happy to jump on a call or chat more about this if it doesn't make sense, and maybe I'm missing something too :))
Description:
See #1659
This fixes the error described inside the issue, however it will strip types in a bit different way which will result in slightly less savings on average.
Biggest difference between the new and old approaches is that:
outerenumenums, unless there are some variants that do not reference any types we intend to retain.Size comparison with the previous stripping approach:
size reduction Comparison:
Comparison for old and new stripping methods. be warned, it's 1K lines