Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[move-2] Decorated values and resource viewer for enum types #14144

Merged
merged 1 commit into from
Jul 30, 2024
Merged

Conversation

wrwg
Copy link
Contributor

@wrwg wrwg commented Jul 28, 2024

Description

This adapts more functionality around enums/struct variants for move value representation:

  • MoveStructLayout::WithVariants is a new decorated representation for variant types, and MoveStruct::WithVariant is the matching decorated value representation. Code has been adapted to deal with those decorated versions
  • Unfortunately, we do not only have serializers/deserializers in value.rs (which where already implemented for variants) but also value_impl.rs. It is unclear what of this code is used where, so implemented now the serialization logic a 2nd time in value_impl.rs with support of custom serializers.
  • Adapted yet another redundant type representation to support variants, FatType in resource viewer. Extended AnnotatedMoveValue to carry an optional field for variant names. This led to some upstream changes in api.
  • Changed serialization logic and runtime representation. Serialization did not really work as we require deserialize_seq for variants and henceforth also for serialization. Since values are serialized type free, they need to be distinguished whether variant or not. This in turn makes it more logical to call out the variant tag explicitly in MoveValue.
  • Implement stackless bytecode generator for enums to make Aptos CLI working. (Required by extended checks.)

Type of Change

  • New feature
  • Bug fix
  • Breaking change
  • Performance improvement
  • Refactoring
  • Dependency update
  • Documentation update
  • Tests

Which Components or Systems Does This Change Impact?

  • Validator Node
  • Full Node (API, Indexer, etc.)
  • Move/Aptos Virtual Machine
  • Aptos Framework
  • Aptos CLI/SDK
  • Developer Infrastructure
  • Other (specify)

How Has This Been Tested?

This is tested by existing tests. Also a new roundtrip test is added. Its unclear how resource viewer and related functionality are tested at all besides e2e tests, but did some manual tests via Aptos CLI.

@wrwg wrwg added CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR CICD:run-framework-upgrade-test CICD:run-windows-tests Runs Windows related CI/CD tests. labels Jul 28, 2024
Copy link

trunk-io bot commented Jul 28, 2024

⏱️ 21h 42m total CI duration on this PR
Job Cumulative Duration Recent Runs
windows-build 8h 8m 🟩🟩🟩🟩🟩 (+4 more)
execution-performance / single-node-performance 3h 19m 🟩🟩🟩🟩🟩 (+4 more)
forge-compat-test / forge 1h 49m 🟩🟩🟩🟥🟩 (+2 more)
forge-e2e-test / forge 1h 45m 🟩🟩🟥🟩 (+3 more)
forge-framework-upgrade-test / forge 1h 38m 🟥🟥🟥🟩🟩 (+2 more)
execution-performance / test-target-determinator 40m 🟩🟩🟩🟩🟩 (+4 more)
test-target-determinator 27m 🟩🟩🟩🟩 (+2 more)
check 26m 🟩🟩🟩🟩🟩 (+1 more)
rust-move-tests 14m 🟩
rust-doc-tests 14m 🟩
rust-doc-tests 14m 🟩
rust-move-unit-coverage 13m 🟩
rust-move-unit-coverage 13m 🟩
check-dynamic-deps 12m 🟩🟩🟩🟩🟩 (+4 more)
general-lints 11m 🟩🟩🟩🟩🟩 (+1 more)
rust-move-tests 10m 🟩
rust-cargo-deny 10m 🟩🟩🟩🟩🟩 (+1 more)
rust-move-tests 10m 🟩
indexer-grpc-e2e-tests / test-indexer-grpc-docker-compose 10m 🟩🟩🟩🟩🟩 (+1 more)
rust-move-tests 10m 🟩
rust-move-tests 9m 🟩
rust-move-unit-coverage 9m 🟩
rust-move-unit-coverage 8m 🟩
rust-move-unit-coverage 8m 🟩
rust-doc-tests 6m 🟩
rust-doc-tests 6m 🟩
rust-doc-tests 6m 🟩
semgrep/ci 4m 🟩🟩🟩🟩🟩 (+4 more)
rust-move-tests 3m
rust-doc-tests 3m
file_change_determinator 1m 🟩🟩🟩🟩🟩 (+2 more)
file_change_determinator 1m 🟩🟩🟩🟩🟩 (+2 more)
file_change_determinator 1m 🟩🟩🟩🟩🟩 (+1 more)
permission-check 28s 🟩🟩🟩🟩🟩 (+5 more)
permission-check 24s 🟩🟩🟩🟩🟩 (+5 more)
permission-check 24s 🟩🟩🟩🟩🟩 (+3 more)
permission-check 22s 🟩🟩🟩🟩🟩 (+2 more)
permission-check 18s 🟩🟩🟩🟩🟩 (+2 more)
Backport PR 18s 🟥🟥🟥🟥
determine-docker-build-metadata 11s 🟩🟩🟩🟩🟩 (+2 more)
permission-check 8s 🟩🟩🟩🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job Duration vs 7d avg Delta
forge-framework-upgrade-test / forge 19m 13m +48%

settingsfeedbackdocs ⋅ learn more about trunk.io

Copy link

codecov bot commented Jul 28, 2024

Codecov Report

Attention: Patch coverage is 23.42216% with 546 lines in your changes missing coverage. Please review.

Project coverage is 59.0%. Comparing base (f991f0d) to head (2d8b393).
Report is 9 commits behind head on main.

Files Patch % Lines
...model/bytecode/src/stackless_bytecode_generator.rs 0.0% 258 Missing ⚠️
...party/move/move-vm/types/src/values/values_impl.rs 30.4% 89 Missing ⚠️
third_party/move/move-core/types/src/value.rs 17.1% 87 Missing ⚠️
...d_party/move/tools/move-resource-viewer/src/lib.rs 47.1% 55 Missing ⚠️
...ty/move/tools/move-resource-viewer/src/fat_type.rs 42.8% 28 Missing ⚠️
aptos-move/framework/src/natives/string_utils.rs 0.0% 22 Missing ⚠️
third_party/move/move-model/src/model.rs 0.0% 3 Missing ⚠️
...ypes/src/delayed_values/derived_string_snapshot.rs 0.0% 2 Missing ⚠️
...ve/move-vm/types/src/values/serialization_tests.rs 95.2% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##             main   #14144     +/-   ##
=========================================
- Coverage    59.1%    59.0%   -0.2%     
=========================================
  Files         827      828      +1     
  Lines      201015   201584    +569     
=========================================
+ Hits       118830   118943    +113     
- Misses      82185    82641    +456     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Copy link
Contributor Author

@wrwg wrwg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the reviews, all comments should be addressed.

api/src/accounts.rs Show resolved Hide resolved
abort_code: EARGS_MISMATCH,
});
}
let tag = elems.pop().unwrap().value_as::<u32>()? as usize;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, fixed. This is no handled as a general helper in values_impl.

/// A decorated representation with human-readable field names
WithFields(Vec<(Identifier, MoveValue)>),
/// An even more decorated representation with both types and human-readable field names
WithTypes {
type_: StructTag,
fields: Vec<(Identifier, MoveValue)>,
},
/// A decorated representation of a variant, with the variant name, tag value, and field values.
WithVariant(Identifier, u16, Vec<(Identifier, MoveValue)>),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

#[derive(Debug, Clone, Hash, Serialize, Deserialize, PartialEq, Eq)]
#[cfg_attr(any(test, feature = "fuzzing"), derive(arbitrary::Arbitrary))]
pub enum MoveStructLayout {
/// The representation used by the MoveVM for plain structs
Runtime(Vec<MoveTypeLayout>),
/// The representation used by the MoveVM for struct variants.
/// The representation used by the MoveVM for plaint struct variants.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@@ -40,6 +40,7 @@ mod limit;
pub struct AnnotatedMoveStruct {
pub abilities: AbilitySet,
pub ty_tag: StructTag,
pub variant_tag: Option<(u16, Identifier)>,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

.collect(),
MoveStruct::WithFields(fields) | MoveStruct::WithTypes { fields, .. } => fields,
MoveStruct::Runtime(values) => {
let (_, field_names) = self.get_field_information(&ty, None)?;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a debug assert, but this is clearly satisfied invariant since if you pass in None to this function, the returned tag will always be None. Does not depend on external input.

},
_ => bail!("inconsistent layout information"),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, but not boiling the ocean. All those things should be asserts but we can't have crashes. We really should have an internal_bail! which creates a bug report with stack trace or something.

if let Some((_, name)) = &self.variant_tag {
let mut s = serializer.serialize_map(Some(self.value.len() + 1))?;
s.serialize_entry("$variant", name)?;
for (f, v) in &self.value {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This adapts more functionality around enums/struct variants for move value representation:

- `MoveStructLayout::WithVariants` is a new decorated representation for variant types, and `MoveStruct::WithVariant` is the matching decorated value representation. Code has been adapted to deal with those decorated versions
- Unfortunately, we do not only have serializers/deserializers in `value.rs` (which where already implemented for variants) but also `value_impl.rs`. It is unclear what of this code is used where, so implemented now the serialization logic a 2nd time in `value_impl.rs` with support of custom serializers.
- Adapted yet another redundant type representation to support variants, `FatType` in resource viewer. Extended `AnnotatedMoveValue` to carry an optional field for variant names. This led to some upstream changes in api.
- Changed serialization logic and runtime representation. Serialization did not really work as we require `deserialize_seq` for variants and henceforth also for serialization. Since values are serialized type free, they need to be distinguished whether variant or not.  This in turn makes it more logical to call out the variant tag explicitly in MoveValue.
- Implement stackless bytecode generator for enums to make Aptos CLI working. (Required by extended checks.)

This is tested by existing tests. Also a new roundtrip test is added. Its unclear how resource viewer and related functionality are tested at all besides e2e tests, but did some manual tests via Aptos CLI.

This comment has been minimized.

@wrwg wrwg enabled auto-merge (squash) July 30, 2024 19:01

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Copy link
Contributor

✅ Forge suite realistic_env_max_load success on 2d8b393d2691002dc6034d550bfd33f19c2ce82b

two traffics test: inner traffic : committed: 9393.659474693695 txn/s, latency: 4253.2442209268465 ms, (p50: 4200 ms, p90: 4800 ms, p99: 11400 ms), latency samples: 3571680
two traffics test : committed: 100.0915816479703 txn/s, latency: 2311.024157303371 ms, (p50: 2100 ms, p90: 2300 ms, p99: 14700 ms), latency samples: 1780
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.242, avg: 0.218", "QsPosToProposal: max: 1.852, avg: 1.774", "ConsensusProposalToOrdered: max: 0.339, avg: 0.293", "ConsensusOrderedToCommit: max: 0.414, avg: 0.397", "ConsensusProposalToCommit: max: 0.702, avg: 0.691"]
Max round gap was 1 [limit 4] at version 2018766. Max no progress secs was 5.827048 [limit 15] at version 2018766.
Test Ok

This comment has been minimized.

Copy link
Contributor

✅ Forge suite framework_upgrade success on 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 2d8b393d2691002dc6034d550bfd33f19c2ce82b

Compatibility test results for 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 2d8b393d2691002dc6034d550bfd33f19c2ce82b (PR)
Upgrade the nodes to version: 2d8b393d2691002dc6034d550bfd33f19c2ce82b
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1096.861794443383 txn/s, submitted: 1098.1953650141347 txn/s, failed submission: 1.3335705707518335 txn/s, expired: 1.3335705707518335 txn/s, latency: 2898.7295845997974 ms, (p50: 2100 ms, p90: 4800 ms, p99: 11700 ms), latency samples: 98700
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1099.0886365759047 txn/s, submitted: 1102.8143607676875 txn/s, failed submission: 3.725724191782728 txn/s, expired: 3.725724191782728 txn/s, latency: 2675.73221335992 ms, (p50: 2100 ms, p90: 4500 ms, p99: 12600 ms), latency samples: 100300
5. check swarm health
Compatibility test for 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 2d8b393d2691002dc6034d550bfd33f19c2ce82b passed
Upgrade the remaining nodes to version: 2d8b393d2691002dc6034d550bfd33f19c2ce82b
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1151.5511440084542 txn/s, submitted: 1153.7609907634119 txn/s, failed submission: 2.209846754957694 txn/s, expired: 2.209846754957694 txn/s, latency: 2829.6557954327386 ms, (p50: 2100 ms, p90: 5400 ms, p99: 10500 ms), latency samples: 104220
Test Ok

This comment has been minimized.

Copy link
Contributor

✅ Forge suite compat success on 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 2d8b393d2691002dc6034d550bfd33f19c2ce82b

Compatibility test results for 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 2d8b393d2691002dc6034d550bfd33f19c2ce82b (PR)
1. Check liveness of validators at old version: 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5
compatibility::simple-validator-upgrade::liveness-check : committed: 7101.499255178765 txn/s, latency: 3957.3825184839807 ms, (p50: 3000 ms, p90: 6200 ms, p99: 22900 ms), latency samples: 300260
2. Upgrading first Validator to new version: 2d8b393d2691002dc6034d550bfd33f19c2ce82b
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7203.527654373658 txn/s, latency: 3888.087085625617 ms, (p50: 4000 ms, p90: 5400 ms, p99: 5600 ms), latency samples: 141780
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7488.5652823682785 txn/s, latency: 4219.135155555556 ms, (p50: 4100 ms, p90: 6800 ms, p99: 6900 ms), latency samples: 247500
3. Upgrading rest of first batch to new version: 2d8b393d2691002dc6034d550bfd33f19c2ce82b
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 1807.5364613360023 txn/s, latency: 16311.277290552585 ms, (p50: 18800 ms, p90: 24400 ms, p99: 24500 ms), latency samples: 56100
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 2068.6640747039496 txn/s, latency: 14432.740780961183 ms, (p50: 15400 ms, p90: 24600 ms, p99: 29300 ms), latency samples: 86560
4. upgrading second batch to new version: 2d8b393d2691002dc6034d550bfd33f19c2ce82b
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 10584.592144968612 txn/s, latency: 2682.3664925058056 ms, (p50: 2900 ms, p90: 3400 ms, p99: 3500 ms), latency samples: 189480
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 6751.772473374452 txn/s, latency: 4547.899613208637 ms, (p50: 2800 ms, p90: 8300 ms, p99: 23300 ms), latency samples: 346440
5. check swarm health
Compatibility test for 1c2ee7082d6eff8c811ee25d6f5a7d00860a75d5 ==> 2d8b393d2691002dc6034d550bfd33f19c2ce82b passed
Test Ok

@wrwg wrwg merged commit be4712a into main Jul 30, 2024
74 of 90 checks passed
@wrwg wrwg deleted the wrwg/enum-4 branch July 30, 2024 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CICD:run-e2e-tests when this label is present github actions will run all land-blocking e2e tests from the PR CICD:run-framework-upgrade-test CICD:run-windows-tests Runs Windows related CI/CD tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants