Introduce automatic ABI maintenance mechanism (1/2; prepare)

docs/src/SUMMARY.md

@@ -101,6 +101,7 @@
   * [Snapshot Verification](implemented-proposals/snapshot-verification.md)
   * [Cross-Program Invocation](implemented-proposals/cross-program-invocation.md)
   * [Program Derived Addresses](implemented-proposals/program-derived-addresses.md)
+  * [ABI Management](implemented-proposals/abi-management.md)
 * [Accepted Design Proposals](proposals/README.md)
   * [Ledger Replication](proposals/ledger-replication-to-implement.md)
   * [Optimistic Confirmation and Slashing](proposals/optimistic-confirmation-and-slashing.md)
@@ -114,6 +115,5 @@
   * [Slashing](proposals/slashing.md)
   * [Tick Verification](proposals/tick-verification.md)
   * [Block Confirmation](proposals/block-confirmation.md)
-  * [ABI Management](proposals/abi-management.md)
   * [Rust Clients](proposals/rust-clients.md)
   * [Optimistic Confirmation](proposals/optimistic_confirmation.md)

docs/src/proposals/abi-management.md → docs/src/implemented-proposals/abi-management.md

@@ -55,7 +55,7 @@ fields.
 # Example
 
 ```patch
-+#[frozen_abi(digest="1c6a53e9")]
++#[frozen_abi(digest="eXSMM7b89VY72V...")]
  #[derive(Serialize, Default, Deserialize, Debug, PartialEq, Eq, Clone)]
  pub struct Vote {
      /// A stack of votes starting with the oldest vote
@@ -73,16 +73,85 @@ digest from the assertion test error message.
 
 In general, once we add `frozen_abi` and its change is published in the stable
 release channel, its digest should never change. If such a change is needed, we
-should opt for defining a new struct like `FooV1`. And special release flow like
-hard forks should be approached.
+should opt for defining a new `struct` like `FooV1`. And special release flow
+like hard forks should be approached.
 
 # Implementation remarks
 
 We use some degree of macro machinery to automatically generate unit tests
 and calculate a digest from ABI items. This is doable by clever use of
-`serde::Serialize` (`[1]`) and `any::typename` (`[2]`). For a precedent for similar
+`serde::Serialize` (`[1]`) and `any::type_name` (`[2]`). For a precedent for similar
 implementation, `ink` from the Parity Technologies `[3]` could be informational.
 
+# Implementation details
+
+The implementation's goal is to detect unintended ABI changes automatically as
+much as possible. To that end, the digest of structural ABI information is
+calculated with best-effort accuracy and stability.
+
+When the ABI digest check is run, it dynamically computes an ABI digest by
+recursively digesting the ABI of fields of the ABI item, by re-using the
+`serde`'s serialization functionality, proc macro and generic specialization.
+And then, the check `assert!`s that its finalized digest value is identical as
+what is specified in the `frozen_abi` attribute.
+
+To realize that, it creates an example instance of the type and a custom
+`Serializer` instance for `serde` to recursively traverse its fields as if
+serializing the example for real. This traversing must be done via `serde` to
+really capture what kinds of data actually would be serialized by `serde`, even
+considering custom non-`derive`d `Serialize` trait implementations.
+
+# The ABI digesting process
+
+This part is a bit complex. There is three inter-depending parts: `AbiExample`,
+`AbiDigester` and `AbiEnumVisitor`.
+
+First, the generated test creates an example instance of the digested type with
+a trait called `AbiExample`, which should be implemented for all of digested
+types like the `Serialize` and return `Self` like the `Default` trait. Usually,
+it's provided via generic trait specialization for most of common types.  Also
+it is possible to `derive` for `struct` and `enum` and can be hand-written if
+needed.
+
+The custom `Serializer` is called `AbiDigester`. And when it's called by `serde`
+to serialize some data, it recursively collects ABI information as much as
+possible. `AbiDigester`'s internal state for the ABI digest is updated
+differentially depending on the type of data. This logic is specifically
+redirected via with a trait called `AbiEnumVisitor` for each `enum` type. As the
+name suggests, there is no need to implement `AbiEnumVisitor` for other types.
+
+To summarize this interplay, `serde` handles the recursive serialization control
+flow in tandem with `AbiDigester`. The initial entry point in tests and child
+`AbiDigester`s use `AbiExample` recursively to create an example object
+hierarchal graph. And `AbiDigester` uses `AbiEnumVisitor` to inquiry the actual
+ABI information using the constructed sample.
+
+`Default` isn't enough for `AbiExample`. Various collection's `::default()` is
+empty, yet, we want to digest them with actual items. And, ABI digesting can't
+be realized only with `AbiEnumVisitor`. `AbiExample` is required because an
+actual instance of type is needed to actually traverse the data via `serde`.
+
+On the other hand, ABI digesting can't be done only with `AbiExample`, either.
+`AbiEnumVisitor` is required because all variants of an `enum` cannot be
+traversed just with a single variant of it as a ABI example.
+
+Digestable information:
+
+- rust's type name
+- `serde`'s data type name
+- all fields in `struct`
+- all variants in `enum`
+- `struct`: normal(`struct {...}`) and tuple-style (`struct(...)`)
+- `enum`: normal variants and `struct`- and `tuple`- styles.
+- attributes: `serde(serialize_with=...)` and `serde(skip)`
+
+Not digestable information:
+
+- Any custom serialize code path not touched by the sample provided by
+  `AbiExample`. (technically not possible)
+- generics (must be a concrete type; use `frozen_abi` on concrete type
+  aliases)
+
 # References
 
 1. [(De)Serialization with type info · Issue #1095 · serde-rs/serde](https://github.com/serde-rs/serde/issues/1095#issuecomment-345483479)

sdk/Cargo.toml

@@ -54,9 +54,13 @@ ed25519-dalek = { version = "=1.0.0-pre.3", optional = true }
 solana-crate-features = { path = "../crate-features", version = "1.3.0", optional = true }
 solana-logger = { path = "../logger", version = "1.3.0", optional = true }
 solana-sdk-macro = { path = "macro", version = "1.3.0" }
+solana-sdk-macro-frozen-abi = { path = "macro-frozen-abi", version = "1.3.0" }
 
 [dev-dependencies]
 tiny-bip39 = "0.7.0"
 
 [package.metadata.docs.rs]
 targets = ["x86_64-unknown-linux-gnu"]
+
+[build-dependencies]
+rustc_version = "0.2"

sdk/build.rs

@@ -0,0 +1,22 @@
+extern crate rustc_version;
+use rustc_version::{version_meta, Channel};
+
+fn main() {
+    // Copied and adapted from
+    // https://github.com/Kimundi/rustc-version-rs/blob/1d692a965f4e48a8cb72e82cda953107c0d22f47/README.md#example
+    // Licensed under Apache-2.0 + MIT
+    match version_meta().unwrap().channel {
+        Channel::Stable => {
+            println!("cargo:rustc-cfg=RUSTC_WITHOUT_SPECIALIZATION");
+        }
+        Channel::Beta => {
+            println!("cargo:rustc-cfg=RUSTC_WITHOUT_SPECIALIZATION");
+        }
+        Channel::Nightly => {
+            println!("cargo:rustc-cfg=RUSTC_WITH_SPECIALIZATION");
+        }
+        Channel::Dev => {
+            println!("cargo:rustc-cfg=RUSTC_WITH_SPECIALIZATION");
+        }
+    }
+}

sdk/macro-frozen-abi/Cargo.toml

@@ -0,0 +1,22 @@
+[package]
+name = "solana-sdk-macro-frozen-abi"
+version = "1.3.0"
+description = "Solana SDK Macro frozen abi"
+authors = ["Solana Maintainers <maintainers@solana.com>"]
+repository = "https://github.com/solana-labs/solana"
+homepage = "https://solana.com/"
+license = "Apache-2.0"
+edition = "2018"
+build = "../build.rs"
+
+[lib]
+proc-macro = true
+
+[dependencies]
+proc-macro2 = "1.0"
+quote = "1.0"
+syn = { version = "1.0", features = ["full", "extra-traits"] }
+lazy_static = "1.4.0"
+
+[build-dependencies]
+rustc_version = "0.2"