From ad0cb709730c26f143bb1a47b2cd2ef608e04b95 Mon Sep 17 00:00:00 2001 From: Matthew Hammer Date: Mon, 3 Feb 2020 15:37:32 -0700 Subject: [PATCH 01/14] initial draft --- docs/design/dfx-testing-story.adoc | 175 +++++++++++++++++++++++++++++ 1 file changed, 175 insertions(+) create mode 100644 docs/design/dfx-testing-story.adoc diff --git a/docs/design/dfx-testing-story.adoc b/docs/design/dfx-testing-story.adoc new file mode 100644 index 0000000000..e54057f284 --- /dev/null +++ b/docs/design/dfx-testing-story.adoc @@ -0,0 +1,175 @@ +## Design Doc: DFX Testing Story + +Creator(s): Matthew Hammer, after discussion with and input from Hans Larsen, on 2020-02-03. + +### Terms and goals + +In CI, we want to write: + +`dfx test` + +And have this statement mean two things at once: + + - "Run all end-to-end tests for the project" + + - "Run all unit tests for all libraries used by this project" (we + e currently recompile these transitive libraries ourselves, so this + "transitive testing" is a reasonable thing to do, too). + +These two classes of tests are quite different in their requirements +of the `dfx` tool. We consider them separately as parts 1 and 2 below. + +### Part 1: Authoring a CI-driven, end-to-end (e2e) test of a Motoko project + +Usecase: _We want to author a Motoko project, and have CI rerun a +suite of end-to-end (e2e) tests for each commit, e.g., using Travis +CI, or something similar._ + +#### Simplifying assumptions + +- we do not discuss building cansiters here; we assume they have been built somehow; this design is about testing, not building. +- each e2e test consists of starting a replica, running one or more "test actions" (see below), and then stopping the replica. +- two distinct e2e tests are independent; they do not share any service state; they only share overlapping source code for canisters (probably). +- distinct e2e tests can be run in sequence, in parallel, or even on distinct replicas (if necessary; not a priority yet). + +In the remainder of this document, we focus on the case of a single +e2e test in isolation, knowing that each _additional_ e2e test is +always an independent (additional/separable) concern. + +Each e2e test is given by a single sequence of test actions. We refer +to this sequence as the test's _action sequence_. We use the term +"test script" and "action sequence" interchangeably throughout. + +#### Test actions + +Each "test action" is a scriptable dfx command with an associated +response that it expects, perhaps as a _pattern_. + +` ::= ` (three distinct cases:) + +- `| dfx-canister-install ` +- `| dfx-canister-call ` +- `| dfx-canister-query ` + +In each case, the last part of the action gives a "dfx result +pattern", which permits the sequence to destructure results and bind +variables, which may appear within the `` of subsequent actions +of the same sequence. + +**Static checks:** The test system statically checks a test's action +sequence before running it; consequently, variables cannot be used in +the sequence without being first defined earlier in the sequence. We +do not check the IDL result and argument types match, however (or do +we want to do that too?). + +#### Dfx result patterns + +Each time the test script gets a response from an action (`install` or `call` or +`query`) we may either want to bind that value to a test script +variable (to use in a subsequent action), or insist that that value is +a certain value, or both. + +The dfx result pattern grammar permits each of these uses: + +``` + ::= bind + | match + + ::= + | (, ..., ) + | record { lab_1: , ..., lab_k: } + | variant { lab } + | + | + | + | ( as ) +...TODO +``` + +The second case (`match`) is a generalization of the first one +(`bind`), as it binds variables as one possible usecase. + +More generally, it consists of patterns that each attempt to match an +IDL value, destructure it, and bind its parts to zero or more variable +names. The grammar above should agree with the IDL value syntax (but +probably doesn't yet). + +In either the `bind` or `match` cases, the test variables being +defined by the action's result exist within the scope of the remaining +actions in the same sequence, but not outside of it. These variables +are useful for forming functional chains of test calls within the +script symbolically, where they avoid the script explicitly mentioning +all of the intermittent input and output values explicitly, as IDL +values. + +#### Technical clarification on error detection and reporting: + +The result pattern-matching logic aluded to above (for the + grammar) should build on the existing Rust +codebase for Candid, not be based on commands within bash, over +stringified versions, etc. In particular, the test fails with special +kinds of test errors if this scripting syntax is malformed in some +way; likewise, the test actions themselves must follow a formal +syntax, and they emit associated errors, if at all, before the test +suite runs any test actions. + +#### Remaining details: + +Many practical details remain to nail down, but the sketch above gives +the most important definitions; we still need a discipline for listing +multiple e2e tests within the `dfx.json` file, perhaps with distinct +names, so that `dfx test e2e foo` and `dfx test e2e bar` do different +things. + +Likewise, we may want regex logic over these names, so that `dfx test +e2e foo*` and `dfx test e2e bar*Baz` mean and do the expected things. +This is all TBD. + + +### Part 2: Unit tests on canister Wasm from Motoko source + +Suppose a developer writes a Motoko library `FooLib` with a set of +public functions and a set of (publicly visible) unit test functions +that exercise the other (public or private) functions of `FooLib`. + +Currently, we have a story for a Motoko project to import `FooLib` and +use its public interface when it can point to `FooLib` somewhere on the +local filesystem. + +**However, we have no story for the client program running the unit +tests of FooLib. Where does it look to find them?** + +When unit tests or functional tests they only exercise the public +interface of FooLib, and never enter its private implementation +directly, we may want these test functions to exist outside of the +implementation file/directory for FooLib. Currently, we organize +the Motoko `stdlib` source and test trees as parallel, but distinct: + +(Omitting all but two modules, for concision:) +``` +/src/buf.mo +/src/hash.mo +/test/bufTest.mo +/test/hashTest.mo +``` + +#### Key Assumption, to simplify public/private visibility: + +Let's assume that all Motoko code that we want to directly unit test +is public in its defining module. + +Under this assumption, we can assume that testing code is seperable +from implementation code, and the `src` versus `test` directory +distinction is workable. + +Of course, this implementation module versus test module separation +does not make sense for unit tests of private functions. Those tests +require a different approach, where the (publicly-exposed) test code +is mixed into the module itself, where it can access private members +to test them. While this usecase is important, it's a complex use +case in terms of visibility (public test and private code being +tested), so let's set it aside for now. + +#### Remaining details: + +_(Very similar to those of Part 1.)_ \ No newline at end of file From 51cea0ad628e9294e22cd252f49c110aa5fc4180 Mon Sep 17 00:00:00 2001 From: Matthew Hammer Date: Mon, 3 Feb 2020 15:43:48 -0700 Subject: [PATCH 02/14] nits --- docs/design/dfx-testing-story.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design/dfx-testing-story.adoc b/docs/design/dfx-testing-story.adoc index e54057f284..4e6604c48e 100644 --- a/docs/design/dfx-testing-story.adoc +++ b/docs/design/dfx-testing-story.adoc @@ -13,7 +13,7 @@ And have this statement mean two things at once: - "Run all end-to-end tests for the project" - "Run all unit tests for all libraries used by this project" (we - e currently recompile these transitive libraries ourselves, so this + currently recompile these transitive libraries ourselves, so this "transitive testing" is a reasonable thing to do, too). These two classes of tests are quite different in their requirements @@ -53,7 +53,7 @@ response that it expects, perhaps as a _pattern_. In each case, the last part of the action gives a "dfx result pattern", which permits the sequence to destructure results and bind -variables, which may appear within the `` of subsequent actions +variables, which may appear as either the `` or within the `` of subsequent actions of the same sequence. **Static checks:** The test system statically checks a test's action From 01c5858af836cc88aa8594b72f2163c35df3faaf Mon Sep 17 00:00:00 2001 From: Matthew Hammer Date: Mon, 3 Feb 2020 15:44:55 -0700 Subject: [PATCH 03/14] nits --- docs/design/dfx-testing-story.adoc | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/docs/design/dfx-testing-story.adoc b/docs/design/dfx-testing-story.adoc index 4e6604c48e..c6357f2b75 100644 --- a/docs/design/dfx-testing-story.adoc +++ b/docs/design/dfx-testing-story.adoc @@ -27,10 +27,18 @@ CI, or something similar._ #### Simplifying assumptions -- we do not discuss building cansiters here; we assume they have been built somehow; this design is about testing, not building. -- each e2e test consists of starting a replica, running one or more "test actions" (see below), and then stopping the replica. -- two distinct e2e tests are independent; they do not share any service state; they only share overlapping source code for canisters (probably). -- distinct e2e tests can be run in sequence, in parallel, or even on distinct replicas (if necessary; not a priority yet). +- we do not discuss building cansiters here; we assume they have been + built somehow; this design is about testing, not building. + +- each e2e test consists of starting a replica, running one or more + "test actions" (see below), and then stopping the replica. + +- two distinct e2e tests are independent; they do not share any + service state; they only share overlapping source code for canisters + (probably). + +- distinct e2e tests can be run in sequence, in parallel, or even on + distinct replicas (if necessary; not a priority yet). In the remainder of this document, we focus on the case of a single e2e test in isolation, knowing that each _additional_ e2e test is @@ -53,8 +61,8 @@ response that it expects, perhaps as a _pattern_. In each case, the last part of the action gives a "dfx result pattern", which permits the sequence to destructure results and bind -variables, which may appear as either the `` or within the `` of subsequent actions -of the same sequence. +variables, which may appear as either the `` or within the +`` of subsequent actions of the same sequence. **Static checks:** The test system statically checks a test's action sequence before running it; consequently, variables cannot be used in @@ -64,10 +72,10 @@ we want to do that too?). #### Dfx result patterns -Each time the test script gets a response from an action (`install` or `call` or -`query`) we may either want to bind that value to a test script -variable (to use in a subsequent action), or insist that that value is -a certain value, or both. +Each time the test script gets a response from an action (`install` or +`call` or `query`) we may either want to bind that value to a test +script variable (to use in a subsequent action), or insist that that +value is a certain value, or both. The dfx result pattern grammar permits each of these uses: From 7483a22fc9c8138f8297c853d669bcf98a8eb720 Mon Sep 17 00:00:00 2001 From: Matthew Hammer Date: Mon, 3 Feb 2020 15:53:40 -0700 Subject: [PATCH 04/14] nits --- docs/design/dfx-testing-story.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design/dfx-testing-story.adoc b/docs/design/dfx-testing-story.adoc index c6357f2b75..4f0b46dbc2 100644 --- a/docs/design/dfx-testing-story.adoc +++ b/docs/design/dfx-testing-story.adoc @@ -147,9 +147,9 @@ local filesystem. **However, we have no story for the client program running the unit tests of FooLib. Where does it look to find them?** -When unit tests or functional tests they only exercise the public -interface of FooLib, and never enter its private implementation -directly, we may want these test functions to exist outside of the +When unit tests (or functional tests) only exercise the _public_ +interface of FooLib, and _never enter its private implementation +directly_, we may want these test functions to exist outside of the implementation file/directory for FooLib. Currently, we organize the Motoko `stdlib` source and test trees as parallel, but distinct: From 40c45b2f9d3e392d4cff4315cf288c1aed6e9789 Mon Sep 17 00:00:00 2001 From: Matthew Hammer Date: Mon, 3 Feb 2020 15:55:57 -0700 Subject: [PATCH 05/14] spelling --- docs/design/dfx-testing-story.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design/dfx-testing-story.adoc b/docs/design/dfx-testing-story.adoc index 4f0b46dbc2..755fa89030 100644 --- a/docs/design/dfx-testing-story.adoc +++ b/docs/design/dfx-testing-story.adoc @@ -27,7 +27,7 @@ CI, or something similar._ #### Simplifying assumptions -- we do not discuss building cansiters here; we assume they have been +- we do not discuss building canisters here; we assume they have been built somehow; this design is about testing, not building. - each e2e test consists of starting a replica, running one or more @@ -112,7 +112,7 @@ values. #### Technical clarification on error detection and reporting: -The result pattern-matching logic aluded to above (for the +The result pattern-matching logic alluded to above (for the grammar) should build on the existing Rust codebase for Candid, not be based on commands within bash, over stringified versions, etc. In particular, the test fails with special @@ -166,7 +166,7 @@ the Motoko `stdlib` source and test trees as parallel, but distinct: Let's assume that all Motoko code that we want to directly unit test is public in its defining module. -Under this assumption, we can assume that testing code is seperable +Under this assumption, we can assume that testing code is separable from implementation code, and the `src` versus `test` directory distinction is workable. From b1377848ae7237bde0e155e0312a77806daf1adf Mon Sep 17 00:00:00 2001 From: Matthew Hammer Date: Mon, 3 Feb 2020 16:19:49 -0700 Subject: [PATCH 06/14] write down loose ends --- docs/design/dfx-testing-story.adoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/design/dfx-testing-story.adoc b/docs/design/dfx-testing-story.adoc index 755fa89030..dd3d39cf2d 100644 --- a/docs/design/dfx-testing-story.adoc +++ b/docs/design/dfx-testing-story.adoc @@ -180,4 +180,6 @@ tested), so let's set it aside for now. #### Remaining details: -_(Very similar to those of Part 1.)_ \ No newline at end of file +- How does `dfx` discover the test functions of the `test` modules? +- For instance, can it discover all of the public methods of each test module and merely assume each is a unit test? +- Other details that are similar to those of Part 1 with respect to `dfx.json`. \ No newline at end of file From 6abc559f18ed53fde87e3e8e469cb378cb155d2d Mon Sep 17 00:00:00 2001 From: Matthew Hammer Date: Tue, 4 Feb 2020 14:44:13 -0700 Subject: [PATCH 07/14] add examples of the design --- .../design/dfx-testing-story-example/dfx.json | 31 ++++++++++ .../dfx-testing-story-example/src/main.mo | 1 + .../test/aliceBob/aliceCanister.mo | 1 + .../test/aliceBob/bobCanister.mo | 1 + .../test/aliceBob/test-script | 10 +++ .../test/aliceBobCharlie/charlieCanister.mo | 1 + .../test/aliceBobCharlie/test-script | 11 ++++ docs/design/dfx-testing-story.adoc | 62 ++++++++++++++++--- 8 files changed, 108 insertions(+), 10 deletions(-) create mode 100644 docs/design/dfx-testing-story-example/dfx.json create mode 100644 docs/design/dfx-testing-story-example/src/main.mo create mode 100644 docs/design/dfx-testing-story-example/test/aliceBob/aliceCanister.mo create mode 100644 docs/design/dfx-testing-story-example/test/aliceBob/bobCanister.mo create mode 100644 docs/design/dfx-testing-story-example/test/aliceBob/test-script create mode 100644 docs/design/dfx-testing-story-example/test/aliceBobCharlie/charlieCanister.mo create mode 100644 docs/design/dfx-testing-story-example/test/aliceBobCharlie/test-script diff --git a/docs/design/dfx-testing-story-example/dfx.json b/docs/design/dfx-testing-story-example/dfx.json new file mode 100644 index 0000000000..e30fa79233 --- /dev/null +++ b/docs/design/dfx-testing-story-example/dfx.json @@ -0,0 +1,31 @@ +{ + "canisters": { + "cleanSheets": { + "_comment:": "we list `main` service code versus its `unit-tests` (each is a single Motoko function to run) versus its `functional-tests` (each is an `install-and-go`), versus `e2e-tests` that each involve more than a single `install-and-go`, and require a `test-script` over several canisters." + + "main": "src/main.mo" + "unit-tests": { + "all": "test/main.mo" + }, + "func-tests": { + "functionalTest1": "test/simpleAdaptonDivByZero.mo", + "functionalTest2": "test/simpleExcelEmulation.mo", + }, + "e2e-tests": { + "AliceBob": { + "_comment:": "test-script installs and scripts bots `alice` and `bob`" + "test-script": "test/aliceBob/test-script" + "alice": "test/aliceBob/aliceCanister.mo" + "bob": "test/aliceBob/bobCanister.mo" + } + "AliceBobCharlie": { + "_comment:": "reuse bots `alice` and `bob`, and includes bot `charlie`" + "test-script": "test/aliceBobCharlie/test-script" + "alice": "test/aliceBob/aliceCanister.mo" + "bob": "test/aliceBob/bobCanister.mo" + "charlie": "test/aliceBobCharlie/charlieCanister.mo" + } + } + }, + } +} diff --git a/docs/design/dfx-testing-story-example/src/main.mo b/docs/design/dfx-testing-story-example/src/main.mo new file mode 100644 index 0000000000..5ae6c68efa --- /dev/null +++ b/docs/design/dfx-testing-story-example/src/main.mo @@ -0,0 +1 @@ +(EXAMPLE CODE OMITTED) diff --git a/docs/design/dfx-testing-story-example/test/aliceBob/aliceCanister.mo b/docs/design/dfx-testing-story-example/test/aliceBob/aliceCanister.mo new file mode 100644 index 0000000000..5ae6c68efa --- /dev/null +++ b/docs/design/dfx-testing-story-example/test/aliceBob/aliceCanister.mo @@ -0,0 +1 @@ +(EXAMPLE CODE OMITTED) diff --git a/docs/design/dfx-testing-story-example/test/aliceBob/bobCanister.mo b/docs/design/dfx-testing-story-example/test/aliceBob/bobCanister.mo new file mode 100644 index 0000000000..5ae6c68efa --- /dev/null +++ b/docs/design/dfx-testing-story-example/test/aliceBob/bobCanister.mo @@ -0,0 +1 @@ +(EXAMPLE CODE OMITTED) diff --git a/docs/design/dfx-testing-story-example/test/aliceBob/test-script b/docs/design/dfx-testing-story-example/test/aliceBob/test-script new file mode 100644 index 0000000000..84b5b53057 --- /dev/null +++ b/docs/design/dfx-testing-story-example/test/aliceBob/test-script @@ -0,0 +1,10 @@ +install bobCanister +==> #ok(bobId) +install aliceCanister +==> #ok(aliceId) +call bobId addFriend aliceId +==> #ok(()) +call AliceId addFriend BobId +==> #ok(()) +call bobId collaborateWithAlice +==> #ok(()) \ No newline at end of file diff --git a/docs/design/dfx-testing-story-example/test/aliceBobCharlie/charlieCanister.mo b/docs/design/dfx-testing-story-example/test/aliceBobCharlie/charlieCanister.mo new file mode 100644 index 0000000000..5ae6c68efa --- /dev/null +++ b/docs/design/dfx-testing-story-example/test/aliceBobCharlie/charlieCanister.mo @@ -0,0 +1 @@ +(EXAMPLE CODE OMITTED) diff --git a/docs/design/dfx-testing-story-example/test/aliceBobCharlie/test-script b/docs/design/dfx-testing-story-example/test/aliceBobCharlie/test-script new file mode 100644 index 0000000000..7a404e874f --- /dev/null +++ b/docs/design/dfx-testing-story-example/test/aliceBobCharlie/test-script @@ -0,0 +1,11 @@ +install bobCanister ==> #ok(bobId) +install aliceCanister ==> #ok(aliceId) +install charlieCanister ==> #ok(charlieId) +# comment out these calls, to try alternatives where Charlie starts all interactions +# call bobId addFriend aliceId ==> #ok(()) +# call bobId addFriend charlieId ==> #ok(()) +# call AliceId addFriend BobId ==> #ok(()) +# call AliceId addFriend charlieId ==> #ok(()) +call charlieId addFriend bobId ==> #ok(()) +call charlieId addFriend aliceId ==> #ok(()) +call charlieId collaborateWithAliceAndBob ==> #ok(()) diff --git a/docs/design/dfx-testing-story.adoc b/docs/design/dfx-testing-story.adoc index dd3d39cf2d..8cb05e5473 100644 --- a/docs/design/dfx-testing-story.adoc +++ b/docs/design/dfx-testing-story.adoc @@ -4,20 +4,24 @@ Creator(s): Matthew Hammer, after discussion with and input from Hans Larsen, on ### Terms and goals -In CI, we want to write: +In CI, we want to write, as humans and as CI scripts: `dfx test` And have this statement mean two things at once: - - "Run all end-to-end tests for the project" + - "Run all end-to-end tests for the project" (`dfx test e2e` for _just_ this behavior). - - "Run all unit tests for all libraries used by this project" (we + - "Run all unit tests for all libraries used by this project" (`dfx test unit` for _just_ this behavior. we currently recompile these transitive libraries ourselves, so this "transitive testing" is a reasonable thing to do, too). -These two classes of tests are quite different in their requirements -of the `dfx` tool. We consider them separately as parts 1 and 2 below. + - "Run all functional tests for all libraries used by this project" + (for tests that are between unit tests and e2e tests in complexity; see the example project tree.) + + +These classes of tests are quite different in their requirements +of the `dfx` tool. We consider the first two separately as parts 1 and 2 below. ### Part 1: Authoring a CI-driven, end-to-end (e2e) test of a Motoko project @@ -83,7 +87,7 @@ The dfx result pattern grammar permits each of these uses: ::= bind | match - ::= + ::= | (, ..., ) | record { lab_1: , ..., lab_k: } | variant { lab } @@ -95,7 +99,7 @@ The dfx result pattern grammar permits each of these uses: ``` The second case (`match`) is a generalization of the first one -(`bind`), as it binds variables as one possible usecase. +(`bind`), as it binds variables as one possible usecase. More generally, it consists of patterns that each attempt to match an IDL value, destructure it, and bind its parts to zero or more variable @@ -110,7 +114,7 @@ script symbolically, where they avoid the script explicitly mentioning all of the intermittent input and output values explicitly, as IDL values. -#### Technical clarification on error detection and reporting: +#### Technical clarification on error detection and reporting: The result pattern-matching logic alluded to above (for the grammar) should build on the existing Rust @@ -161,7 +165,7 @@ the Motoko `stdlib` source and test trees as parallel, but distinct: /test/hashTest.mo ``` -#### Key Assumption, to simplify public/private visibility: +#### Key Assumption, to simplify public/private visibility: Let's assume that all Motoko code that we want to directly unit test is public in its defining module. @@ -182,4 +186,42 @@ tested), so let's set it aside for now. - How does `dfx` discover the test functions of the `test` modules? - For instance, can it discover all of the public methods of each test module and merely assume each is a unit test? -- Other details that are similar to those of Part 1 with respect to `dfx.json`. \ No newline at end of file +- Other details that are similar to those of Part 1 with respect to `dfx.json`. + + + +### Example project files + +To help nail down the remaining details listed above, this proposal +includes an example project tree, illustrating how the `dfx.json` file +relates to this tree, and the files that it contains, including the +canister source that is just for tests, and the test scripts that +install and call these canisters: + +``` +dfx-testing-story-example +├── dfx.json +├── src +│ ├── main.mo +└── test + ├── aliceBob + │ ├── aliceCanister.mo + │ ├── bobCanister.mo + │ └── test-script + └── aliceBobCharlie + ├── charlieCanister.mo + └── test-script +``` + +The tree contains two example test scripts, for a 2-party (2-canister) +interaction, and for a 3-party (3-canister) interaction. + +The example test scripts (the `test-script` files) in this project +tree give example-based proposals for concrete syntax for the test +scripts, and their test actions. + +Note: The Motoko-like pattern syntax in those files should be replaced +with syntax that is consistent with Candid's concrete syntax +definitions. (the Motoko-like syntax is generally shorter and +more readable to me.) + From caa802bf351a2902e68ed62e033a6315967ce2f9 Mon Sep 17 00:00:00 2001 From: Matthew Hammer Date: Tue, 4 Feb 2020 14:51:01 -0700 Subject: [PATCH 08/14] add example functional tests; each is just a single canister, and a single call --- .../test/simpleAdaptonDivByZero.mo | 208 ++++++++++++++++++ .../test/simpleExcelEmulation.mo | 77 +++++++ 2 files changed, 285 insertions(+) create mode 100644 docs/design/dfx-testing-story-example/test/simpleAdaptonDivByZero.mo create mode 100644 docs/design/dfx-testing-story-example/test/simpleExcelEmulation.mo diff --git a/docs/design/dfx-testing-story-example/test/simpleAdaptonDivByZero.mo b/docs/design/dfx-testing-story-example/test/simpleAdaptonDivByZero.mo new file mode 100644 index 0000000000..0833023a42 --- /dev/null +++ b/docs/design/dfx-testing-story-example/test/simpleAdaptonDivByZero.mo @@ -0,0 +1,208 @@ +import R "mo:stdlib/result"; +import P "mo:stdlib/prelude"; + +import T "../src/types"; +import A "../src/adapton"; +import E "../src/eval"; + +/* + +This file reproduces the intro example from the Adapton Rust docs, +found here: + + https://docs.rs/adapton/0/adapton/#demand-driven-change-propagation + +The DCG update behavior asserted here (see uses of +`assertLogEventLast` below) matches the cleaning/dirtying behavior +described in the link above, and in other documents and papers. + +In particular, the same example is used in a recorded adapton talk, +from the http://adapton.org website. The link above has slides with +still pictures from the talk. + +*/ + +actor SimpleAdaptonDivByZero { + + public func go() { + let ctx : T.Adapton.Context = A.init(true); + + // "cell 1 holds 42": + let cell1 : T.Adapton.NodeId = assertOkPut(A.put(ctx, #nat(1), #nat(42))); + A.assertLogEventLast + (ctx, #put(#nat(1), #nat(42), [])); + + // "cell 2 holds 2": + let cell2 : T.Adapton.NodeId = assertOkPut(A.put(ctx, #nat(2), #nat(2))); + A.assertLogEventLast + (ctx, #put(#nat(2), #nat(2), [])); + + // "cell 3 holds a suspended closure for this expression: + // + // get(cell1) / get(cell2) + // + // ...and it is still unevaluated". + // + let cell3 : T.Adapton.NodeId = assertOkPut( + A.putThunk(ctx, #nat(3), + E.closure( + null, + #strictBinOp(#div, + #get(#refNode(cell1)), + #get(#refNode(cell2)) + ))) + ); + + // "cell 4 holds a suspended closure for this expression: + // + // if (get(cell2) == 0) { 0 } + // else { get(cell3) } + // + // ...and it is still unevaluated". + // + let cell4 : T.Adapton.NodeId = assertOkPut( + A.putThunk(ctx, #nat(4), + E.closure( + null, + #ifCond(#strictBinOp(#eq, + #get(#refNode(cell2)), + #nat(0)), + #nat(0), + #get(#refNode(cell3))))) + ); + + // demand division: + let res1 = assertOkGet(A.get(ctx, cell4)); + A.assertLogEventLast(ctx, + #get(#nat(4), #ok(#nat(21)), + [ + #evalThunk( + #nat(4), #ok(#nat(21)), + [#get(#nat(2), #ok(#nat(2)), []), + #get(#nat(3), #ok(#nat(21)), + [ + #evalThunk( + #nat(3), #ok(#nat(21)), + [ + #get(#nat(1), #ok(#nat(42)), []), + #get(#nat(2), #ok(#nat(2)), []) + ]) + ]) + ]) + ]) + ); + + // "cell 2 holds 0": + ignore A.put(ctx, #nat(2), #nat(0)); + A.assertLogEventLast + (ctx, + #put(#nat(2), #nat(0), + [ + #dirtyIncomingTo( + #nat(2), + [ + #dirtyEdgeFrom( + #nat(3), + [ + #dirtyIncomingTo( + #nat(3), + [ + #dirtyEdgeFrom( + #nat(4), + [ + #dirtyIncomingTo(#nat(4), []) + ]) + ]) + ]), + #dirtyEdgeFrom( + #nat(4), + [ + #dirtyIncomingTo( + #nat(4), []) + ]) + ]) + ]) + ); + + // re-demand division: + let res2 = assertOkGet(A.get(ctx, cell4)); + A.assertLogEventLast + (ctx, + #get(#nat(4), #ok(#nat(0)), + [ + #cleanThunk( + #nat(4), false, + [ + #cleanEdgeTo( + #nat(2), false, []) + ]), + #evalThunk( + #nat(4), #ok(#nat(0)), + [ + #get(#nat(2), #ok(#nat(0)), []) + ]) + ])); + + // "cell 2 holds 2": + ignore A.put(ctx, #nat(2), #nat(2)); + A.assertLogEventLast + (ctx, + #put(#nat(2), #nat(2), + [ + #dirtyIncomingTo( + #nat(2), + [ + #dirtyEdgeFrom( + #nat(4), + [ + #dirtyIncomingTo( + #nat(4), []) + ]) + ]) + ])); + + // re-demand division: + let res3 = assertOkGet(A.get(ctx, cell4)); + A.assertLogEventLast + (ctx, + #get(#nat(4), #ok(#nat(21)), + [ + #cleanThunk( + #nat(4), false, + [ + #cleanEdgeTo(#nat(2), false, []) + ]), + #evalThunk( + #nat(4), #ok(#nat(21)), + [ + #get(#nat(2), #ok(#nat(2)), []), + #get(#nat(3), #ok(#nat(21)), + [ + #cleanThunk( + #nat(3), true, + [ + #cleanEdgeTo(#nat(1), true, []), + #cleanEdgeTo(#nat(2), true, []) + ]) + ]) + ]) + ])); + }; + + + func assertOkPut(r:R.Result) : T.Adapton.NodeId { + switch r { + case (#ok(id)) { id }; + case _ { P.unreachable() }; + } + }; + + func assertOkGet(r:R.Result) : T.Eval.Result { + switch r { + case (#ok(res)) { res }; + case _ { P.unreachable() }; + } + }; +}; + +//SimpleAdaptonDivByZero.go(); diff --git a/docs/design/dfx-testing-story-example/test/simpleExcelEmulation.mo b/docs/design/dfx-testing-story-example/test/simpleExcelEmulation.mo new file mode 100644 index 0000000000..e23030835e --- /dev/null +++ b/docs/design/dfx-testing-story-example/test/simpleExcelEmulation.mo @@ -0,0 +1,77 @@ +import R "mo:stdlib/result"; +import P "mo:stdlib/prelude"; +import Debug "mo:stdlib/debug"; + +import T "../src/types"; +import A "../src/adapton"; +import E "../src/eval"; + +actor simpleExcelEmulation { + + public func go() { + + let sheetExp : T.Eval.Exp = + #sheet( + #text("S"), + [ + [ #nat(1), #nat(2) ], + [ #strictBinOp(#add, + #cellOcc(0,0), + #cellOcc(0,1)), + #strictBinOp(#mul, + #nat(2), + #cellOcc(1,0)) ] + ]); + + // Adapton maintains our dependence graph + let actx : T.Adapton.Context = A.init(true); + + // create the initial Sheet datatype from the DSL expression above + let s : T.Sheet.Sheet = { + switch (E.evalExp(actx, null, sheetExp)) { + case (#ok(#sheet(s))) s; + case _ { P.unreachable() }; + }}; + + // Demand that the sheet's results are fully refreshed + ignore E.Sheet.refresh(actx, s); + ignore E.Sheet.refresh(actx, s); + A.assertLogEventLast( + actx, + #get(#tagTup(#text("S"), [#nat(1), #nat(1), #text("out")]), #ok(#nat(6)), []) + ); + + // Update the sheet by overwriting (0,0) with a new formula: + ignore E.Sheet.update(actx, s, 0, 0, + #strictBinOp(#add, #nat(666), #cellOcc(0,1))); + + // Demand that the sheet's results are fully refreshed + ignore E.Sheet.refresh(actx, s); + ignore E.Sheet.refresh(actx, s); + assert (s.errors.len() == 0); + A.assertLogEventLast( + actx, + #get(#tagTup(#text("S"), [#nat(1), #nat(1), #text("out")]), #ok(#nat(1_340)), []) + ); + + // Update the sheet, creating a cycle at (0,0): + ignore E.Sheet.update(actx, s, 0, 0, #cellOcc(0,0)); + ignore E.Sheet.refresh(actx, s); + assert (s.errors.len() != 0); + + // Update the sheet, removing the cycle: + ignore E.Sheet.update(actx, s, 0, 0, + #strictBinOp(#add, #nat(666), #cellOcc(0,1))); + + // Demand that the sheet's results are fully refreshed + ignore E.Sheet.refresh(actx, s); + ignore E.Sheet.refresh(actx, s); + assert (s.errors.len() == 0); + A.assertLogEventLast( + actx, + #get(#tagTup(#text("S"), [#nat(1), #nat(1), #text("out")]), #ok(#nat(1_340)), []) + ); + }; +}; + +//simpleExcelEmulation.go() From 2ad18e0144f388e719b08878242ce356770d1249 Mon Sep 17 00:00:00 2001 From: Matthew Hammer Date: Tue, 4 Feb 2020 14:58:32 -0700 Subject: [PATCH 09/14] fix typos --- .../dfx-testing-story-example/test/aliceBob/test-script | 2 +- .../test/aliceBobCharlie/test-script | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design/dfx-testing-story-example/test/aliceBob/test-script b/docs/design/dfx-testing-story-example/test/aliceBob/test-script index 84b5b53057..feb88b1290 100644 --- a/docs/design/dfx-testing-story-example/test/aliceBob/test-script +++ b/docs/design/dfx-testing-story-example/test/aliceBob/test-script @@ -4,7 +4,7 @@ install aliceCanister ==> #ok(aliceId) call bobId addFriend aliceId ==> #ok(()) -call AliceId addFriend BobId +call aliceId addFriend BobId ==> #ok(()) call bobId collaborateWithAlice ==> #ok(()) \ No newline at end of file diff --git a/docs/design/dfx-testing-story-example/test/aliceBobCharlie/test-script b/docs/design/dfx-testing-story-example/test/aliceBobCharlie/test-script index 7a404e874f..c30a278352 100644 --- a/docs/design/dfx-testing-story-example/test/aliceBobCharlie/test-script +++ b/docs/design/dfx-testing-story-example/test/aliceBobCharlie/test-script @@ -4,8 +4,8 @@ install charlieCanister ==> #ok(charlieId) # comment out these calls, to try alternatives where Charlie starts all interactions # call bobId addFriend aliceId ==> #ok(()) # call bobId addFriend charlieId ==> #ok(()) -# call AliceId addFriend BobId ==> #ok(()) -# call AliceId addFriend charlieId ==> #ok(()) +# call aliceId addFriend bobId ==> #ok(()) +# call aliceId addFriend charlieId ==> #ok(()) call charlieId addFriend bobId ==> #ok(()) call charlieId addFriend aliceId ==> #ok(()) call charlieId collaborateWithAliceAndBob ==> #ok(()) From c3a36d2f989b429470ea50f3e92e0b6243b6fa33 Mon Sep 17 00:00:00 2001 From: Matthew Hammer Date: Tue, 4 Feb 2020 15:01:53 -0700 Subject: [PATCH 10/14] JSON canister names agree with those used in test-script files --- docs/design/dfx-testing-story-example/dfx.json | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/design/dfx-testing-story-example/dfx.json b/docs/design/dfx-testing-story-example/dfx.json index e30fa79233..1c1d4d664b 100644 --- a/docs/design/dfx-testing-story-example/dfx.json +++ b/docs/design/dfx-testing-story-example/dfx.json @@ -15,15 +15,15 @@ "AliceBob": { "_comment:": "test-script installs and scripts bots `alice` and `bob`" "test-script": "test/aliceBob/test-script" - "alice": "test/aliceBob/aliceCanister.mo" - "bob": "test/aliceBob/bobCanister.mo" + "aliceCanister": "test/aliceBob/aliceCanister.mo" + "bobCanister": "test/aliceBob/bobCanister.mo" } "AliceBobCharlie": { "_comment:": "reuse bots `alice` and `bob`, and includes bot `charlie`" "test-script": "test/aliceBobCharlie/test-script" - "alice": "test/aliceBob/aliceCanister.mo" - "bob": "test/aliceBob/bobCanister.mo" - "charlie": "test/aliceBobCharlie/charlieCanister.mo" + "aliceCanister": "test/aliceBob/aliceCanister.mo" + "bobCanister": "test/aliceBob/bobCanister.mo" + "charlieCanister": "test/aliceBobCharlie/charlieCanister.mo" } } }, From 10df321a998fe657fe29f45afc122df86e79b175 Mon Sep 17 00:00:00 2001 From: Matthew Hammer Date: Tue, 4 Feb 2020 15:15:31 -0700 Subject: [PATCH 11/14] clean up definitions for test categories --- docs/design/dfx-testing-story.adoc | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/docs/design/dfx-testing-story.adoc b/docs/design/dfx-testing-story.adoc index 8cb05e5473..6302536188 100644 --- a/docs/design/dfx-testing-story.adoc +++ b/docs/design/dfx-testing-story.adoc @@ -4,24 +4,38 @@ Creator(s): Matthew Hammer, after discussion with and input from Hans Larsen, on ### Terms and goals + - unit tests --- one canister for all the unit tests, together; + a standard library provides the framework for this canister (TBD). + + - functional tests --- one canister for each test, built and run separately. + + - e2e tests --- many canisters each; each requires a `test-script` to coordinate. + + In CI, we want to write, as humans and as CI scripts: `dfx test` -And have this statement mean two things at once: - - - "Run all end-to-end tests for the project" (`dfx test e2e` for _just_ this behavior). +And variations that permit more control over what tests run: - - "Run all unit tests for all libraries used by this project" (`dfx test unit` for _just_ this behavior. we + - `dfx test unit` for "Run all unit tests for all libraries used by this project" (we currently recompile these transitive libraries ourselves, so this "transitive testing" is a reasonable thing to do, too). - - "Run all functional tests for all libraries used by this project" - (for tests that are between unit tests and e2e tests in complexity; see the example project tree.) + - `dfx test func` for "Run all functional tests". + + - `dfx test func X` for "Run only the functional test named X". + + - `dfx test e2e` for "Run all end-to-end tests for the project". + + - `dfx test e2e X` for "Run only end-to-end test named X". + +We consider e2e tests and unit tests in the parts below. +These classes of tests are quite different in their requirements of +the `dfx` tool. The functional tests provide a common half-way point; +see the example project tree for more details. -These classes of tests are quite different in their requirements -of the `dfx` tool. We consider the first two separately as parts 1 and 2 below. ### Part 1: Authoring a CI-driven, end-to-end (e2e) test of a Motoko project From 58b05fab905d5a212b2a02d062f73749ba3b3b82 Mon Sep 17 00:00:00 2001 From: Matthew Hammer Date: Tue, 4 Feb 2020 15:37:13 -0700 Subject: [PATCH 12/14] nit --- docs/design/dfx-testing-story.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/dfx-testing-story.adoc b/docs/design/dfx-testing-story.adoc index 6302536188..020b94d4c4 100644 --- a/docs/design/dfx-testing-story.adoc +++ b/docs/design/dfx-testing-story.adoc @@ -216,7 +216,7 @@ install and call these canisters: dfx-testing-story-example ├── dfx.json ├── src -│ ├── main.mo +│ └── main.mo └── test ├── aliceBob │ ├── aliceCanister.mo From 315a202a3b3ebff34d384f5e1e0ff32ab4284520 Mon Sep 17 00:00:00 2001 From: Matthew Hammer Date: Tue, 4 Feb 2020 15:40:52 -0700 Subject: [PATCH 13/14] add (empty) example files so that the functional tests make more sense --- docs/design/dfx-testing-story-example/dfx.json | 2 +- docs/design/dfx-testing-story-example/src/adapton.mo | 1 + docs/design/dfx-testing-story-example/src/cleanSheets.mo | 1 + docs/design/dfx-testing-story-example/src/eval.mo | 1 + docs/design/dfx-testing-story-example/src/types.mo | 1 + 5 files changed, 5 insertions(+), 1 deletion(-) create mode 100644 docs/design/dfx-testing-story-example/src/adapton.mo create mode 100644 docs/design/dfx-testing-story-example/src/cleanSheets.mo create mode 100644 docs/design/dfx-testing-story-example/src/eval.mo create mode 100644 docs/design/dfx-testing-story-example/src/types.mo diff --git a/docs/design/dfx-testing-story-example/dfx.json b/docs/design/dfx-testing-story-example/dfx.json index 1c1d4d664b..3eb81cbc4d 100644 --- a/docs/design/dfx-testing-story-example/dfx.json +++ b/docs/design/dfx-testing-story-example/dfx.json @@ -3,7 +3,7 @@ "cleanSheets": { "_comment:": "we list `main` service code versus its `unit-tests` (each is a single Motoko function to run) versus its `functional-tests` (each is an `install-and-go`), versus `e2e-tests` that each involve more than a single `install-and-go`, and require a `test-script` over several canisters." - "main": "src/main.mo" + "main": "src/cleanSheets.mo" "unit-tests": { "all": "test/main.mo" }, diff --git a/docs/design/dfx-testing-story-example/src/adapton.mo b/docs/design/dfx-testing-story-example/src/adapton.mo new file mode 100644 index 0000000000..764f9f5596 --- /dev/null +++ b/docs/design/dfx-testing-story-example/src/adapton.mo @@ -0,0 +1 @@ +(OMITTING EXAMPLE CODE) diff --git a/docs/design/dfx-testing-story-example/src/cleanSheets.mo b/docs/design/dfx-testing-story-example/src/cleanSheets.mo new file mode 100644 index 0000000000..764f9f5596 --- /dev/null +++ b/docs/design/dfx-testing-story-example/src/cleanSheets.mo @@ -0,0 +1 @@ +(OMITTING EXAMPLE CODE) diff --git a/docs/design/dfx-testing-story-example/src/eval.mo b/docs/design/dfx-testing-story-example/src/eval.mo new file mode 100644 index 0000000000..764f9f5596 --- /dev/null +++ b/docs/design/dfx-testing-story-example/src/eval.mo @@ -0,0 +1 @@ +(OMITTING EXAMPLE CODE) diff --git a/docs/design/dfx-testing-story-example/src/types.mo b/docs/design/dfx-testing-story-example/src/types.mo new file mode 100644 index 0000000000..764f9f5596 --- /dev/null +++ b/docs/design/dfx-testing-story-example/src/types.mo @@ -0,0 +1 @@ +(OMITTING EXAMPLE CODE) From 672ace75159a09e18ca1ab60d96d66623a11a156 Mon Sep 17 00:00:00 2001 From: Matthew Hammer Date: Tue, 4 Feb 2020 15:42:32 -0700 Subject: [PATCH 14/14] add new files to tree listed in doc --- docs/design/dfx-testing-story.adoc | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/docs/design/dfx-testing-story.adoc b/docs/design/dfx-testing-story.adoc index 020b94d4c4..263c675a2f 100644 --- a/docs/design/dfx-testing-story.adoc +++ b/docs/design/dfx-testing-story.adoc @@ -216,15 +216,21 @@ install and call these canisters: dfx-testing-story-example ├── dfx.json ├── src -│ └── main.mo +│   ├── adapton.mo +│   ├── cleanSheets.mo +│   ├── eval.mo +│   ├── main.mo +│   └── types.mo └── test ├── aliceBob - │ ├── aliceCanister.mo - │ ├── bobCanister.mo - │ └── test-script - └── aliceBobCharlie - ├── charlieCanister.mo - └── test-script + │   ├── aliceCanister.mo + │   ├── bobCanister.mo + │   └── test-script + ├── aliceBobCharlie + │   ├── charlieCanister.mo + │   └── test-script + ├── simpleAdaptonDivByZero.mo + └── simpleExcelEmulation.mo ``` The tree contains two example test scripts, for a 2-party (2-canister)