-
-
Notifications
You must be signed in to change notification settings - Fork 545
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Replace optional key with scenarios #1677
Conversation
177744b
to
efbc413
Compare
6a05134
to
e8b6e4e
Compare
@@ -102,7 +102,7 @@ | |||
, "required" : ["description", "cases"] | |||
, "properties" : | |||
{ "description": { "$ref": "#/definitions/description" } | |||
, "optional" : { "$ref": "#/definitions/optional" } | |||
, "scenarios" : { "$ref": "#/definitions/scenarios" } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For now, I've allowed the possiblity of also having scenarios at the test group level.
"scenario": | ||
{ "description": "An identifier for a specific scenario (kebab-case)" | ||
, "type": "string" | ||
, "enum": ["big-integer", "floating-point", "library-test", "unicode"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've taken the liberty of adding the actual values here, which makes the JSON schema validation a lot nicer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems easy to miss when adding a new scenario. Could we somehow merge both lists?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought of maybe having a CI check to verify that the two match? The benefit of the file is that is easier to read. This has the benefit of actually verifying the actual scenario values.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we regenerate the schema when the file changes?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But a check works too, this isn't blocking
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we regenerate the schema when the file changes?
We could. Good idea.
38a66e8
to
929c3d1
Compare
Co-authored-by: Victor Goff <[email protected]>
Thanks for the reviews! |
* resistor-color-duo: add test case to ignore additional colors (exercism#1569) resistor-color-duo: ignore additional colors * [Space Age] Put Earth in it's proper order - Move Earth into its proper position in the solar system - Make wording more consistent * Remove commas from seconds amount Co-Authored-By: Victor Goff <[email protected]> * Add data structures to topic list We have a number of exercises that are about data structures, but the topic was missing. * Scrabble is a trademark name. It should be capitalized. It is also a noun, meaning a "doodle", and a verb, such as what this description could be considered. In our case, we are referring to the trademarked name of the game Scrabble. Signed-off-by: Victor Goff <[email protected]> * Add testcase * Matrix: Add symmetric test cases for non-square matrices * matrix: fix whitespace issue (exercism#1578) * matrix: fix whitespace issue Fix whitespace issue introduced by exercism#1576, closes $1577. * Matrix: bump version number. * leap: Add years that refute some unusual factors (exercism#1581) leap 1.6.0 Some implementations unexpectedly pass the entire test suite: * Replacing 4 with 499, 998, or 1996 * Replacing 100 with 5, 10, or 20 * Replacing 100 with 3, 6, 12, 15, 30, 60, 75, 150, 300 * Replacing 400 with 125, 250, 500, 1000, or 2000 Adding these test cases would correctly point out that these solutions are incorrect. I think it's a bit unusual since no student is going to write such an implementation except a student deliberately trying to slip past the tests. Or maybe a student trying to micro-optimise? Maybe they are trying to test whether division by small numbers is faster than division by large numbers? But discussion participants have deemed that the cost of three tests is worth the benefit of reducing mental overhead of mentors, since such solutions have in fact been seen in the wild. * Add micro-blog exercise (exercism#1509) * Add micro-blog exercise This is an exercise requiring students to truncate unicode strings. Solves exercism#1507 * Micro-blog: Don't assume native English speaker Thank you @SaschaMann for the feedback and suggestion. exercism#1509 (comment) > I don't like that this assumes the perspective of a native English > speaker. English is a foreign language to most of the world. Perhaps > something along the lines of "text in most of the world's languages and > scripts" would be a better description. * Micro-blog: Add tests for different languages Feedback from @SaschaMann exercism#1509 (comment) > I think it would be nice to add some test cases that aren't emoji or > English - perhaps cases with germanic umlauts, cyrillic and/or greek > letters, historic scripts etc. - because that's one of the main uses > and goals of unicode. I've added German, Bulgarian, and Greek examples. All of them have non-English characters. None of these characters use multiple UTF-16 codepoints. As such, if you use a UTF-8 programming language you may first have trouble with the German example, but if you use a UTF-16 language you will probably first have trouble at the Emoji example. I chose not to add an example with historic scripts, because I'm not aware of any that display nicely in my terminal or text-editor. Perhaps in future some could be added. I wanted another example that would be problematic in UTF-16, so I added a poker hand example using playing cards. * Micro-blog: Add German truncated example Comically, it goes from "bear carpet" to "beards". @SaschaMann, thank you for finding the example for me: exercism#1509 (comment) * Micro-blog: Add longer maths example Empty set is a proper subset of the natural numbers which is a proper subset of the integers, which is a proper subset of the rational numbers which is a proper subset of the reals which is a proper subset of the complex numbers. It remains true when truncated which is quite nice * paasio: Title as "PaaS I/O" instead of default "Paasio" (exercism#1589) Improves the copy only, so acceptable in light of exercism#1560. * Fix broken link to website contributing document * topics: add 'pointers' in section 'data types' * robot-name description: singular "robot" Use singular forms consistently throughout robot-name/description.md * connect: Replaced rhombus with parallelogram Replaced "rhombus" with "parallelogram" to clarify that the game's board height and width must not be the same. Closes exercism#1597 * minesweeper: remove border (exercism#1602) * minesweeper: remove border Updates the board example to remove the need for a border that doesn't match the test data and clarifies a couple small points. * minesweeper: fix characters Mispelled "asterisk" and didn't use the same character as `diamond` for the blank spaces. * minesweeper: use mine counts Updates "score totals" -> "mine counts". * list-ops: change order in append test case description (exercism#1611) * Change order in append test case description * Bump patch number * isbin-verifier: add EOF newline (exercism#1616) This fix the issue that `configlet generate .` is not idempotent as the track's README might manually adjust for the newline instead. This is something I came across while working on exercism/julia#161. * exercism#1623 `grade-school` canonical data does not correspond to the exercise description (exercism#1624) - case description is altered according to the discussion results - patch version of the canonical-data is updated * fix: Remove trailing spaces I have confirmed that no JSON version change is needed * Bob: Cleans up language on a couple of test cases Resolves exercism#1630 * Bob: Replaces DMV with dentist (exercism#1632) Resolves exercism#1630 Replaces ambiguous `DMV` with `dentist` and `NASA`. Co-authored-by: Peter Goodspeed-Niklaus <[email protected]> * luhn: check a number with an even remainder (exercism#1635) * test(luhn): check a long number with an even remainder * fix(luhn): minor version bump instead of a patch * [two-bucket] Make valid moves clearer (exercism#1644) There has been confusion over the reuse of the word "one" in both the descriptions of the bucket ("Bucket one") and the moves ("Move from one bucket"). This removes that confusion, which will help reduce misunderstanding, especially amongst non-native English speakers. * scale-generator: fix name of augmented interval (exercism#1643) The interval described is an augmented *second*, not an augmented first. Added also its composition in terms of steps for completeness. Fixes exercism#1642 Signed-off-by: Rafael Fonseca <[email protected]> * Add requirements of the exercise (exercism#1645) * two-fer: Update description.md (exercism#1583) * Update description.md I don't think it's great anyways but I'm tired of seeing students submit X as the variable name because of picking up some subliminal hint here. * Update exercises/two-fer/description.md Co-Authored-By: Victor Goff <[email protected]> Co-authored-by: Victor Goff <[email protected]> * Update Simple Linked List metadata (exercism#1648) Previous link was no longer working properly, so use web.archive.org need this before merging exercism/rust#935 * Fix metadata source_url for simple-linked-list exercise (exercism#1649) Previous web.archive.org link pointed to the book homepage. Change it to point to simple-linked-list page. * Fix apophenia error slowing student progress (exercism#1650) * Fix apophenia error slowing student progress apophenia - noun "the tendency to perceive a connection or meaningful pattern between unrelated or random things" compare: pareidolia - noun "to perceive a specific, often meaningful image in a random or ambiguous visual pattern" - Merriam Webster's * Fix premature doubling in 1st example for Luhn * Update description.md (exercism#1653) The test suite is testing against the key having only lowercase letters, not alphanumeric. * two tests to check case insensitive behavior (exercism#1658) * Revert "two tests to check case insensitive behavior (exercism#1658)" (exercism#1659) This reverts commit ea9db9b. * Palindrome Products: refine the problem (exercism#1662) For me it took a while to understand that we calculate product of exactly two numbers. BTW in the [original problem](https://projecteuler.net/problem=4) it is stated clearly. * robot-name: remove dead link (exercism#1663) gSchool.it is now a domain parking spot * Add uuid field to all test cases (exercism#1676) * Add uuid field to documentation * Update canonical schema * Add uuid field to test cases * Update canonical-schema.json Co-authored-by: Sascha Mann <[email protected]> * Update comments in README * Remove version check Co-authored-by: Sascha Mann <[email protected]> * Remove version (exercism#1678) * Replace optional key with scenarios (exercism#1677) * Replace optional key with scenarios Co-authored-by: Victor Goff <[email protected]> * Add GH Actions CI workflow (exercism#1680) * Add GH Actions CI workflow * install yarn dependencies * fix node-version field * Remove check_optional Co-authored-by: Jeremy Walker <[email protected]> * cache yarn dependencies * use node 12 * remove Travis CI config * Update .github/workflows/ci.yml Co-authored-by: Sascha Mann <[email protected]> Co-authored-by: Jeremy Walker <[email protected]> Co-authored-by: Sascha Mann <[email protected]> * Add description of tests.toml (exercism#1682) * Add description of tests.toml * Update CONTRIBUTING.md Co-authored-by: Jeremy Walker <[email protected]> Co-authored-by: Jeremy Walker <[email protected]> * Add scenarios to example and describe purpose (exercism#1683) * CI: Clean up workflow and clarify what "JS tests" are (exercism#1697) * Move required files check to separate job * Clarify what JS tests are * Rename schema-tests to schema-validation * CI: Add codeowners (exercism#1699) * CI: Add codeowners * Update .github/CODEOWNERS Co-authored-by: Jeremy Walker <[email protected]> Co-authored-by: Jeremy Walker <[email protected]> * Add reimplements to schema (exercism#1703) * Add reimplements to schema * Fix formatting * list-ops: add append case (exercism#1612) * Add case to append empty list to list * Add UUID Co-authored-by: wolf99 <[email protected]> * resistor-color/resistor-color-duo: extend exercise explanation (partial copy from resistor-color-duo) (exercism#1667) * CI: Check that UUIDs are unique (exercism#1700) * diffie-hellman: Reword 'in range 1 .. p' (exercism#1688) There are some languages where the syntax `1 .. p` includes both the endpoints. For example, the Nim code: ``` for i in 1 .. 3: echo i ``` shows the output: ``` 1 2 3 ``` So let's prefer the wording from the `description.md`: "Alice picks a private key, a, greater than 1 and less than p." * run-length-encoding: 'lower case' -> 'lowercase' (exercism#1708) This makes it more consistent with another test: "description": "lowercase characters", Both forms are correct, but "lowercase" is more common. See: https://english.stackexchange.com/questions/59409 * isbn-verifier: 'isbn number' -> 'isbn' (exercism#1690) Most of the test descriptions just used 'isbn', so let's be consistent and avoid "International Standard Book Number number". * perfect-numbers: 'natural number' -> 'positive integer' (exercism#1691) There is no convention whether zero is a natural number, so let's avoid the issue. For example, see: - https://mathworld.wolfram.com/NaturalNumber.html - https://en.wikipedia.org/wiki/Natural_number - https://math.stackexchange.com/questions/283/is-0-a-natural-number The previous wording of ``` "description": "Zero is rejected (not a natural number)", ``` could be especially jarring for some tracks. For example, Nim defines `Natural` and `Positive` types, and `Natural` includes 0. See: - https://nim-lang.github.io/Nim/system.html#Natural Co-authored-by: Sascha Mann <[email protected]> Co-authored-by: Sascha Mann <[email protected]> * grains: Be more verbose in test-level descriptions (exercism#1707) The top-level `description` of these cases was: ``` "description": "returns the number of grains on the square", ``` but each test-level `description` contained only a number. The comments in `tests.toml` are generated only from the test-level `description`, and so the comments there will now be more informative than the previously seen: ``` # 1 "9fbde8de-36b2-49de-baf2-cd42d6f28405" = true # 2 "ee1f30c2-01d8-4298-b25d-c677331b5e6d" = true ``` * Add Style Guide (exercism#1713) * Add Style Guide * Update STYLE-GUIDE.md Co-authored-by: Sascha Mann <[email protected]> * Update STYLE-GUIDE.md Co-authored-by: Sascha Mann <[email protected]> * Update STYLE-GUIDE.md Co-authored-by: Ryan Potts <[email protected]> * Update STYLE-GUIDE.md Co-authored-by: Sascha Mann <[email protected]> * Update STYLE-GUIDE.md * Update STYLE-GUIDE.md Co-authored-by: Sascha Mann <[email protected]> Co-authored-by: Colin Caine <[email protected]> Co-authored-by: Ryan Potts <[email protected]> * CI: Add check that reimplemented-values are valid UUIDs (exercism#1702) Co-authored-by: ee7 <[email protected]> * CI: Use jq instead of grep/sed (exercism#1710) * Add esoteric example. (exercism#1717) * Square-Root: Add new exercise (exercism#1582) * Add square root exercie * Cite myself in metadata * Add radicand explantion, use PR as source_url * Add UUIDs, remove version * Adjust description.md Co-authored-by: wolf99 <[email protected]> * CI: Add workflow_dispatch as manual trigger (exercism#1711) This is useful for debugging CI or when working on a branch before opening a PR * security: CVE-2020-7598 (exercism#1706) * ⬆️ latest ajv-cli * 🔒 CVE-2020-7598 * Add abbreviations and restructure slightly (exercism#1718) * Add abbreviations and restructure slightly * Update STYLE-GUIDE.md * Update with better text * Update STYLE-GUIDE.md Co-authored-by: Colin Caine <[email protected]> * Update STYLE-GUIDE.md Co-authored-by: Colin Caine <[email protected]> Co-authored-by: Derk-Jan Karrenbeld <[email protected]> * Style: Fix whitespace issues (exercism#1720) * Style: Remove trailing whitespace With this commit, people who run `configlet generate` and commit the generated READMEs as-is will no longer introduce trailing whitespace into their track repos. * Style: Add missing final newlines * grade-school: Fix typo in test description (exercism#1689) * queen-attack: Fix typo in test description (exercism#1692) * Update nucleotide-count description (exercism#1719) See also exercism#1716. Thanks to SleeplessByte, rpottsoh, kotp, ErikSchierboom, etc :) * CI: Add /rebase command (exercism#1698) (stolen from v3 repo) * Add workflow recommendations and templates (exercism#1722) Adds a document that explains how to set up Continuous Integration (CI) workflows for an Exercism Language Track using GitHub Actions (GHA). It provides best practices and examples for you to use to make your own fast, reliable and robust CI workflows. Additionally, it provides GHA workflows. The GHA workflows can be adapted to work with any CI, track, or project, because the base structure will remain the same. * CI: Ensure immutability of test cases (exercism#1712) * CI: Check immutability of test data * Iterate old cases instead * Add to ci.yml workflow * Fix path to python script in workflow * Set chmod=+x for python script * Apparently latest does not mean latest * Fix path * Test mutated test in PR * Revert "Test mutated test in PR" This reverts commit 18dbfa2. * Fix build for workflow_dispatch triggers * Break out of loop early on failure * Fix typo * Handle removal of tests better * Add scenarios check * Update bin/check-immutability.py * Make variable names longer & use sys.exit instead of exit Co-authored-by: BethanyG <[email protected]> Co-authored-by: Corey McCandless <[email protected]> Co-authored-by: BethanyG <[email protected]> Co-authored-by: Corey McCandless <[email protected]> * Add test data for future edits to the script Co-authored-by: BethanyG <[email protected]> Co-authored-by: Corey McCandless <[email protected]> * rational-numbers: Make formulas more readable (exercism#1655) * rational-numbers: Make formulas more readable * Remove superscript characters * Readd parentheses * leap: fix typo (exercism#1726) * rational-numbers: Remove redundant factor (exercism#1727) If `b₁ = 0`, then `r₁` is not a rational number anyway. From https://github.com/exercism/problem-specifications/pull/1655/files#r421573667 * two-bucket: test inability to reach the goal (exercism#1580) 1. A test where the goal is too large. The student solution would need to either: * (If searching the state space) Notice that there are no further states to be visited, and yet the solution has not been reached. * Notice that the goal is larger than the larger bucket, therefore can be rejected immediately. 2. A test where the goal is not too large yet still can't be reached The student solution would need to either: * (If searching the state space) Notice that there are no further states to be visited, and yet the solution has not been reached. * Notice that the goal is not divisible by the GCD of the bucket sizes, therefore can be rejected immediately. In case the student assumes that all non-coprime bucket counts will invalidate the goal, a counterexample to that is given as well (buckets not coprime but goal is still possible). There are ten implementing tracks: bash csharp fsharp go java javascript python ruby rust typescript Of these tracks, only two of them (Bash, Go) currently test the condition where it is not possible to reach the goal. Having this test serves as a reminder that it remains wise to handle the situation where a search has not found its goal. It doesn't seem like this was discussed in the original submission: exercism/DEPRECATED.javascript#68 So it seems like it would be good to have a discussion of it on record. * [CI] Bump rebase action to fix CVE-2020-15228 (exercism#1731) * [CI] Only run immutability check on PRs (exercism#1730) Let's remove this until exercism#1728 has been resolved properly to avoid build errors on master. * Remove linkless words (exercism#1733) * build(deps): bump ini from 1.3.5 to 1.3.7 (exercism#1734) Bumps [ini](https://github.com/isaacs/ini) from 1.3.5 to 1.3.7. - [Release notes](https://github.com/isaacs/ini/releases) - [Commits](npm/ini@v1.3.5...v1.3.7) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Grade school exercise: A student cannot simultaneously be in two grades since the users' names are unique (exercism#1735) * Add configlet (exercism#1737) * change: add simplest test case (exercism#1739) * prime-factors: add further variations (exercism#1755) * [transpose] added 'jagged triangle' test (exercism#1748) * kindergarten-garden: completionist (exercism#1744) * list-ops: reimplement ambiguous tests (exercism#1746) * Format * Re-implement ambigious test cases * Clarifying comment * Re-add old versions of reimplemented tests * Remove accidantally commited executable * Clarifying commentary about integer division. * Add advice * Hamming: remove tests that don't make sense per 1761 (exercism#1762) * Hamming: remove tests that dont make sense * return tests & normalise error messages * grade-school: Change a UUID to be version 4 (exercism#1760) Commit 03529d0 (exercism#1735) added a new test case for the `grade-school` exercise, but the UUID added was version 1 rather than version 4. This was not caught by CI because the regex in the schema validator was too permissive (fixed by exercism#1759). Some reasons to prefer version 4 UUIDs: - Version 4 is the right version for a fully random and unique identifer; we don't want to indicate anything about the machine that generated the UUID, and version 1 UUIDs have a timestamp and MAC address component. - All of the other UUIDs currently in `problem-specifications` are valid version 4 UUIDs. - `configlet uuid` generates a version 4 UUID. - A reader familiar with the UUID specification (RFC 4122) might see a version 1 UUID and infer that the timestamp or MAC address component is useful, causing them to wonder why the others are version 4. - A user who generates a version 1 UUID might unintentionally leak their MAC address. See also: - https://tools.ietf.org/html/rfc4122.html * canonical-schema: Fix UUID regex pattern (exercism#1759) Commit cea02af (exercism#1676) added a UUID to each test case. However, the regex pattern that it added to the canonical schema was too permissive, meaning that CI would pass on a PR that added, for example, a version 1 UUID (see exercism#1735). Changes: - Use `a-f` instead of `a-z` - The third group must start with `4`. - The fourth group must start with `8`, `9`, `a` or `b`. * [CI] Verify that scenarios are defined in schema Co-authored-by: Josh Goebel <[email protected]> Co-authored-by: Victor Goff <[email protected]> Co-authored-by: Victor Goff <[email protected]> Co-authored-by: Katrina Owen <[email protected]> Co-authored-by: wolf99 <[email protected]> Co-authored-by: DagmarTimmreck <[email protected]> Co-authored-by: Michael Morehouse <[email protected]> Co-authored-by: Peter Tseng <[email protected]> Co-authored-by: Chris Couzens <[email protected]> Co-authored-by: Sam Warner <[email protected]> Co-authored-by: Gabriel Nelle <[email protected]> Co-authored-by: Chris White <[email protected]> Co-authored-by: traxam <[email protected]> Co-authored-by: ShaoWei Teo <[email protected]> Co-authored-by: Elena Parovyshnaia <[email protected]> Co-authored-by: Florian Keller <[email protected]> Co-authored-by: Ryan Potts <[email protected]> Co-authored-by: Ryan Potts <[email protected]> Co-authored-by: Peter Goodspeed-Niklaus <[email protected]> Co-authored-by: Pranas Ziaukas <[email protected]> Co-authored-by: Jeremy Walker <[email protected]> Co-authored-by: Rafael F <[email protected]> Co-authored-by: Zuzanna Kru <[email protected]> Co-authored-by: Bruce Park <[email protected]> Co-authored-by: Tejas Bubane <[email protected]> Co-authored-by: Jubilee <[email protected]> Co-authored-by: Charles Ross <[email protected]> Co-authored-by: Ole Kröger <[email protected]> Co-authored-by: DmitrySamoylov <[email protected]> Co-authored-by: Colin Caine <[email protected]> Co-authored-by: Sascha Mann <[email protected]> Co-authored-by: Corey McCandless <[email protected]> Co-authored-by: wolf99 <[email protected]> Co-authored-by: wolf99 <[email protected]> Co-authored-by: Kirill Artamonov <[email protected]> Co-authored-by: ee7 <[email protected]> Co-authored-by: Derk-Jan Karrenbeld <[email protected]> Co-authored-by: Derk-Jan Karrenbeld <[email protected]> Co-authored-by: BethanyG <[email protected]> Co-authored-by: Yunseon Choi <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pedro Rolo <[email protected]> Co-authored-by: peerreynders <[email protected]>
This PR does not yet document the new
tests.toml
functionality and its tooling that we'll introduce to allow tracks to keep track of which tests are included/excluded.