Implementation Agnostic Testing

[nikitawootten-nist]

Committer Notes

Fixes #1771

Side-effects:

• Fixed typo in spec, added missing @level to a few spec requirements
• Removed extra copy of metaschema datatypes xsd

All Submissions:

• Have you selected the correct base branch per Contributing guidance?
• Have you set "Allow edits and access to secrets by maintainers"?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?
• Have you squashed any non-relevant commits and commit messages? [instructions]
• Do all automated CI/CD checks pass?

By submitting a pull request, you are agreeing to provide this contribution under the CC0 1.0 Universal public domain dedication.

(For reviewers: The wiki has guidance on code review and overall issue review for completeness.)

Changes to Core Features:

• Have you added an explanation of what your changes do and why you'd like us to include them?
• Have you written new tests for your core changes, as applicable?
• Have you included examples of how to use your new feature(s)?
• Have you updated all OSCAL website and readme documentation affected by the changes you made? Changes to the OSCAL website can be made in the docs/content directory of your branch.

[nikitawootten-nist]

It seems this PR's develop has diverged from usnistgov's develop. I will cherry pick the relevant commits out of this branch and continue work on it in the upcoming week.

Edit: it seems fixing the history was simple.

[nikitawootten-nist]

Progress update: I have implemented a "coverage" subcommand to show the spec test coverage against the spec document.

[nikitawootten-nist]

Progress update: The tester is now runnable, however I do not have a working test file yet.

Below is a screenshot of the report generated by the script:

The script returns a list of XPath-like selections that yielded incorrect results.

[wendellpiez]

Eager to see this work! very cool.

[aj-stein-nist]

Non-blocking: as an aside, did you consider/review the Test Anything Protocol (TAP) or any similar efforts while designing the MVP JSON test data format you are proposing?

[nikitawootten-nist]

I have not heard of this protocol, though it looks like compelling follow-up work. The format seems simple enough to adapt the current output to (perhaps behind a CLI flag). The current output format is arbitrarily chosen as it's "easy enough" to read both as a human and to grep for basic automation.

[Compton-US]

I think this is worth moving forward and trying, noting the comment that the JSON schema for the tests would be a future specification.

[wendellpiez]

@Compton-NIST how about a little metaschema for the test harness format? We don't really need an XML capability (since XSLT can read JSON if need be) but it might be a fun little exercise.