Inconsistent behavior for information type security level constraints from use of | or operator

[aj-stein-gsa]

This relates to ...

• the FedRAMP OSCAL Validations

What happened?

We have a common pattern in NIST and FedRAMP's OSCAL constraints separately. For the FedRAMP constraints, the eager use of the | operator for grouping and testing of multiple roles is problematic. Today, we learned of inconsistency with how these behave, much like the legacy Schematron constraints did (because the underlying XPath 3 assumption is the same; I cannot find the exact relevant details in the W3C spec quickly).

fedramp-automation/src/validations/constraints/fedramp-external-allowed-values.xml

Lines 377 to 389 in 9d50039

	<context>
	<metapath target="/system-security-plan/system-characteristics/security-sensitivity-level"/>
	<metapath target="/system-security-plan/system-characteristics/security-impact-level/(security-objective-confidentiality|security-objective-integrity|security-objective-availability)"/>
	<metapath target="/system-security-plan/system-characteristics/system-information/information-type/(confidentiality-impact|integrity-impact|availability-impact)/(base|selected)"/>
	<constraints>
	<allowed-values id="security-level" target="." allow-other="no" level="ERROR">
	<formal-name>Security Impact Level</formal-name>
	<description>The security objective level as defined by <a href="https://doi.org/10.6028/NIST.SP.800-60v1r1">NIST SP 800-60</a>.</description>
	<enum value="fips-199-low">Low</enum>
	<enum value="fips-199-moderate">Moderate</enum>
	<enum value="fips-199-high">High</enum>
	</allowed-values>
	</constraints>

This issue is to address that with a bug fix here, then integrate changes to the style guide in #760 or elsewhere if appropriate to strictly limit the use of this construct for similar use cases by developers. See the repro bug branch for details.

Commit on repro branch: aj-stein-gsa@0248864
Failure on CI/CD run of full test suite: https://github.com/aj-stein-gsa/fedramp-automation/actions/runs/11282608164/job/31380414397

Thanks to @DimitriZhurkin for locating and reporting this bug to the group today with some questions inadvertently during standup.

Relevant log output

me@computer fedramp-automation % git branch --show-current
772-pipe-operator-bug-repro
me@computer fedramp-automation % git rev-parse HEAD       
0248864c3b8a05676ca88888d76bedbd66a12658
me@computer fedramp-automation % make init && make test
git submodule update --init --recursive
Installing node modules...
npm install

up to date, audited 371 packages in 633ms

60 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
npx oscal@latest use latest
Installing version: 2.2.0
Downloading OSCAL CLI...
Downloading version 2.2.0 from https://repo1.maven.org/maven2/dev/metaschema/oscal/oscal-cli-enhanced/2.2.0/oscal-cli-enhanced-2.2.0-oscal-cli.zip
Extracting OSCAL CLI...
Setting executable permissions for CLI at /Users/alexanderjstein/.asdf/installs/nodejs/20.17.0/lib/node_modules/oscal-cli/bin/oscal-cli
Creating OSCAL CLI symlink: oscal-cli => /Users/alexanderjstein/.asdf/installs/nodejs/20.17.0/lib/node_modules/oscal-cli/bin/oscal-cli
OSCAL CLI installed to /Users/alexanderjstein/.asdf/installs/nodejs/20.17.0/lib/node_modules/oscal-cli
Alias created at /Users/alexanderjstein/.asdf/installs/nodejs/20.17.0/bin/oscal-cli

up to date, audited 371 packages in 581ms

60 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
npx oscal@latest use latest
Installing version: 2.2.0
Downloading OSCAL CLI...
Downloading version 2.2.0 from https://repo1.maven.org/maven2/dev/metaschema/oscal/oscal-cli-enhanced/2.2.0/oscal-cli-enhanced-2.2.0-oscal-cli.zip
Extracting OSCAL CLI...
Setting executable permissions for CLI at /Users/alexanderjstein/.asdf/installs/nodejs/20.17.0/lib/node_modules/oscal-cli/bin/oscal-cli
Creating OSCAL CLI symlink: oscal-cli => /Users/alexanderjstein/.asdf/installs/nodejs/20.17.0/lib/node_modules/oscal-cli/bin/oscal-cli
OSCAL CLI installed to /Users/alexanderjstein/.asdf/installs/nodejs/20.17.0/lib/node_modules/oscal-cli
Alias created at /Users/alexanderjstein/.asdf/installs/nodejs/20.17.0/bin/oscal-cli
src/web/module.mk:10: *** node.js version `v20.16.0` required, `v20.17.0` installed.  Stop.
me@computer fedramp-automation % npm run constraint security-level

> [email protected] constraint
> node ./src/scripts/dev-constraint.js security-level

Found 64 constraints.
Selected 1 constraints for analysis.

Constraint Analysis and Test Execution:
security-level: Fully covered
//home/me/code/fedramp-automation/features/fedramp_extensions.feature {
  pass: 'security-level-PASS.yaml',
  fail: 'security-level-FAIL.yaml',
  fail_file: 'security-level-FAIL.yaml',
  pass_file: 'security-level-PASS.yaml'
} security-level
..Processing test case:Test Invalid Security Level Use
Description: Test that the specified security level is invalid.
Loaded content from: //home/me/code/fedramp-automation/src/validations/constraints/content/ssp-security-level-INVALID.xml
oscal-cli validate //home/me/code/fedramp-automation/src/validations/constraints/content/ssp-security-level-INVALID.xml --disable-schema-validation -c //home/me/code/fedramp-automation/src/validations/constraints/fedramp-external-allowed-values.xml -c //home/me/code/fedramp-automation/src/validations/constraints/fedramp-external-constraints.xml -o oscal-cli-sarif-log-19dc296f-cd7d-4f49-8710-9b907bd5f2e4.json --sarif-include-pass --show-stack-trace
Error: OSCAL CLI process exited with code 1:
The file 'file://home/me/code/fedramp-automation/src/validations/constraints/content/ssp-security-level-INVALID.xml' is invalid.

Checking status of constraint: security-level expecting: fail
Received: 13 matching mixed results (8 pass, 5 fail)
F

Failures:

1) Scenario: Validating OSCAL documents with metaschema constraints # features/fedramp_extensions.feature:142
   ✔ Given I have Metaschema extensions documents # file://home/me/code/fedramp-automation/features/steps/fedramp_extensions_steps.ts:86
       | filename                            |
       | fedramp-external-allowed-values.xml |
       | fedramp-external-constraints.xml    |
       | oscal-external-constraints.xml      |
   ✔ When I process the constraint unit test "security-level-FAIL.yaml" # file://home/me/code/fedramp-automation/features/steps/fedramp_extensions_steps.ts:94
   ✖ Then the constraint unit test should pass # file://home/me/code/fedramp-automation/features/steps/fedramp_extensions_steps.ts:103
       AssertionError: Negative test failed: Test failed with the following errors:
       security-level: invalid results received. 62% passing. Expected: pass_count "none", fail_count "all". Actual: 8 pass, 5 fail.
       : expected 'fail' to equal 'pass'
           + expected - actual
       
           -fail
           +pass
       
           at World.<anonymous> (file:////home/me/code/fedramp-automation/features/steps/fedramp_extensions_steps.ts:110:34)
           at fulfilled (file:////home/me/code/fedramp-automation/features/steps/fedramp_extensions_steps.ts:4:58)
           at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

1 scenario (1 failed)
3 steps (1 failed, 2 passed)
0m06.805s (executing steps: 0m06.795s)
Cucumber test for security-level failed: Command failed: NODE_OPTIONS="--loader ts-node/esm --no-warnings --experimental-specifier-resolution=node" npx cucumber-js "//home/me/code/fedramp-automation/features/fedramp_extensions.feature:142"
security-level: Test failed
me@computer fedramp-automation % 

How do we replicate this issue?

Where, exactly?

In constraints processing with constraint of ID security-level, but I have seen this pattern used frequently elsewhere.

Other relevant details

Need to sort out best practices rules on use of the rules with combinatoric use of group | sets with @david-waltermire.

[aj-stein-gsa]

OK I learned I was very wrong about this issue yesterday, and more to follow. This issue is not with the paths, the failure is occurring with schema enforcement. Takeaways from this:

• It appears there are other issues afoot, but when running this interactively with the CLI, I was not disabling the schema generation argument, like the test harness, but then the constraint properly fires. It does have other behavior with impacts, detailed below:
  • The Metapaths are correct but actually have a symptom in creating multiple SARIF reports for a rule, one for each Metapath target for a given rule, for one error firing correctly (so for this example, there are three). This behavior will be reported in metaschema-java, this probably has negative impacts on the test harness. /cc @david-waltermire (Addressed in SARIF output from validation of constraints duplicate error reports metaschema-framework/metaschema-java#184)
  • In the interim, we should put refactor the targets to not have repeat rule reports per error. (Ongoing, but also more tightly managed per style guide requirements in 675)
  • Our test definition in YAML could support more precision on counts and/or message output or paths, because we may be too strict or too permissive on tests. /cc @wandmagic

So in the interim, we can refactor the test and improve the test harness it seems.