Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check oscal-version against fedramp-version in digital authorization package #833

Open
14 tasks done
Tracked by #804
aj-stein-gsa opened this issue Oct 29, 2024 · 7 comments · May be fixed by #884
Open
14 tasks done
Tracked by #804

Check oscal-version against fedramp-version in digital authorization package #833

aj-stein-gsa opened this issue Oct 29, 2024 · 7 comments · May be fixed by #884

Comments

@aj-stein-gsa
Copy link
Contributor

aj-stein-gsa commented Oct 29, 2024

Constraint Task

As a digital authorization package maintainer, to have confidence that each document in the package conforms with the right version of FedRAMP requirements, I want each OSCAL document in the package to check //metadata/oscal-version aligns with //metadata/fedramp-version.

Intended Outcome

Goal

The constraint(s) will validate that the fedramp-version and oscal-version in the document conforms to the release strategy. (NOTE: As part of this effort, this task can be worked alongside the completion of GSA/automate.fedramp.gov#104, this issue has been updated to account for that change.)

Syntax

Write a constraint (id="oscal-version-matches-fedramp-version" with level="ERROR") that checks that each digit for the value of oscal-version (1.1.2) is greater than or equal to the version number embedded in the end of the prop[@name="fedramp-version"] and less than the major version (the first digit in a version, so oscal-version MUST be >=1.1.2 and < 2.0.0.

VALID:

<system-security-plan xmlns="http://csrc.nist.gov/ns/oscal/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" uuid="12345678-1234-4321-8765-123456789012">
  <metadata>
    <oscal-version>1.1.2</oscal-version>
    <prop name="fedramp-version" ns="http://fedramp.gov/ns/oscal" value="3.0.0"/>
  </metadata>
</system-security-plan>
<system-security-plan xmlns="http://csrc.nist.gov/ns/oscal/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" uuid="12345678-1234-4321-8765-123456789012">
  <metadata>
    <oscal-version>1.1.3</oscal-version>
    <prop name="fedramp-version" ns="http://fedramp.gov/ns/oscal" value="3.0.0"/>
  </metadata>
</system-security-plan>
<system-security-plan xmlns="http://csrc.nist.gov/ns/oscal/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" uuid="12345678-1234-4321-8765-123456789012">
  <metadata>
    <oscal-version>1.2.0</oscal-version>
    <prop name="fedramp-version" ns="http://fedramp.gov/ns/oscal" value="3.0.0"/>
  </metadata>
</system-security-plan>
<system-security-plan xmlns="http://csrc.nist.gov/ns/oscal/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" uuid="12345678-1234-4321-8765-123456789012">
  <metadata>
    <oscal-version>1.8.1</oscal-version>
    <prop name="fedramp-version" ns="http://fedramp.gov/ns/oscal" value="3.0.0"/>
  </metadata>
</system-security-plan>

INVALID:

<system-security-plan xmlns="http://csrc.nist.gov/ns/oscal/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" uuid="12345678-1234-4321-8765-123456789012">
  <metadata>
    <oscal-version>2.1.1</oscal-version>
    <prop name="fedramp-version" ns="http://fedramp.gov/ns/oscal" value="3.0.0"/>
  </metadata>
</system-security-plan>
<system-security-plan xmlns="http://csrc.nist.gov/ns/oscal/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" uuid="12345678-1234-4321-8765-123456789012">
  <metadata>
    <oscal-version>1.0.6</oscal-version>
    <prop name="fedramp-version" ns="http://fedramp.gov/ns/oscal" value="3.0.0"/>
  </metadata>
</system-security-plan>

Syntax Type

This is required core OSCAL syntax.

Allowed Values

There are no relevant allowed values.

Metapath(s) to Content

/(assessment-plan|assessment-results|plan-of-action-and-milestones|system-security-plan)/metadata/oscal-version
/(assessment-plan|assessment-results|plan-of-action-and-milestones|system-security-plan)/metadata/prop[@name="fedramp-version"]

NOTE: Updated due to relevant pairing feedback from @wandmagic and others: during profile resolution the original //metadata query MAY bring non-FedRAMP artifacts into scope, and in our use cases that MAY be profile or catalog, so we exclude them by absence for now.

Purpose of the OSCAL Content

The FedRAMP version prop added in #800 for #789 but this has a sub-part that indicates the minimal oscal-version required by FedRAMP, but does not cross-check the explicit use of oscal-version in an adjacent field.

Dependencies

No response

Acceptance Criteria

  • All OSCAL adoption content affected by the change in this issue have been updated in accordance with the Documentation Standards.
    • Explanation is present and accurate
    • sample content is present and accurate
    • Metapath is present, accurate, and does not throw a syntax exception using oscal-cli metaschema metapath eval -e "expression".
  • All constraints associated with the review task have been created
  • The appropriate example OSCAL file is updated with content that demonstrates the FedRAMP-compliant OSCAL presentation.
  • The constraint conforms to the FedRAMP Constraint Style Guide.
    • All automated and manual review items that identify non-conformance are addressed; or technical leads (David Waltermire; AJ Stein) have approved the PR and “override” the style guide requirement.
  • Known good test content is created for unit testing.
  • Known bad test content is created for unit testing.
  • Unit testing is configured to run both known good and known bad test content examples.
  • Passing and failing unit tests, and corresponding test vectors in the form of known valid and invalid OSCAL test files, are created or updated for each constraint.
  • A Pull Request (PR) is submitted that fully addresses the goals section of the User Story in the issue.
  • This issue is referenced in the PR.

Other information

No response

@aj-stein-gsa
Copy link
Contributor Author

@Gabeblis is working on a PR to develop for the web repo to update docs about this pending constraint logically evaluating prop[@name="fedramp-version"] and oscal-version (this comes after GSA/automate.fedramp.gov#94 was merged, so there needs to be an enhancement).

@aj-stein-gsa
Copy link
Contributor Author

@Gabeblis I actually updated the examples because the number logic for 2.1.1 as a possible valid update to 1.x.y, such 1.1.2 is misguided. The correct behavior is that this is not allowed, this would require FedRAMP to release new guidance.

I forget the key part of the semantic versioning guidance when I was trying to implement the logic, you update the major version (e.g. from 1.x.y -> 2.a.b or 2.a.b -> 3.c.d when there is an incompatible change to the API or we bump the the minor middle value. So when OSCAL changes, we should be mindful not to openly accept an OSCAL 2.x or 3.x, because that could be completely disjoint and the constraints "wrong" for that; in short the API would be different.

So, I am going to adjust back the changes I had not upstreamed yet, sorry for the confusion, I just wanted to update that I refreshed the issue, because you took the time to discuss it with me and I assume it will come back up in review.

@Gabeblis
Copy link
Contributor

I see.. that does make perfect sense. So will want to make sure that the value of the first digit is 1 (any higher would mean significant changes that could potentially make constraints invalid) and we'll keep the logic the same for the 2 trailing digits (middle and end). I noticed one thing..in the examples under syntax, it says check that the digitis in oscal-version are greater than or equal to the digits in fedramp-version, but the VALID example does not reflect this. Also, the same example is under both VALID and INVALID. Just a little confused by that.

@aj-stein-gsa
Copy link
Contributor Author

Thanks for the update. Also, message from the Big Giant Head: the release strategy I pointed to in the repo, ADR#2 is no longer valid and the canonical source has been the similar (by skimming) but very different release strategy on the website. The tags are confusing, but we should have moved to vanilla SemVer a little while ago.

So, the issue requirements require rework (no pun intended), so I will close the PR for now and put this into Backlog state. 😦

@aj-stein-gsa aj-stein-gsa moved this from 🏗 In progress to 📋 Backlog in FedRAMP Automation Oct 30, 2024
@aj-stein-gsa
Copy link
Contributor Author

So @Gabeblis, re #833 (comment), hold on the website update for now. Do not open that PR. We have some others to tighten up the release guidance, doubly so the GSA/fedramp-automation repo.

Apologies!

@aj-stein-gsa aj-stein-gsa linked a pull request Nov 15, 2024 that will close this issue
5 tasks
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 18, 2024
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
Thanks to @kyhu65867 for humoring me and and walking through concepts
that need better explanation for example here.
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
Shout out to @kyhu65867 for trying to warn me about this during a long
pairing session, but me missing the hint.
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
Add three different scenarios for failing tests to improve coverage for
the different error states.
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
Thanks to @kyhu65867 for humoring me and and walking through concepts
that need better explanation for example here.
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
Shout out to @kyhu65867 for trying to warn me about this during a long
pairing session, but me missing the hint.
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
Add three different scenarios for failing tests to improve coverage for
the different error states.
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Nov 19, 2024
@Gabeblis Gabeblis moved this from 🏗 In progress to 👀 In review in FedRAMP Automation Dec 3, 2024
@aj-stein-gsa aj-stein-gsa moved this from 👀 In review to 🏗 In progress in FedRAMP Automation Dec 4, 2024
@aj-stein-gsa
Copy link
Contributor Author

aj-stein-gsa commented Dec 4, 2024

Now that we have anonymous functions in m-j 2.0.7, I may heed Dave's advice from the standup call and give that whack.

So ... I'll be refactoring this PR.

aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Dec 5, 2024
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Dec 5, 2024
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Dec 5, 2024
aj-stein-gsa added a commit to kyhu65867/fedramp-automation that referenced this issue Dec 5, 2024
@aj-stein-gsa
Copy link
Contributor Author

Le sigh, I forgot last night that #833 (comment) is not viable since that functionality is merged into metaschema-framework/metaschema-java, but not oscal-cli. Woops! Back to the drawing board.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: 🏗 In progress
Development

Successfully merging a pull request may close this issue.

3 participants