Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exception on malformatted unit tests in YMLs #300

Merged
merged 5 commits into from
Oct 15, 2024

Conversation

pyth0n1c
Copy link
Contributor

@pyth0n1c pyth0n1c commented Oct 4, 2024

This PR includes two additional fixes:

  1. improve regex and logic used to search for macros in SPL. Previously, a comment with a newline in it (or other edge cases) could be parsed as a macro when it should have been parsed as a comment. This resulted in some comments appearing to be "macros not found in contentctl macros/ folder"
  2. When testing under --mode changes, an individual piece of content could be erroneously.tested multiple times if it met several conditions. Logic has been updated (by tracking changes in a set rather than a list) so that content cannot be added more than once to the group of content that must be tested in a single test run.
  • It was new content not in the target branch
  • It had been changed locally, but the changes had not yet been committed.

Raise exception on parse of unittest from yml.
Do this rather than trying to convert it into an integrationtest or manualtest.
For example, the following YML section is NOT if the file test_data.log does not exist:

tests:
- name: True Positive Test Which is not valid but fails silently
  attack_data:
  - data: data/data_file_that_does_not_exist.log
    sourcetype: anySourcetype
    source: anySource.log

Because the typing of detection.tests is as follows (since additional tests will be created at runtime):

tests: List[Annotated[Union[UnitTest, IntegrationTest, ManualTest], Field(union_mode='left_to_right')]] = []

if it fails to parse from YML as UnitTest, it will then try to parse as the other types (which may pass - but we do not want this behavior).
https://github.com/splunk/contentctl/blob/main/contentctl/objects/unit_test.py

For example, this will construct a VALID Integration Test (which is not desired behavior):
https://github.com/splunk/contentctl/blob/main/contentctl/objects/integration_test.py

This PR cause a ValueError to be generated if the data read from the YML cannot be used to create a valid UnitTest.

pyth0n1c and others added 5 commits October 4, 2024 16:31
…ying to convert it into an integrationtest or manualtest.
content that has already been committed
to the current branch AND there are local,
uncommitted changes to that content,
GitService will pick up BOTH the fact that
this is new content AND the fact that it has been
modified. This will result in double-testing
the content. This commit removes that as
a possibility by adding content to be tested
to a SET instead of appending it to a LIST,
which couild have included duplicates.
detection. Throw an error when four
or more ```` appear in a row in the
search field, which is invalid SPL.
Copy link
Contributor Author

@pyth0n1c pyth0n1c left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are fairly minor changes - I have reviewed them and they look good!

@pyth0n1c pyth0n1c merged commit 02eb5d7 into main Oct 15, 2024
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant