[FR] Add Ability to Filter Rule Exports from Kibana

[eric-forte-elastic]

Pull Request

Issue link(s):
Resolves #4768

Summary - What I changed

I added the ability to filter the output of the export rules kibana API call. This was accomplished using the bulk actions API Endpoint's query parameter. A small update was required to the Kibana library to enable using this feature.

New parameters to export rules: --custom-rules-only/-cro and --export-query/-eq

How To Test

To test this use the following command with a test stack to try and export only custom rules
python -m detection_rules kibana export-rules -d dac_test/rules -sv -ac -e -cro

Example Output

detection-rules on  4768-fr-filter-prebuilt-rules-in-kibana-export-command [!?] is  v1.2.12 via  v3.12.11 (detection-rules-build) on  eric.forte took 2s 
❯ python -m detection_rules kibana export-rules -d dac_test/rules -sv -ac -e -cro
Loaded config file: /home/forteea1/Code/dac_demo/test/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

2 results exported
1 rules converted
0 exceptions exported
0 action connectors exported
1 rules saved to dac_test/rules
0 exception lists saved to /home/forteea1/Code/dac_demo/test/detection-rules/dac_test/exceptions
0 action connectors saved to /home/forteea1/Code/dac_demo/test/detection-rules/dac_test/action_connectors

detection-rules on  4768-fr-filter-prebuilt-rules-in-kibana-export-command [!?] is  v1.2.12 via  v3.12.11 (detection-rules-build) on  eric.forte took 2s 
❯ python -m detection_rules kibana export-rules -d dac_test/rules -sv -ac -e
Loaded config file: /home/forteea1/Code/dac_demo/test/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

1429 results exported
1428 rules converted
0 exceptions exported
0 action connectors exported
1428 rules saved to dac_test/rules
0 exception lists saved to /home/forteea1/Code/dac_demo/test/detection-rules/dac_test/exceptions
0 action connectors saved to /home/forteea1/Code/dac_demo/test/detection-rules/dac_test/action_connectors

Additionally, try searching for a specific tag or other filter of your custom rules
python -m detection_rules kibana export-rules -d dac_test/rules -sv -ac -e -cro -eq "alert.attributes.tags: \"test\""

Example Output

detection-rules on  4768-fr-filter-prebuilt-rules-in-kibana-export-command [!?] is  v1.2.12 via  v3.12.11 (detection-rules-build) on  eric.forte 
❯ python -m detection_rules kibana export-rules -d dac_test/rules -sv -ac -e -cro -eq "alert.attributes.tags: \"test\""
Loaded config file: /home/forteea1/Code/dac_demo/test/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

1 results exported
0 rules converted
0 exceptions exported
0 action connectors exported
0 rules saved to dac_test/rules
0 exception lists saved to /home/forteea1/Code/dac_demo/test/detection-rules/dac_test/exceptions
0 action connectors saved to /home/forteea1/Code/dac_demo/test/detection-rules/dac_test/action_connectors

Additional Testing

Details

detection-rules on  4768-fr-filter-prebuilt-rules-in-kibana-export-command [!?] is  v1.2.13 via  v3.12.11 (detection-rules-build) on  eric.forte 
❯ python -m detection_rules kibana export-rules -d dac_test/rules -sv -ac -e -cro -eq "alert.attributes.tags: \"test\""
Loaded config file: /home/forteea1/Code/dac_demo/test/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

2 results exported
1 rules converted
0 exceptions exported
0 action connectors exported
1 rules saved to dac_test/rules
0 exception lists saved to /home/forteea1/Code/dac_demo/test/detection-rules/dac_test/exceptions
0 action connectors saved to /home/forteea1/Code/dac_demo/test/detection-rules/dac_test/action_connectors

detection-rules on  4768-fr-filter-prebuilt-rules-in-kibana-export-command [!?] is  v1.2.13 via  v3.12.11 (detection-rules-build) on  eric.forte took 2s 
❯ python -m detection_rules kibana export-rules -d dac_test/rules -sv -ac -e -eq "alert.attributes.tags: \"test\""
Loaded config file: /home/forteea1/Code/dac_demo/test/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

3 results exported
2 rules converted
0 exceptions exported
0 action connectors exported
2 rules saved to dac_test/rules
0 exception lists saved to /home/forteea1/Code/dac_demo/test/detection-rules/dac_test/exceptions
0 action connectors saved to /home/forteea1/Code/dac_demo/test/detection-rules/dac_test/action_connectors

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[github-actions]

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a feature to the code.

Documentation and Context

• Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
• Include additional context or screenshots.
• Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Code changes do not introduce new warnings or errors.
• Variables and functions are well-named and descriptive.
• Any unnecessary / commented-out code is removed.
• Ensure that the code is modular and reusable where applicable.
• Check for proper exception handling and messaging.

Testing

• New unit tests have been added to cover the enhancement.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
• Validate that any rules affected by the enhancement are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.

Additional Checks

• Ensure that the enhancement does not break existing functionality.
• Review the enhancement with a peer or team member for additional insights.
• Verify that the enhancement works across all relevant environments (e.g., different OS versions).
• Confirm that all dependencies are up-to-date and compatible with the changes.
• Confirm that the proper version label is applied to the PR patch, minor, major.