feat: add custom search command support #1672

hetangmodi-crest · 2025-04-01T11:06:20Z

Issue number: ADDON-76780

PR Type

What kind of change does this PR introduce?

Summary

Added support for custom search command. This PR contains the dev changes for custom search command.
Test case PR #1654

Changes

customSearchCommand tag has been added to the global configuration, allowing users to generate their custom search commands using ucc-gen build. Users will only need to define the logic for their command and update the customSearchCommand in globalConfig.

User experience

Users can now generate custom search commands using the ucc-gen build command. To do so, they need to define the command logic and update the globalConfig accordingly.

Checklist

If an item doesn't apply to your changes, leave it unchecked.

Review

self-review - I have performed a self-review of this change according to the development guidelines
Changes are documented. The documentation is understandable, examples work (more info)
PR title and description follows the contributing principles
meeting - I have scheduled a meeting or recorded a demo to explain these changes (if there is a video, put a link below and in the ticket)

Tests

See the testing doc.

Unit - tests have been added/modified to cover the changes
Smoke - tests have been added/modified to cover the changes
UI - tests have been added/modified to cover the changes
coverage - I have checked the code coverage of my changes (see more)

Demo/meeting:

Reviewers are encouraged to request meetings or demos if any part of the change is unclear

…command

kkedziak-splunk · 2025-04-07T12:14:42Z

splunk_add_on_ucc_framework/templates/custom_command/reporting.template

+from splunklib.searchcommands import \
+    dispatch, ReportingCommand, Configuration, Option, validators
+
+{% if import_map %}


I am not sure about this need of checking whether we can import "map". One of concerns is that importing a module may create unexpected effects and in my opinion we could avoid that.

I tested locally in ipython some possibilities:

So basically we could change this code a bit:

... from {{ imported_file_name }} import reduce try: from {{ imported_file_name }} import map as module_map except ImportError: module_map = None @Configuration() class {{class_name}}Command(ReportingCommand): ... if module_map is not None: @Configuration() def map(self, events): return module_map(self, events) ...

Alternatively:

... import {{ imported_file_name }} as module @Configuration() class {{class_name}}Command(ReportingCommand): ... if hasattr(module, "map"): ...

In both solutions the condition will be executed only once, when the class is constructed.

kkedziak-splunk · 2025-04-07T12:43:33Z

docs/custom_search_commands.md

There is something about the files that are generated but what about adding just a few words what the command usage looks like from the user perspective?

Okay, will update the last given example and add a screenshot of the search query.

kkedziak-splunk · 2025-04-07T12:53:02Z

docs/custom_search_commands.md

+- For `Generating` command, the Python file must include a `generate` function.
+- For `Streaming` command, the Python file must include a `stream` function.
+- For `Eventing` command, the Python file must include a `transform` function.
+- For `Reporting` command, the Python file must include a `reduce` function, and optionally a `map` function if a streaming pre-operation is required.


I see that "Reporting" has been changed to "Transforming":

Note: In earlier versions of Splunk Enterprise, transforming commands were referred to as "reporting commands."

(source)

Shall we use the newer name? Or at least mention it?

Also, the "Eventing" can also be called "Dataset processing" (as seen here)

Will change the logic and update the documentation as well.

kkedziak-splunk · 2025-04-07T14:12:12Z

splunk_add_on_ucc_framework/const.py

+# TODO: Update the list as and when Splunk introduces new commands.
+# Links to use: https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference
+#               https://docs.splunk.com/Documentation/Splunk/latest/SearchReference
+SPLUNK_COMMANDS = [


I am not sure about maintaining this list.

Do we really want to prevent users from overwriting commands? Is it really forbidden?

If we want to maintain this list, we should automate it as much as possible. For example, when should we check it?

If we want to keep this list, then some test would be useful, e.g.:

import re import urllib.request from splunk_add_on_ucc_framework.const import SPLUNK_COMMANDS def test_command_list_up_to_date(): with urllib.request.urlopen("https://docs.splunk.com/Documentation/Splunk/latest/SearchReference") as resp: content = resp.read().decode() search_commands_ul = re.search(r"Search\s+Commands.+?<ul.+?>(.+?)</ul>", content, re.S).group(1) search_commands = re.findall(r"<li[^>]*>.*?<a[^>]*>\s*([^\s<]+)\s+?</a>", search_commands_ul, re.S) assert set(search_commands) == set(SPLUNK_COMMANDS)

This would alert us that the list is not up to date and unfortunately we would have failures when the website changes.

Btw. I ran this code and it seems there are already 4 entries missing 😄

E AssertionError: assert {'abstract', ...efields', ...} == {'abstract', ...efields', ...} E E Extra items in the left set: E 'snoweventstream' E 'snowevent' E 'snowincidentstream' E 'snowincident' E Use -v to get more diff

It's a great idea to have this test case. However, this 4 mentioned commands are only for ServiceNow add-on. That's why I had omitted it from the list.

kkedziak-splunk · 2025-04-07T14:14:29Z

splunk_add_on_ucc_framework/generators/conf_files/create_commands_conf.py

+
+    def _set_attributes(self, **kwargs: Any) -> None:
+        self.conf_file = "commands.conf"
+        if self._global_config and self._global_config.has_custom_search_commands():


What about changing this line similarly to one of previous PR's?

I.e. instead of:

if self._global_config and self._global_config.has_custom_search_commands():

do:

if self._global_config.has_custom_search_commands():

Yes, I’ll remove the condition.
This change was part of PR #1671, and since it’s now merged, I’ll update it here as well.

kkedziak-splunk · 2025-04-07T14:24:16Z

splunk_add_on_ucc_framework/commands/build.py

            app_manifest=app_manifest,
            addon_version=addon_version,
            has_ui=global_config.meta.get("isVisible", True),
+            custom_search_commands=global_config.custom_search_commands,


Shouldn't a generator take that directly from the global config?

Yes, will remove it.
In previous implementation it was useful. But as PR #1671 is now merged, now it doesn't add a value.

kkedziak-splunk · 2025-04-07T14:40:14Z

splunk_add_on_ucc_framework/generators/python_files/create_custom_command_python.py

+                if arg["validate"].get("minimum") and arg["validate"].get("maximum"):
+                    arg_str += f", validate=validators.{arg['validate']['type']}"
+                    arg_str += f"(minimum={arg['validate'].get('minimum')}, "
+                    arg_str += f"maximum={arg['validate'].get('maximum')})"
+                elif arg["validate"].get("minimum") and not arg["validate"].get(
+                    "maximum"
+                ):
+                    arg_str += f", validate=validators.{arg['validate']['type']}"
+                    arg_str += f"(minimum={arg['validate'].get('minimum')})"
+                elif not arg["validate"].get("minimum") and arg["validate"].get(
+                    "maximum"
+                ):
+                    arg_str += f", validate=validators.{arg['validate']['type']}"
+                    arg_str += f"(maximum={ arg['validate'].get('maximum')})"
+                else:
+                    arg_str += f", validate=validators.{arg['validate']['type']}()"


This looks too complicated and I needed a few seconds to read it.

What do you think about simplifying it?

Example:

validator_kwargs = {} for param in ("minimum", "maximum"): value = arg["validate"].get(param) if value: validator_kwargs[param] = value validator_kwargs = ", ".join(f"{name}={value}" for name, value in validator_kwargs.items()) validator_arg = f"validators.{arg['validate']['type']}({validator_kwargs})"

Sure, I will try to simplify the implementation.

sgoral-splunk · 2025-04-10T14:05:10Z

splunk_add_on_ucc_framework/generators/conf_files/create_commands_conf.py

+
+    def generate_conf(self) -> Union[Dict[str, str], None]:
+        if not (
+            self._global_config and self._global_config.has_custom_search_commands()


I believe this self._global_config also can be removed due to this PR, right?

Yes, you are right, will make the changes accordingly, once the ReportingCommand issue is resolved.

sgoral-splunk · 2025-04-10T14:05:32Z

splunk_add_on_ucc_framework/generators/conf_files/create_searchbnf_conf.py

+    def _set_attributes(self, **kwargs: Any) -> None:
+        self.conf_file = "searchbnf.conf"
+        self.searchbnf_info = []
+        if self._global_config and self._global_config.has_custom_search_commands():


same as above, self._global_config can be removed.

sgoral-splunk · 2025-04-10T14:06:35Z

splunk_add_on_ucc_framework/generators/python_files/create_custom_command_python.py

+
+    def _set_attributes(self, **kwargs: Any) -> None:
+        self.commands_info = []
+        if self._global_config and self._global_config.has_custom_search_commands():


same as above, self._global_config can be removed.

sgoral-splunk · 2025-04-10T14:09:25Z

splunk_add_on_ucc_framework/generators/python_files/create_custom_command_python.py

+
+    def generate_python(self) -> Union[Dict[str, str], None]:
+        if not (
+            self._global_config and self._global_config.has_custom_search_commands()


same as above, self._global_config can be removed.

sgoral-splunk · 2025-04-10T14:22:14Z

splunk_add_on_ucc_framework/generators/python_files/create_custom_command_python.py

+        arg_str = ""
+        arg_str = arg_str + f"{arg['name']} = Option(name='{arg['name']}'"


nit: Am I missing smth or this can be replace just by
arg_str = f"{arg['name']} = Option(name='{arg['name']}'"

Will cover this in newer, approach as the current implementation is a bit difficult to understand.

hetangmodi-crest · 2025-04-16T11:47:48Z

Closing this PR as the following changes are covered in
PR #1693 : Contains base code for supporting custom search command.
PR #1694 : Contains logic and test cases for the generation of commands.conf
PR #1695 : Contains logic and test cases for the generation of searchbnf.conf
PR #1697 : Contains logic and test cases for the generation of Python files

hetangmodi-crest added 4 commits March 21, 2025 17:07

docs: documentation for custom search command

3e1dbb4

feat: provide support for custom search command

68f0319

fix: updated handling of requiredSearchAssistant

af15e1b

chore: Merge branch 'develop' into feat/implementation-custom-search-…

b15c43a

…command

hetangmodi-crest self-assigned this Apr 1, 2025

hetangmodi-crest requested review from a team as code owners April 1, 2025 11:06

hetangmodi-crest requested review from artemrys and sgoral-splunk April 1, 2025 11:06

pull-request-size bot added the size/XXL label Apr 1, 2025

hetangmodi-crest mentioned this pull request Apr 1, 2025

feat: add custom search command support #1653

Closed

13 tasks

refactor: moved argument logic from template files to source code

564a753

kkedziak-splunk reviewed Apr 7, 2025

View reviewed changes

sgoral-splunk reviewed Apr 10, 2025

View reviewed changes

hetangmodi-crest added 2 commits April 11, 2025 18:21

refactor: renamed reporting and eventing command

0848c84

refactor: removed PyGenerator class

bdd58e4

hetangmodi-crest closed this Apr 16, 2025

github-actions bot locked and limited conversation to collaborators Apr 16, 2025

		arg_str = ""
		arg_str = arg_str + f"{arg['name']} = Option(name='{arg['name']}'"

feat: add custom search command support #1672

feat: add custom search command support #1672

Uh oh!

Conversation

hetangmodi-crest commented Apr 1, 2025

PR Type

Summary

Changes

User experience

Checklist

Review

Tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hetangmodi-crest commented Apr 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants