[IDEA] Pluggable Filters

[rolweber]

I propose to implement a plug-in mechanism for custom filters.
The BOM created by whichever parser should be handed to each custom filter in turn, before the output is generated.
Example invocation:

cyclonedx-bom -i condalist.json -cj --filter custom.filter --filter another.filter --format json

The custom.filter and another.filter arguments are names of modules, to be loaded from PYTHONPATH.

Expected usage for filters:

• add information from the conda package cache
• add information from .dist-info/METADATA or .egg-info/METADATA
• detect problems, like components without license information
• report statistics
• remove components that are not meant/required to be processed downstream
• add components detected by scanning for vendored packages
• ...

The idea is to introduce a generic filter API, for which all users can implement custom filters according to their needs.
In the future, some filters of general use might be contributed to and shipped with cyclonedx-python itself.

A filter chain enables re-use of filters in different contexts and combinations. That would be harder if users had to implement new parsers in order to plug in custom code.

Background

I need to generate SBOMs for conda environments, with much more information than is available from conda list --json. Therefore, I'm looking for a way to plug in the code I'll have to write anyway. Some of that will be specific to my team's needs. Some might be of general interest. I got management approval to contribute to this project, so I could help with implementing this proposal.

I thought about a separate tool using cyclonedx-python-lib, but that cannot yet read the output of cyclonedx-python. This proposal would create a plug-in point with access to the BOM in memory, before it gets converted to JSON or XML.

[jkowalleck]

TBH: if one does not like the output, they could write their own tool to reduce the result.
This tool is a generator, that generates output based on the detected truth. It is not on the generator to cut its own features.

---

I thought about a separate tool using cyclonedx-python-lib, but that CycloneDX/cyclonedx-python-lib#185 (comment) the output of cyclonedx-python. This proposal would create a plug-in point with access to the BOM in memory, before it gets converted to JSON or XML.

Fully agree. If the python does not have proper features, feel free to use another lib. Or use one of the existing tools.
See https://cyclonedx.org/tool-center/

[rolweber]

OK, I'll go for an in-house solution then. I thought others might benefit from code that can extract info from conda recipes in the conda package cache.

[jkowalleck]

if you want to contribute your solution, feel free to make it available in the https://cyclonedx.org/tool-center/.

Open-source, or not, you still can share your work by adding your tool there.

The CycloneDX Tool Center is a community effort to establish a marketplace of free, open source, and proprietary tools and solutions that support the CycloneDX specification. Every effort is made to ensure the accuracy of the information. If there are errors or omissions, please submit modifications in the form of a GitHub pull request by editing tools.yml.

[jkowalleck]

Donating a feature to a project causes the maintainers to also take care of the one feature.
Also, your proposal looks like a tool on its own, with its own domain-specific language, its own logic and such. It looks like its own project - with some touch points to this project.

[rolweber]

My employer allows me to make small code donations (several hundred lines of code or so) to existing projects, if it helps my work. But starting a separate project is out of the question. That would require a commitment and funding for several years, and approvals from several layers up in the hierarchy.

What I need to implement is not unlike what the EnvironmentParser does, just with additional information for conda packages, extracted from the conda package cache. It'll have to run from a separate environment though. And I'll probably need some package info that nobody outside will care about. That's why I suggested a plug-in mechanism that allows for augmenting the data already generated by the CondaListJsonParser. A filter chain would also allow for keeping the code of public interest separate from the code specific to our in-house requirements.

Never mind. I think I can use the CondaListJsonParser along with cyclonedx-python-lib to implement what is required. No need to enhance cyclonedx-bom for that.