Exclude paths regex

[HariSekhon]

Feature Request to be able to define an exclude regex so that I can exclude problematic K8s patch files that Datree is tripping up on.

For an example of these patch yamls, you can see this run in my public Kubernetes repo:

https://github.com/HariSekhon/Kubernetes-configs/runs/7412852029?check_suite_focus=true

Proposed Solution

--exclude switch that can take a regex

Example usage:

$ datree test **/*.yaml --exclude '\.patch\.yaml$'

Describe alternatives you've considered

I do this in shell (via function in my .bash.d/kubernetes.sh):

find . -type f -iname '*.y*ml' |
grep -v patch |
tr '\n' '\0' | 
xargs -0 datree test --only-k8s-files --ignore-missing-schemas

However, this doesn't work with action-datree so the CI/CD workflow is still broken.

I think it would be better handled via a native --exclude switch, but I'm open to other workarounds.

[adifayer]

@HariSekhon Did you try the --only-k8s-files flag? The idea of this flag is to skip validating all non-K8s files, that way you can avoid these failures. Let me know if it solves the problem :)

[HariSekhon]

Yes you can see it if you expand the action-datree call:

Run datreeio/action-datree@main
  with:
    path: **/*.y*ml
    cliArguments: --output simple --only-k8s-files --ignore-missing-schemas
    isHelmChart: false
    isKustomization: false
  env:
    DATREE_TOKEN: ***

so it was already there but did not solve this problem because these patch files have enough k8s structure to be picked up, but are missing other bits which they don't patch override - hence why I need a more explicit --exclude switch.

I mean, you guys could also debug all the failing patch files in my public repo and try to make datree smart enough to just deal with them, but whenever there is any unforeseen issue it might help to have a quick workaround like explicit --exclude regex.

[adifayer]

@HariSekhon Oops, my bad! Missed the flag😅

Perhaps we should solve this problem with finding a way to also exclude patch yaml when the flag --only-k8s-files is in use? Since the patch file w/o the base is not a "real" K8s resource, also, this way you're not required to pass another flag or update the flag regex according to changes in the repo.
WDYT?

[HariSekhon]

Tempting but how does one know which files are patch files?

File names are pretty arbitrary and therefore unreliable?

If we do a naive substring match on filename then it's possible that we'll introduce a bug that will miss scanning other files.

[adifayer]

I agree with you @HariSekhon . Had a little more research, and it turns out that excluding a pattern is supported in glob pattern, I believe this is the solution 🤗 WDYT? :)
Attaching a tool for testing out your globes.

[HariSekhon]

not sure if this will work in a shell where the ! will be misinterpreted:

bash --version
GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)
...

echo !(patch)*.yaml
bash: !: event not found

echo *!(patch).yaml
bash: !: event not found

double quoting results in the same:

echo "!(patch)*.yaml"
bash: !: event not found

single quoting will be of course useless as it won't be interpreted:

echo '!(patch)*.yaml'
!(patch)*.yaml

[adifayer]

@HariSekhon Noted :) We'll prioritize this issue. Thx!

[Meyazhagan]

@adifayer shall I work on this

[adifayer]

@Meyazhagan You got it!