Autodiscover filter with negation not working

[The-Engineer]

How are you running Renovate?

Self-hosted Renovate

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

GitLab Selfhosted

Please tell us more about your question or problem

Hello everyone,
I am a huge fan of Renovate and use it in my GitLab. I use it on one of the root groups of repos.
The bot is set up to autodiscover all repos in the root group but exclude some foldergroups within asdf (config and misc in this abstraction).

.
├── asdf
│   ├── apps
│   │   ├── app01
│   │   │   └── repo07
│   │   ├── app02
│   │   │   └── repo08
│   │   └── repo04
│   ├── config
│   │   └── repo05
│   ├── misc
│   │   └── repo06
│   ├── repo01
│   ├── repo02
│   └── repo03
└── wsad

The following command is used to write all relevant repos for a run into a json file before using the same to run the bot:

renovate --autodiscover --autodiscover-filter=asdf/** $RENOVATE_EXTRA_FLAGS --write-discovered-repos=repos-found.json --optimize-for-disabled=true

I tried several ways to exclude the subfolders config and misc:

RENOVATE_EXTRA_FLAGS: "--autodiscover-filter=['!asdf/config/**','!asdf/misc/**']"
RENOVATE_EXTRA_FLAGS: "--autodiscover-filter=['!asdf/config/*','!asdf/misc/*']"
RENOVATE_EXTRA_FLAGS: "--autodiscover-filter=['!asdf/config/.*','!asdf/misc/.*']"
RENOVATE_EXTRA_FLAGS: "--autodiscover-filter=['!/asdf/config/**','!/asdf/misc/**']"
RENOVATE_EXTRA_FLAGS: "--autodiscover-filter=['asdf/!(config)/**','asdf/!(misc)/**']"
RENOVATE_EXTRA_FLAGS: "--autodiscover-filter=['!asdf/{config,config/**}','!cdp/{misc,misc/**}']"

But the autodiscover always contains all repos inside the repos inside the folder asdf.

I already found several issues but non helped me fix the exclude

• Add autodiscoverExclusions to exclude projects fetched by autodiscoverFilter #21347
• Simpler exclusion pattern when globbing isaacs/minimatch#25
• Autodiscover Filter: re2 fails to parse regex with negative lookahead assertion #16472

To run Renovate I use the latest full Docker image. The debug logs show that the autodiscover filter were recognized but the results show that it didn't work :(

Logs (if relevant)

Logs

Replace this text with your logs, between the starting and ending triple backticks

[rarkins]

Can you try with config.js to exclude the chance it's related to env escaping?

[rarkins]

Please show the debug log for the case where you use config.js only

[The-Engineer]

sure, here you are:

$ renovate --autodiscover $RENOVATE_EXTRA_FLAGS --write-discovered-repos=ci-renovate-repos.json --optimize-for-disabled=true
DEBUG: Using RE2 regex engine
DEBUG: Parsing configs
DEBUG: Checking for config file in /builds/asdf/renovate-bot/config.js
 WARN: env config requireConfig property has been changed to required
DEBUG: File config
       "config": {
         "printConfig": true,
         "autodiscoverFilter": ["asdf/**", "!asdf/config/**"]
       }
DEBUG: CLI config
       "config": {
         "optimizeForDisabled": true,
         "autodiscover": true,
         "writeDiscoveredRepos": "ci-renovate-repos.json"
       }
DEBUG: Env config
       "config": {
         "hostRules": [],
         "extends": ["github>whitesource/merge-confidence:beta"],
         "onboarding": false,
         "onboardingConfig": {
           "$schema": "https://docs.renovatebot.com/renovate-schema.json",
           "extends": ["config:base"]
         },
         "requireConfig": "required",
         "platform": "gitlab",
         "endpoint": "https://gitlab.home.felix.wtf/api/v4",
         "token": "***********"
       }
DEBUG: Combined config
       "config": {
         "printConfig": true,
         "autodiscoverFilter": ["asdf/**", "!asdf/config/**"],
         "hostRules": [],
         "extends": ["github>whitesource/merge-confidence:beta"],
         "onboarding": false,
         "onboardingConfig": {
           "$schema": "https://docs.renovatebot.com/renovate-schema.json",
           "extends": ["config:base"]
         },
         "requireConfig": "required",
         "platform": "gitlab",
         "endpoint": "https://gitlab.home.felix.wtf/api/v4",
         "token": "***********",
         "optimizeForDisabled": true,
         "autodiscover": true,
         "writeDiscoveredRepos": "ci-renovate-repos.json"
       }
DEBUG: Adding trailing slash to endpoint
DEBUG: Found valid git version: 2.48.1
DEBUG: Setting global hostRules
DEBUG: hostRules: authentication already set for gitlab.home.felix.wtf
DEBUG: Using queue: host=gitlab.home.felix.wtf, concurrency=16
DEBUG: GitLab version is: 17.8.0
DEBUG: Using platform gitAuthor: renovate <group_91_bot_0b85d1957d56ca79942658b63265e381@noreply.gitlab.felix.lan>
DEBUG: Adding token authentication for gitlab.home.felix.wtf (hostType=gitlab) to hostRules
DEBUG: Using baseDir: /tmp/renovate
DEBUG: Using cacheDir: /tmp/renovate/cache
DEBUG: Using containerbaseDir: /tmp/renovate/cache/containerbase
DEBUG: Initializing Renovate internal cache into /tmp/renovate/cache/renovate/renovate-cache-v1
DEBUG: Commits limit = null
DEBUG: Setting global hostRules
DEBUG: Adding token authentication for gitlab.home.felix.wtf (hostType=gitlab) to hostRules
DEBUG: validatePresets()
DEBUG: Autodiscovering GitLab repositories
DEBUG: hostRules: applying Bearer authentication for gitlab.home.felix.wtf
DEBUG: Discovered 7 project(s)
DEBUG: Autodiscovered 7 repositories
DEBUG: Applying autodiscoverFilter
       "autodiscoverFilter": ["asdf/**", "!asdf/config/**"]
 INFO: Autodiscovered repositories
       "length": 7,
       "repositories": [
         "asdf/config/subconf/test-repo-g",
         "asdf/config/test-repo-e",
         "asdf/apps/app01/test-repo-d",
         "asdf/apps/test-repo-c",
         "asdf/test-repo-b",
         "asdf/test-repo-a",
         "asdf/renovate-bot"
       ]
 INFO: Written discovered repositories to ci-renovate-repos.json
DEBUG: Checking file package cache for expired items
DEBUG: Deleted 0 of 0 file cached entries in 14ms
$ cat ci-renovate-repos.json | jq
[
  "asdf/config/subconf/test-repo-g",
  "asdf/config/test-repo-e",
  "asdf/apps/app01/test-repo-d",
  "asdf/apps/test-repo-c",
  "asdf/test-repo-b",
  "asdf/test-repo-a",
  "asdf/renovate-bot"
]

[rarkins]

There's two possibilities:

• a bug
• your repos don't start with asdf/ and the changes you're making to your logs is masking the real problem

[The-Engineer]

Then my feeling is that we might be having a bug. To test the behavior independent I really created the asdf group in my GitLab

Can I provide any more data to create a bug ticket?

Software versions:
Renovate 39.116.0
GitLab 17.8.0
GitLab runner 17.7.0
Architecture x64
image used: renovate/renovate:latest

Please let me know if there's any more details I can provide or help in any other way. And thanks a lot again for taking your time!

[The-Engineer]

friendly ping @rarkins

[github-actions[bot]]

Hi there,

This is intended as a polite, automated request that users avoid @ mentioning repository maintainers like @rarkins or @viceice. Doing so causes annoying mobile notifications and makes it harder to maintain this repository.

We know it might be common elsewhere but we participate in hundreds of discussions a week and would need to turn off GitHub mobile notifications if we were mentioned in every one.

As a general rule, we will read and respond to all discussions in this repository, so there is no need to mention us.

Thanks, the Renovate team

[The-Engineer]

ok, sorry, fair point

[rarkins]

Looking at the docs, it says:

Only use a single negation and don't mix with other filters because all filters are combined with or.

This seems to describe your problem. In other words, it's a documented limitation. Your rule essentially says "include any project matching asdf/** or matching !asdf/config/** - which means everything.

Sorry for not catching that earlier - I'd forgotten and needed to debug it to work it out.

I've created #33774 for the next major release