Skip to content

feat(loader): implement persistent metadata cache #6630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dwisiswant0 merged 14 commits into from
Dec 4, 2025
Merged

feat(loader): implement persistent metadata cache #6630

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
958673f
feat(loader): implement persistent metadata cache
dwisiswant0 Nov 24, 2025
7c26fff
test(loader): adds `BenchmarkLoadTemplates{,OnlyMetadata}` benchs
dwisiswant0 Nov 24, 2025
5891387
ci: cache nuclei-templates index
dwisiswant0 Nov 24, 2025
41c5f0c
chore(index): satisfy lints
dwisiswant0 Nov 24, 2025
04cb66c
fix(index): correct metadata filter logic
dwisiswant0 Nov 24, 2025
84f56c3
test(index): resolve test timing issue
dwisiswant0 Nov 24, 2025
e659c88
ci: cache nuclei with composite action
dwisiswant0 Nov 30, 2025
a49cf2f
fix(index): file locking issue on Windows
dwisiswant0 Nov 30, 2025
dc38f6f
test(index): flaky index tests on Windows
dwisiswant0 Nov 30, 2025
b70a24b
test(cmd): init logger to prevent nil pointer deref
dwisiswant0 Dec 1, 2025
dfae40d
fix(loader): resolve include/exclude paths
dwisiswant0 Dec 1, 2025
7647e3c
feat(index): adds `NewMetadataFromTemplate` func
dwisiswant0 Dec 1, 2025
58ad007
refactor(index): return metadata when `(*Index).cache` is nil
dwisiswant0 Dec 1, 2025
0002035
refactor(loader): restore pre‑index behavior semantics
dwisiswant0 Dec 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/generate-pgo.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ jobs:
- uses: actions/checkout@v6
- uses: projectdiscovery/actions/setup/git@v1
- uses: projectdiscovery/actions/setup/go@v1
- uses: projectdiscovery/actions/cache/nuclei@v1
- name: Generate list
run: for i in {1..${{ matrix.targets }}}; do echo "https://honey.scanme.sh/?_=${i}" >> "${LIST_FILE}"; done
# NOTE(dwisiswant0): use `-no-mhe` flag to get better samples.
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/perf-regression.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ jobs:
- uses: actions/checkout@v6
- uses: projectdiscovery/actions/setup/go@v1
- uses: projectdiscovery/actions/cache/go-rod-browser@v1
- uses: projectdiscovery/actions/cache/nuclei@v1
- run: make build-test
- run: ./bin/nuclei.test -test.run - -test.bench=. -test.benchmem ./cmd/nuclei/ | tee $BENCH_OUT
env:
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ jobs:
- uses: actions/checkout@v6
- uses: projectdiscovery/actions/setup/go@v1
- uses: projectdiscovery/actions/cache/go-rod-browser@v1
- uses: projectdiscovery/actions/cache/nuclei@v1
- uses: projectdiscovery/actions/free-disk-space@v1
with:
llvm: 'false'
Expand Down Expand Up @@ -66,6 +67,7 @@ jobs:
- uses: actions/checkout@v6
- uses: projectdiscovery/actions/setup/go@v1
- uses: projectdiscovery/actions/cache/go-rod-browser@v1
- uses: projectdiscovery/actions/cache/nuclei@v1
- name: "Simple"
run: go run .
working-directory: examples/simple/
Expand All @@ -88,6 +90,7 @@ jobs:
steps:
- uses: actions/checkout@v6
- uses: projectdiscovery/actions/setup/go@v1
- uses: projectdiscovery/actions/cache/nuclei@v1
- uses: projectdiscovery/actions/setup/python@v1
- uses: projectdiscovery/actions/cache/go-rod-browser@v1
- run: bash run.sh "${{ matrix.os }}"
Expand All @@ -108,6 +111,7 @@ jobs:
steps:
- uses: actions/checkout@v6
- uses: projectdiscovery/actions/setup/go@v1
- uses: projectdiscovery/actions/cache/nuclei@v1
- uses: projectdiscovery/actions/setup/python@v1
- uses: projectdiscovery/actions/cache/go-rod-browser@v1
- run: bash run.sh
Expand Down
2 changes: 2 additions & 0 deletions cmd/integration-test/library.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import (
"github.com/logrusorgru/aurora"
"github.com/pkg/errors"
"github.com/projectdiscovery/goflags"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/config"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/disk"
"github.com/projectdiscovery/nuclei/v3/pkg/catalog/loader"
Expand Down Expand Up @@ -70,6 +71,7 @@ func executeNucleiAsLibrary(templatePath, templateURL string) ([]string, error)

defaultOpts := types.DefaultOptions()
defaultOpts.ExecutionId = "test"
defaultOpts.Logger = gologger.DefaultLogger

mockProgress := &testutils.MockProgressClient{}
reportingClient, err := reporting.New(&reporting.Options{ExecutionId: defaultOpts.ExecutionId}, "", false)
Expand Down
1 change: 1 addition & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ require (
github.com/leslie-qiwa/flat v0.0.0-20230424180412-f9d1cf014baa
github.com/lib/pq v1.10.9
github.com/mattn/go-sqlite3 v1.14.28
github.com/maypok86/otter/v2 v2.2.1
github.com/mholt/archives v0.1.5
github.com/microsoft/go-mssqldb v1.9.2
github.com/ory/dockertest/v3 v3.12.0
Expand Down
2 changes: 2 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -703,6 +703,8 @@ github.com/mattn/go-sqlite3 v1.14.28 h1:ThEiQrnbtumT+QMknw63Befp/ce/nUPgBPMlRFEu
github.com/mattn/go-sqlite3 v1.14.28/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/matttproud/golang_protobuf_extensions v1.0.4/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4=
github.com/maypok86/otter/v2 v2.2.1 h1:hnGssisMFkdisYcvQ8L019zpYQcdtPse+g0ps2i7cfI=
github.com/maypok86/otter/v2 v2.2.1/go.mod h1:1NKY9bY+kB5jwCXBJfE59u+zAwOt6C7ni1FTlFFMqVs=
github.com/mholt/acmez v1.2.0 h1:1hhLxSgY5FvH5HCnGUuwbKY2VQVo8IU7rxXKSnZ7F30=
github.com/mholt/acmez v1.2.0/go.mod h1:VT9YwH1xgNX1kmYY89gY8xPJC84BFAisjo8Egigt4kE=
github.com/mholt/archives v0.1.5 h1:Fh2hl1j7VEhc6DZs2DLMgiBNChUux154a1G+2esNvzQ=
Expand Down
Loading
Loading