feat(loader): implement persistent metadata cache

[dwisiswant0]

Summary

This PR introduces a persistent metadata caching (#6626) system that significantly improves template loading performance by enabling early filtering w/o full YAML parsing.

Closes #6626

Motive

With ~11K+ templates (current) in the nuclei-templates repository, template loading performance becomes a bottleneck, especially when using filters. Currently, nuclei must parse every template's YAML to extract metadata (tags, authors, severity, etc.) before filtering, wasting time on templates that will be excluded.

p.s. this work is a continuation of #6619.

Changes

feat(loader): implement persistent metadata cache

for template filtering optimization.

Introduce a new template metadata indexing system
with persistent caching to dramatically improve
template loading perf when filters are applied.
The implementation adds a new index pkg that
caches lightweight template metadata (ID, tags,
authors, severity, .etc) and enables filtering
templates before expensive YAML parsing occurs.

The index uses an in-memory LRU cache backed by
`otter` pkg for efficient memory management with
adaptive sizing based on entry weight, defaulting
to approx. 40MB for 50K templates.
Metadata is persisted to disk using gob encoding
at "~/.cache/nuclei/index.gob" with atomic writes
to prevent corruption. The cache automatically
invalidates stale entries using `ModTime` to
detect file modifications, ensuring metadata
freshness w/o manual intervention.

Filtering has been refactored from the previous
`TagFilter` and `PathFilter` approach into a
unified `index.Filter` type that handles all basic
filtering ops including severity, authors, tags,
template IDs with wildcard support, protocol
types, and path-based inclusion and exclusion. The
filter implements OR logic within each field type
and AND logic across different field types, with
exclusion filters taking precedence over inclusion
filters and forced inclusion via
`IncludeTemplates` and `IncludeTags` overriding
exclusions.

The `loader` integration creates an index filter
from store configuration via `buildIndexFilter`
and manages the cache lifecycle through
`loadTemplatesIndex` and `saveTemplatesIndex`
methods. When `LoadTemplatesOnlyMetadata` or
`LoadTemplatesWithTags` is called, the system
first checks the metadata cache for each template
path. If cached metadata exists and passes
validation, the filter is applied directly against
the metadata without parsing. Only templates
matching the filter criteria proceed to full YAML
parsing, resulting in significant performance
gains.

Advanced filtering via "-tc" flag
(`IncludeConditions`) still requires template
parsing as these are expression-based filters that
cannot be evaluated from metadata alone. The
`TagFilter` has been simplified to handle only
`IncludeConditions` while all other filtering ops
are delegated to the index-based filtering system.

Cache management is fully automatic with no user
configuration required. The cache gracefully
handles errors by logging warnings & falling back
to normal op w/o caching. Cache files use schema
versioning to invalidate incompatible cache
formats across nuclei updates (well, specifically
`Index` and `Metadata` changes).

This optimization particularly benefits repeated
scans with the same filters, CI/CD pipelines
running nuclei regularly, development and testing
workflows with frequent template loading, and any
scenario with large template collections where
filtering would exclude most templates.

New

• index pkg

  1. Persistent cache with otter pkg, gob serialization.
  2. Lightweight template metadata structure with mod time validation.
     Why mod time instead of checksum? Because mod time is $$O(1)$$ with a single stat syscall, while checksumming will be $$O(N)$$ by reading the entire file and then hashing. So, yeah, relying on mod time as the source of truth for freshness, iirc, is standard practice in build systems and caches. So if the timestamp changes, the system assumes the file changed. The cost of one unnecessary re-parse (in the rare case of touch w/o change) is negligible compared to the complexity and risks of trying to be "smarter". tl;dr: keep it strict, if the time changed, treat it as dirty.
  3. Advanced filtering with precedence rules (include/exclude logic).

Modified

• loader pkg

  1. Removed pathFilter (replaced by index.Filter).
  2. Updated LoadTemplatesWithTags() w/ two-stage filtering.
  3. Updated LoadTemplatesOnlyMetadata() w/ cache-first approach.
  4. Simplified tagFilter to handle only IncludeConditions (advanced DSL filtering).

Filter responsibility split:

Before: tagFilter handles all filtering (tags, authors, IDs, paths, conditions).
After:

• index.Filter now handles basic filtering (tags, authors, IDs, paths, severity, protocols).
• tagFilter handles advanced filtering (IncludeConditions DSL expressions).

Proof

Benchmark results using hyperfine with 10 runs each, comparing before/after implementation:

Note

./bin/nuclei built against dev branch, while ./bin/nuclei-patch built against this PR branch.

$ bash compare.sh
Filter by severity:
Benchmark 1: ./bin/nuclei -s critical,high -duc -tl
  Time (mean ± σ):      2.601 s ±  0.027 s    [User: 3.344 s, System: 0.349 s]
  Range (min … max):    2.560 s …  2.648 s    10 runs
 
Benchmark 2: ./bin/nuclei-patch -s critical,high -duc -tl
  Time (mean ± σ):      1.052 s ±  0.012 s    [User: 1.231 s, System: 0.191 s]
  Range (min … max):    1.036 s …  1.079 s    10 runs
 
Summary
  ./bin/nuclei-patch -s critical,high -duc -tl ran
    2.47 ± 0.04 times faster than ./bin/nuclei -s critical,high -duc -tl
==================================================
Filter by author:
Benchmark 1: ./bin/nuclei -author pdteam -duc -tl
  Time (mean ± σ):      2.593 s ±  0.017 s    [User: 3.329 s, System: 0.347 s]
  Range (min … max):    2.572 s …  2.632 s    10 runs
 
Benchmark 2: ./bin/nuclei-patch -author pdteam -duc -tl
  Time (mean ± σ):     450.9 ms ±   5.0 ms    [User: 470.4 ms, System: 109.3 ms]
  Range (min … max):   440.9 ms … 459.7 ms    10 runs
 
Summary
  ./bin/nuclei-patch -author pdteam -duc -tl ran
    5.75 ± 0.07 times faster than ./bin/nuclei -author pdteam -duc -tl
==================================================
Filter by tags:
Benchmark 1: ./bin/nuclei -tags cve,rce -duc -tl
  Time (mean ± σ):      2.578 s ±  0.033 s    [User: 3.264 s, System: 0.346 s]
  Range (min … max):    2.519 s …  2.627 s    10 runs
 
Benchmark 2: ./bin/nuclei-patch -tags cve,rce -duc -tl
  Time (mean ± σ):      1.100 s ±  0.041 s    [User: 1.266 s, System: 0.183 s]
  Range (min … max):    1.058 s …  1.202 s    10 runs
 
Summary
  ./bin/nuclei-patch -tags cve,rce -duc -tl ran
    2.34 ± 0.09 times faster than ./bin/nuclei -tags cve,rce -duc -tl
==================================================
Filter by ID pattern:
Benchmark 1: ./bin/nuclei -id 'CVE-2021-*' -duc -tl
  Time (mean ± σ):      2.724 s ±  0.056 s    [User: 3.269 s, System: 0.352 s]
  Range (min … max):    2.676 s …  2.857 s    10 runs
 
Benchmark 2: ./bin/nuclei-patch -id 'CVE-2021-*' -duc -tl
  Time (mean ± σ):     702.5 ms ±  71.2 ms    [User: 501.0 ms, System: 119.3 ms]
  Range (min … max):   635.5 ms … 861.5 ms    10 runs
 
Summary
  ./bin/nuclei-patch -id 'CVE-2021-*' -duc -tl ran
    3.88 ± 0.40 times faster than ./bin/nuclei -id 'CVE-2021-*' -duc -tl
==================================================
Complex filter:
Benchmark 1: ./bin/nuclei -s high -author pdteam -tags cve -duc -tl
  Time (mean ± σ):      2.754 s ±  0.044 s    [User: 3.267 s, System: 0.363 s]
  Range (min … max):    2.665 s …  2.814 s    10 runs
 
Benchmark 2: ./bin/nuclei-patch -s high -author pdteam -tags cve -duc -tl
  Time (mean ± σ):     559.9 ms ± 127.8 ms    [User: 381.2 ms, System: 103.3 ms]
  Range (min … max):   463.9 ms … 874.1 ms    10 runs
 
Summary
  ./bin/nuclei-patch -s high -author pdteam -tags cve -duc -tl ran
    4.92 ± 1.13 times faster than ./bin/nuclei -s high -author pdteam -tags cve -duc -tl

Click to toggle contents of compare.sh

#!/usr/bin/env bash
# compare.sh

echo "Filter by severity:"
hyperfine \
	"./bin/nuclei -s critical,high -duc -tl" \
	"./bin/nuclei-patch -s critical,high -duc -tl"
echo "=================================================="

echo "Filter by author:"
hyperfine \
	"./bin/nuclei -author DhiyaneshDK -duc -tl" \
	"./bin/nuclei-patch -author DhiyaneshDK -duc -tl"
echo "=================================================="

echo "Filter by tags:"
hyperfine \
	"./bin/nuclei -tags cve,rce -duc -tl" \
	"./bin/nuclei-patch -tags cve,rce -duc -tl"
echo "=================================================="

echo "Filter by ID pattern:"
hyperfine \
	"./bin/nuclei -id 'CVE-2021-*' -duc -tl" \
	"./bin/nuclei-patch -id 'CVE-2021-*' -duc -tl"
echo "=================================================="

echo "Complex filter:"
hyperfine \
	"./bin/nuclei -s high -author pdteam -tags cve -duc -tl" \
	"./bin/nuclei-patch -s high -author pdteam -tags cve -duc -tl"

Filter Scenario	Before	After	Speedup	Time Saved
Severity filtering -s critical,high	2.601s ± 0.027s	1.052s ± 0.012s	2.47×	≈60% faster
Author filtering -author pdteam	2.593s ± 0.017s	0.451s ± 0.005s	5.75×	≈83% faster
Tag filtering -tags cve,rce	2.578s ± 0.033s	1.100s ± 0.041s	2.34×	≈57% faster
ID pattern matching -id 'CVE-2021-*'	2.724s ± 0.056s	0.702s ± 0.071s	3.88×	≈74% faster
Complex multi-filter -s high -author pdteam -tags cve	2.754s ± 0.044s	0.560s ± 0.128s	4.92×	≈80% faster

Memory

Filter Scenario	Total GC Cycles (dev)	Total GC Cycles (patch)	Peak Heap (dev)	Peak Heap (patch)	Max Heap (dev)	Max Heap (patch)
Severity filtering -s critical,high	20	14	101 MB	49 MB	279 MB	88 MB
Author filtering -author pdteam	22	11	144 MB	40 MB	165 MB	72 MB
Tag filtering -tags cve,rce	21	14	165 MB	51 MB	165 MB	51 MB
ID pattern matching -id 'CVE-2021-*'	20	10	131 MB	35 MB	131 MB	35 MB
Complex multi-filter -s high -author pdteam -tags cve	22	10	105 MB	38 MB	105 MB	38 MB

Click to toggle contents of gctrace.sh

#!/usr/bin/env bash
# gctrace.sh

echo "Filter by severity:"
echo "dev:"
GODEBUG=gctrace=1 ./bin/nuclei -silent -s critical,high -duc -tl 1>/dev/null
echo "patch:"
GODEBUG=gctrace=1 ./bin/nuclei-patch -silent -s critical,high -duc -tl 1>/dev/null
echo "=================================================="

echo "Filter by author:"

echo "dev:"
GODEBUG=gctrace=1 ./bin/nuclei -silent -author DhiyaneshDK -duc -tl 1>/dev/null
echo "patch:"
GODEBUG=gctrace=1 ./bin/nuclei-patch -silent -author DhiyaneshDK -duc -tl 1>/dev/null
echo "=================================================="

echo "Filter by tags:"
echo "dev:"
GODEBUG=gctrace=1 ./bin/nuclei -silent -tags cve,rce -duc -tl 1>/dev/null
echo "patch:"
GODEBUG=gctrace=1 ./bin/nuclei-patch -silent -tags cve,rce -duc -tl 1>/dev/null
echo "=================================================="

echo "Filter by ID pattern:"
echo "dev:"
GODEBUG=gctrace=1 ./bin/nuclei -silent -id 'CVE-2021-*' -duc -tl 1>/dev/null
echo "patch:"
GODEBUG=gctrace=1 ./bin/nuclei-patch -silent -id 'CVE-2021-*' -duc -tl 1>/dev/null
echo "=================================================="

echo "Complex filter:"
echo "dev:"
GODEBUG=gctrace=1 ./bin/nuclei -silent -s high -author pdteam -tags cve -duc -tl 1>/dev/null
echo "patch:"
GODEBUG=gctrace=1 ./bin/nuclei-patch -silent -s high -author pdteam -tags cve -duc -tl 1>/dev/null

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Summary by CodeRabbit

• New Features

  • Template metadata filtering with include/exclude by authors, tags, IDs, severities, protocol types, and templates; supports forced includes and wildcard matching.
  • In-memory metadata index with persistent on-disk snapshots for fast discovery and selective loading.
  • Loader now leverages the metadata index for selective template loading.

• Tests

  • Extensive unit tests and benchmarks covering filtering, caching, persistence, concurrency, and loader behavior.

• Chores

  • Added cache dependency and library execution now uses the default logger.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

Adds an otter-backed in-memory Metadata Index with Gob snapshot persistence, a Filter system (include/exclude/forced rules with wildcard matching), integrates index-based pre-filtering into the loader to avoid unnecessary YAML parsing, and includes unit tests and benchmarks for index, filter, and loader behaviors.

Changes

Cohort / File(s)	Summary
External dependency go.mod	Added github.com/maypok86/otter/v2 v2.2.1.
Metadata model pkg/catalog/index/metadata.go	Introduces Metadata struct (gob tags) and helpers: NewMetadataFromTemplate, IsValid, MatchesSeverity, MatchesProtocol, HasTag, HasAuthor.
Filtering logic pkg/catalog/index/filter.go	Adds Filter type with include/exclude/forced-include semantics, wildcard ID/path matching, UnmarshalFilter, FilterFunc, IsEmpty, and String.
Index implementation pkg/catalog/index/index.go	Adds Index (otter-backed cache), cacheSnapshot serialization, atomic Gob Save/Load with versioning, concurrent methods (Get, Set, SetFromTemplate, Has, Delete, Size, Clear, All, GetAll), Filter/FilterFunc, Count, and constructors NewIndex/NewDefaultIndex.
Index tests pkg/catalog/index/filter_test.go, pkg/catalog/index/index_test.go	Adds unit tests covering filter behavior, path/ID patterns, unmarshalling, cache operations, persistence, corruption handling, SetFromTemplate, metadata validation, concurrency, and bulk scenarios.
Loader integration pkg/catalog/loader/loader.go	Replaces pathFilter with metadataIndex and indexFilter; adds buildIndexFilter() and loadTemplatesIndex(); integrates index-based pre-filtering and deferred index persistence into template loading flows.
Loader benchmarks pkg/catalog/loader/loader_bench_test.go	Adds benchmarks for LoadTemplates and LoadTemplatesOnlyMetadata across multiple filter scenarios.
Integration logging tweak cmd/integration-test/library.go	Sets defaultOpts.Logger to gologger.DefaultLogger in library execution helper.

sequenceDiagram
    participant User
    participant Loader as Store/Loader
    participant Index as MetadataIndex
    participant Disk
    participant Parser as YAMLParser

    User->>Loader: LoadTemplates(filter)
    Loader->>Index: loadTemplatesIndex()
    alt snapshot exists
        Index->>Disk: Load snapshot (Gob)
        Disk-->>Index: Snapshot metadata
    else no snapshot
        Index-->>Index: Initialize empty cache
    end

    Loader->>Index: Filter(filter_criteria)
    Index-->>Loader: candidate paths

    loop for each candidate path
        alt metadata indicates parsing needed
            Loader->>Parser: Parse template YAML
            Parser-->>Loader: Template + metadata
            Loader->>Index: SetFromTemplate(path, metadata)
            Index-->>Index: Update cache
        else skip parsing
            Loader-->>Index: (no-op)
        end
    end

    Loader->>Index: Save() (deferred)
    Index->>Disk: Atomic write (temp -> final)
    Disk-->>Index: Persisted snapshot
    Loader-->>User: Loaded templates

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~35 minutes

• Pay extra attention to:
  • pkg/catalog/index/filter.go — inclusion/exclusion precedence, forced-include behavior, wildcard ID/path matching.
  • pkg/catalog/index/index.go — Save/Load atomicity, versioning, snapshot validation, cache weight/eviction logic.
  • pkg/catalog/index/metadata.go — IsValid filesystem checks and time/race edge cases.
  • pkg/catalog/loader/loader.go — correct use of index to short-circuit parsing and deferred Save semantics.
  • Tests/benchmarks — filesystem timing, persistence edge cases, and concurrency flakiness.

Poem

🐇
I hopped through indexes, counted each file,
Tucked metadata snug to save a while.
Filters showed doors that I could skip,
Gobbed the cache with a nimble zip.
Fewer parses now — a joyous nibble.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 23.08% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'feat(loader): implement persistent metadata cache' accurately describes the main change, which is adding a persistent metadata caching system to the loader package for performance improvement.
Linked Issues check	✅ Passed	The PR fully addresses all coding objectives from issue #6626: implements persistent metadata cache, stores lightweight template metadata (tags, severity, authors, IDs, protocols, ModTime, path), persists via gob encoding with atomic writes, validates cache freshness using ModTime, supports basic index-based filtering (severity, tags, authors, patterns, protocols, path), preserves advanced filter correctness via full parsing fallback, and achieves significant runtime reductions for filtered scans.
Out of Scope Changes check	✅ Passed	All changes align with PR objectives: new index package (Filter, Metadata, Index), loader integration (buildIndexFilter, loadTemplatesIndex, removePathFilter), benchmarks for performance validation, and library logger configuration. No unrelated modifications detected.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch dwisiswant0/feat/loader/implement-persistent-metadata-cache

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (6)

pkg/catalog/loader/loader_bench_test.go (1)

47-187: Verify benchmark isolation and state management.

Each sub-benchmark calls LoadTemplates repeatedly within the loop. Ensure that:

1. Template state doesn't accumulate across iterations (which could skew timing)
2. The store is properly reset or isolated between iterations
3. The benchmark measures the intended operation without pollution from previous iterations

Consider whether each iteration should create a fresh store or if the current approach correctly measures incremental loading.

pkg/catalog/index/filter_test.go (1)

148-156: Clarify the filter logic explanation in the test comment.

The comment on line 154 says "With OR logic, matches because author AND severity match" which is contradictory. Based on the test setup and the PR objectives, it appears the filtering uses:

• OR logic within each field (e.g., multiple authors)
• AND logic across fields (e.g., must match author AND severity)

Consider revising the comment to clearly state: "With AND logic across fields, matches because both author and severity match (tag mismatch is ignored since another field matches)."

pkg/catalog/loader/loader.go (2)

372-383: IncludeConditions bypass cache optimization.

When tagFilter is present (for IncludeConditions), the code performs full template parsing even for cached entries. This is expected per the PR objectives but means the cache provides limited benefit when using advanced -tc filters.

Consider documenting this trade-off more prominently, or exploring future optimizations to cache IncludeConditions evaluation results.

---

683-707: Consider adding inline comments to clarify the metadata caching flow.

The logic correctly handles both cached and uncached metadata scenarios, but it's complex:

1. If cached metadata exists and matches filter, proceed to LoadTemplate (for tagFilter check)
2. If not cached, LoadTemplate first, then cache metadata and verify filter match
3. Early return if newly cached metadata doesn't match filter

Adding brief inline comments at lines 686, 693, and 703 would help future maintainers understand the flow.

pkg/catalog/index/index.go (1)

58-83: Cache weigher provides reasonable approximation.

The weigher function calculates memory usage by summing string and slice content lengths. While it doesn't account for all Go runtime overhead (slice headers ~24B, bool 1B, pointers 8B), this approximation is acceptable for LRU eviction decisions.

If cache sizing becomes problematic in production, consider adding overhead estimates:

• ~24B per slice header (Authors, Tags)
• ~1B for Verified bool
• ~8B for pointer fields

pkg/catalog/index/filter.go (1)

257-295: Suggestion to expose direct parsing methods on Holder types is valid but optional.

The verification confirms there are no ParseString or FromString methods on severity.Holder or types.TypeHolder. The private functions toSeverity (severity.go:49) and toProtocolType (types.go:84) exist but are intentionally kept private. The current code correctly uses the public UnmarshalYAML API, which is the canonical interface for these types.

While the suggestion to expose these parsing functions as public methods (e.g., ParseString or FromString) on the Holder types would improve maintainability, the current approach is not fragile—it uses a stable public interface. Exposing the private parsing functions as public methods would be a reasonable refactoring that simplifies future uses of these types.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2997735 and b7f3deb.

⛔ Files ignored due to path filters (4)

• .github/workflows/generate-pgo.yaml is excluded by !**/*.yaml
• .github/workflows/perf-regression.yaml is excluded by !**/*.yaml
• .github/workflows/tests.yaml is excluded by !**/*.yaml
• go.sum is excluded by !**/*.sum

📒 Files selected for processing (8)

• go.mod (1 hunks)
• pkg/catalog/index/filter.go (1 hunks)
• pkg/catalog/index/filter_test.go (1 hunks)
• pkg/catalog/index/index.go (1 hunks)
• pkg/catalog/index/index_test.go (1 hunks)
• pkg/catalog/index/metadata.go (1 hunks)
• pkg/catalog/loader/loader.go (9 hunks)
• pkg/catalog/loader/loader_bench_test.go (2 hunks)

🧰 Additional context used

🧠 Learnings (1)

📚 Learning: 2025-07-16T21:27:14.937Z

Learnt from: hdm
Repo: projectdiscovery/nuclei PR: 6322
File: pkg/templates/compile.go:79-81
Timestamp: 2025-07-16T21:27:14.937Z
Learning: To make the template caching mechanism in pkg/templates/compile.go production-ready, DSLs need to be updated to use runtime options instead of cached variables, rather than restoring the Compile() calls on each request.

Applied to files:

• pkg/catalog/index/index.go
• pkg/catalog/loader/loader.go

🧬 Code graph analysis (2)

pkg/catalog/index/index.go (5)

pkg/catalog/index/metadata.go (1)

• Metadata (13-54)

pkg/catalog/config/constants.go (2)

• Version (34-34)
• BinaryName (40-40)

pkg/model/types/severity/severity.go (1)

• Severity (11-11)

pkg/templates/types/types.go (1)

• ProtocolType (15-15)

pkg/catalog/index/filter.go (2)

• Filter (21-57)
• FilterFunc (236-236)

pkg/catalog/index/index_test.go (3)

pkg/catalog/index/index.go (3)

• NewIndex (46-97)
• IndexFileName (18-18)
• IndexVersion (22-22)

pkg/catalog/index/metadata.go (1)

• Metadata (13-54)

pkg/model/types/stringslice/stringslice.go (1)

• StringSlice (29-31)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: Lint

🔇 Additional comments (12)

pkg/catalog/loader/loader_bench_test.go (1)

212-213: LGTM! Pre-warming approach is appropriate for cache benchmarks.

The pre-warming step correctly ensures the benchmark measures warm cache performance, which aligns with the metadata caching feature being tested.

Also applies to: 233-234

pkg/catalog/loader/loader.go (4)

94-102: LGTM! Index initialization and lifecycle management are well-designed.

The use of sync.OnceFunc for saveMetadataIndexOnce ensures atomic save operations, and graceful error handling with warnings prevents cache failures from blocking normal operations.

Also applies to: 172-185

---

318-337: LGTM! Filter construction correctly maps all configuration fields.

The direct type casts for Severities and ProtocolTypes are safe given they're slice type aliases.

---

339-353: LGTM! Graceful degradation pattern for index loading.

Returning nil on errors and logging warnings ensures the system continues to function even if the cache is unavailable, which aligns with the PR's goal of non-breaking fallback behavior.

---

608-609: LGTM! Pre-allocating slice capacity is a nice optimization.

Using make with capacity avoids reallocation during append operations.

pkg/catalog/index/metadata.go (1)

12-85: LGTM! Metadata struct and helper methods are well-designed.

The IsValid method correctly uses ModTime.Equal for timezone-aware comparison, and the matching helpers are straightforward and efficient. The NOTE about future extensions is valuable guidance for maintainers.

pkg/catalog/index/filter.go (1)

206-233: LGTM! Path and ID wildcard matching logic is sound.

Both functions correctly handle exact matches and shell-style wildcards. The directory prefix matching in matchesPath (line 226) appropriately handles directory-based include/exclude patterns.

pkg/catalog/index/index.go (3)

105-122: LGTM! Cache Get/validation with async deletion is well-designed.

The asynchronous deletion on invalid entries (line 116) is safe because IsValid() checks would consistently fail, and multiple Delete calls are idempotent. This approach avoids blocking readers on write locks.

---

202-273: LGTM! Persistence implementation is robust with proper atomic writes and error handling.

The Save method correctly implements atomic writes using temp file + rename. The Load method appropriately handles missing files, corruption, and version mismatches by removing invalid cache files and falling back gracefully.

---

275-353: LGTM! Query methods are thread-safe and efficient.

All methods properly use RLock for concurrent reads, and All() preallocates capacity to avoid reallocation. The early returns for empty filters are good optimizations.

pkg/catalog/index/index_test.go (1)

1-689: LGTM! Comprehensive test coverage for the Index implementation.

The test suite covers happy paths, error paths, edge cases, concurrency, and persistence. The tests properly use temp directories, clean up resources, and verify both positive and negative cases.

go.mod (1)

90-90: No issues found.

The version v2.2.1 is the latest stable release, and no security vulnerabilities have been detected for this version. The dependency is appropriately versioned and secure.

[dwisiswant0]

• Rebase after chore(deps): bump utils #6638 chore: bump PD modules & update httputil calls #6629 (fastdialer bug).

[Mzack9999]

• Merge conflict
• Failing tests

[dwisiswant0]

Yay! Everything's green now.

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

pkg/catalog/loader/loader.go (1)

631-713: Same “no-index” semantics gap exists in LoadTemplatesWithTags

LoadTemplatesWithTags mirrors LoadTemplatesOnlyMetadata: all checks against indexFilter are guarded by if store.metadataIndex != nil. If the index can’t be created/loaded, templates are still parsed and then filtered only by tagFilter and execution-time options (DAST, headless, code, self-contained, signatures), but all basic metadata filters (authors/tags/IDs/severity/protocols/path) are skipped.

To keep CLI behavior independent of cache availability, it would be safer to:

• Either compute a Metadata view from parsed and call indexFilter.Matches even when metadataIndex is nil, or
• Detect a nil metadataIndex early and route this code path through the previous non-index filtering mechanism.

Currently, users could see different template sets under the same flags depending solely on whether the metadata cache file can be used.

🧹 Nitpick comments (1)

pkg/catalog/index/filter.go (1)

285-344: Consider including more fields in Filter.String for easier debugging

Filter.String currently only renders authors, tags, exclude-tags, IDs, severities, and protocol types. It omits other influential fields like IncludeTags, IncludeTemplates, ExcludeTemplates, ExcludeIDs, ExcludeSeverities, and ExcludeProtocolTypes, which can make logs less informative when debugging complex filter combinations.

If log readability becomes an issue, consider extending String() to include these additional fields (possibly abbreviated) to give a fuller picture of the active filter.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b7f3deb and dfae40d.

⛔ Files ignored due to path filters (4)

• .github/workflows/generate-pgo.yaml is excluded by !**/*.yaml
• .github/workflows/perf-regression.yaml is excluded by !**/*.yaml
• .github/workflows/tests.yaml is excluded by !**/*.yaml
• go.sum is excluded by !**/*.sum

📒 Files selected for processing (9)

• cmd/integration-test/library.go (2 hunks)
• go.mod (1 hunks)
• pkg/catalog/index/filter.go (1 hunks)
• pkg/catalog/index/filter_test.go (1 hunks)
• pkg/catalog/index/index.go (1 hunks)
• pkg/catalog/index/index_test.go (1 hunks)
• pkg/catalog/index/metadata.go (1 hunks)
• pkg/catalog/loader/loader.go (9 hunks)
• pkg/catalog/loader/loader_bench_test.go (2 hunks)

🚧 Files skipped from review as they are similar to previous changes (4)

• pkg/catalog/index/metadata.go
• pkg/catalog/index/filter_test.go
• pkg/catalog/loader/loader_bench_test.go
• pkg/catalog/index/index.go

🧰 Additional context used

📓 Path-based instructions (1)

**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.go: Format Go code using go fmt ./...
Run static analysis using go vet ./...

Files:

• pkg/catalog/index/filter.go
• cmd/integration-test/library.go
• pkg/catalog/loader/loader.go
• pkg/catalog/index/index_test.go

🧬 Code graph analysis (3)

pkg/catalog/index/filter.go (4)

pkg/model/types/severity/severities.go (1)

• Severities (13-13)

pkg/model/types/severity/severity.go (2)

• Severity (11-11)
• Holder (70-72)

pkg/templates/types/types.go (3)

• ProtocolTypes (142-142)
• ProtocolType (15-15)
• TypeHolder (103-105)

pkg/catalog/index/metadata.go (1)

• Metadata (13-54)

pkg/catalog/loader/loader.go (5)

pkg/catalog/index/index.go (2)

• Index (32-37)
• NewDefaultIndex (101-103)

pkg/catalog/index/filter.go (1)

• Filter (20-56)

pkg/model/types/severity/severities.go (1)

• Severities (13-13)

pkg/model/types/severity/severity.go (1)

• Severity (11-11)

pkg/templates/types/types.go (2)

• ProtocolTypes (142-142)
• ProtocolType (15-15)

pkg/catalog/index/index_test.go (2)

pkg/catalog/index/index.go (4)

• NewIndex (46-97)
• IndexFileName (18-18)
• IndexVersion (22-22)
• NewDefaultIndex (101-103)

pkg/catalog/index/metadata.go (1)

• Metadata (13-54)

🔇 Additional comments (6)

cmd/integration-test/library.go (1)

18-19: Initialize logger on defaultOpts to avoid nil-logger paths

Wiring defaultOpts.Logger = gologger.DefaultLogger here looks correct and aligns this integration path with the main CLI behavior, which should keep the new loader/index code from ever encountering a nil logger on types.Options.

Only thing to double‑check is whether the additional logging in these integration tests is acceptable for your CI output; if not, you may want to configure DefaultLogger (or an alternative logger instance) to be quieter in this context.

Please run go test ./cmd/integration-test (plus go vet ./... and go fmt ./... as per guidelines) and confirm the logging output is as expected in your test/CI environment.

Also applies to: 72-75

go.mod (1)

48-100: New otter/v2 dependency looks appropriate; just ensure tooling is updated

The addition of github.com/maypok86/otter/v2 v2.2.1 for the metadata cache is reasonable and consistent with the new index package. Please just make sure go mod tidy and your usual go vet ./... / CI checks are run so the module graph and diagnostics stay in sync with this dependency.

pkg/catalog/loader/loader.go (2)

141-186: Index initialization & save hook are clean and lifecycle-aware

Using loadTemplatesIndex, buildIndexFilter, and sync.OnceFunc for saveMetadataIndexOnce gives a clear lifecycle: lazy load at store creation, single save after first load operation, and graceful logging on errors. This aligns well with the “automatic cache lifecycle, non-fatal failures” objective.

---

318-356: Index filter construction matches config fields and types

buildIndexFilter correctly maps loader Config fields (authors/tags/IDs/severities/protocols and include/exclude templates) into the index.Filter, with proper type conversions for severities and protocol types. This keeps the filtering semantics centralized and avoids duplicating parsing logic in the loader.

pkg/catalog/index/filter.go (1)

58-176: Filter matching semantics now align with the documented AND/OR behavior

The combination of Matches, isForcedInclude, isExcluded, and matchesIncludes correctly implements:

• Forced include via IncludeTemplates / IncludeTags
• Exclusions taking precedence via Exclude*
• AND across different include fields (authors, tags, IDs, severities, protocol types)
• OR within each field (any author/tag/ID/severity/protocol type value can match)

The wildcard and case-insensitive handling in matchesID and flexible path rules in matchesPath also look appropriate for typical template ID/path patterns.

pkg/catalog/index/index_test.go (1)

21-708: Index test suite is comprehensive and exercises critical behaviors

The tests here do a solid job covering the new index: basic cache ops, versioning, persistence (including corrupted files and atomic saves), metadata validation, SetFromTemplate extraction (including multi-protocol and nil-classification cases), concurrency on Set/Has, and large datasets. This should give good confidence that the index behaves correctly under both normal and adverse conditions.

[coderabbitai]

Actionable comments posted: 0

♻️ Duplicate comments (2)

pkg/catalog/loader/loader.go (2)

358-417: Good: basic filters now apply even when the index is unavailable (for metadata-only load)

In LoadTemplatesOnlyMetadata, the fallback branch that builds a Metadata via index.NewMetadataFromTemplate when metadataIndex is nil ensures indexFilter.Matches is still enforced, addressing the previous issue where basic filters were skipped if the index couldn’t be initialized. The tagFilter/IncludeConditions path is also preserved.

---

635-787: Basic filters are still skipped in LoadTemplatesWithTags when the metadata index fails to initialize

When store.metadataIndex is nil (e.g., default cache dir can’t be created), LoadTemplatesWithTags never evaluates indexFilter.Matches:

• The early if store.metadataIndex != nil { … } block is skipped entirely.
• The later if store.metadataIndex != nil && parsed != nil && !metadataCached block is also skipped.
• There is no index.NewMetadataFromTemplate fallback here as you added in LoadTemplatesOnlyMetadata.

As a result, when the index can’t be created, CLI options backed solely by index.Filter (authors, tags/exclude-tags/include-tags, include/exclude IDs, severities, protocols, include/exclude templates) become no-ops for this main loading path, contradicting the “graceful fallback without caching” requirement. Only tagFilter/IncludeConditions continue to be honored.

You can mirror the LoadTemplatesOnlyMetadata behavior by always deriving a Metadata from the parsed template and applying indexFilter.Matches, caching it via SetFromTemplate only when metadataIndex is available. For example:

@@ func (store *Store) LoadTemplatesWithTags(templatesList, tags []string) []*templates.Template {
-           if loaded {
-               parsed, err := templates.Parse(templatePath, store.preprocessor, store.config.ExecutorOptions)
-
-               if store.metadataIndex != nil && parsed != nil && !metadataCached {
-                   metadata, _ := store.metadataIndex.SetFromTemplate(templatePath, parsed)
-                   if metadata != nil && !indexFilter.Matches(metadata) {
-                       return
-                   }
-               }
+           if loaded {
+               parsed, err := templates.Parse(templatePath, store.preprocessor, store.config.ExecutorOptions)
+
+               if parsed != nil && indexFilter != nil {
+                   var metadata *index.Metadata
+                   if store.metadataIndex != nil {
+                       // Warm/populate the index when available.
+                       metadata, _ = store.metadataIndex.SetFromTemplate(templatePath, parsed)
+                   } else {
+                       // Fallback: still honor filters without persisting.
+                       metadata = index.NewMetadataFromTemplate(templatePath, parsed)
+                   }
+                   if metadata != nil && !indexFilter.Matches(metadata) {
+                       return
+                   }
+               }

This keeps the cache optional while ensuring filter semantics are identical whether the index is usable or not.

🧹 Nitpick comments (3)

pkg/catalog/index/index_test.go (1)

199-218: Consider deduplicating the corrupted-cache tests

TestCacheCorruptedFile and TestCacheLoadCorruptedRemoval exercise nearly the same behavior (loading a corrupted gob file removes it and results in an empty cache). This is great coverage, but you can likely merge them or factor out a helper to avoid duplication and keep future changes to the corruption behavior localized to one place.

Also applies to: 566-589

pkg/catalog/loader/loader.go (1)

318-356: Optional: surface include/exclude template resolution errors in buildIndexFilter

buildIndexFilter resolves IncludeTemplates/ExcludeTemplates via Catalog.GetTemplatesPath but ignores the returned error maps, unlike other call sites where you log via logErroredTemplates. If resolution fails here, the filter silently drops those include/exclude constraints.

If you want stricter behavior, you could log those errors or at least reuse logErroredTemplates for consistency; otherwise this is fine as a best-effort optimization.

pkg/catalog/index/metadata.go (1)

13-55: Metadata shape and helpers align well with index usage

The Metadata fields and gob tags are well-chosen for lightweight filtering, and NewMetadataFromTemplate cleanly mirrors the template’s core info (with ModTime delegated to SetFromTemplate). IsValid’s ModTime.Equal check is a simple and cheap freshness guard; if you ever hit edge cases on low-resolution filesystems, you might consider allowing a small skew, but for now this is a reasonable trade-off.

The MatchesSeverity, MatchesProtocol, HasTag, and HasAuthor helpers are straightforward and keep filter code tidy.

Also applies to: 57-73, 75-104

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between dfae40d and 93dd108.

📒 Files selected for processing (4)

• pkg/catalog/index/index.go (1 hunks)
• pkg/catalog/index/index_test.go (1 hunks)
• pkg/catalog/index/metadata.go (1 hunks)
• pkg/catalog/loader/loader.go (9 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.go: Format Go code using go fmt ./...
Run static analysis using go vet ./...

Files:

• pkg/catalog/index/metadata.go
• pkg/catalog/index/index_test.go
• pkg/catalog/loader/loader.go
• pkg/catalog/index/index.go

🧬 Code graph analysis (4)

pkg/catalog/index/metadata.go (2)

pkg/model/types/severity/severity.go (1)

• Severity (11-11)

pkg/templates/types/types.go (1)

• ProtocolType (15-15)

pkg/catalog/index/index_test.go (2)

pkg/catalog/index/index.go (3)

• NewIndex (46-97)
• IndexFileName (18-18)
• IndexVersion (22-22)

pkg/catalog/index/metadata.go (2)

• Metadata (14-55)
• NewMetadataFromTemplate (58-73)

pkg/catalog/loader/loader.go (3)

pkg/catalog/index/index.go (2)

• Index (32-37)
• NewDefaultIndex (101-103)

pkg/catalog/index/filter.go (1)

• Filter (20-56)

pkg/catalog/index/metadata.go (2)

• Metadata (14-55)
• NewMetadataFromTemplate (58-73)

pkg/catalog/index/index.go (5)

pkg/catalog/index/metadata.go (2)

• Metadata (14-55)
• NewMetadataFromTemplate (58-73)

pkg/catalog/config/constants.go (2)

• Version (34-34)
• BinaryName (40-40)

pkg/model/types/severity/severity.go (1)

• Severity (11-11)

pkg/templates/types/types.go (1)

• ProtocolType (15-15)

pkg/catalog/index/filter.go (2)

• Filter (20-56)
• FilterFunc (212-212)

🔇 Additional comments (2)

pkg/catalog/loader/loader.go (1)

94-185: Index initialization and one-time persistence look solid

The addition of metadataIndex, indexFilter, and saveMetadataIndexOnce is wired cleanly: you initialize the index (with graceful logging on failure), build a reusable index.Filter from config, and gate Save() behind a sync.OnceFunc, so the gob snapshot is written at most once per process. This matches the “automatic cache lifecycle with graceful fallback” goal without adding complexity to callers.

pkg/catalog/index/index.go (1)

16-37: Index implementation matches the intended persistent LRU metadata cache

NewIndex and NewDefaultIndex set up an otter-backed cache with a reasonable MaximumWeight and a weigher that tracks key and metadata field sizes, then persist it via gob to index.gob with a versioned cacheSnapshot. Get's IsValid check and async Delete, combined with Save's atomic temp-file rename and Load's "delete on corruption or version mismatch" behavior, give you a robust, low-friction cache that degrades safely when something goes wrong.

Filter/FilterFunc/All/GetAll/Count are straightforward wrappers over cache.All() under RW locks and line up with how the loader uses them. The external RWMutex correctly synchronizes multi-call sequences (e.g., Load's decode + Set loop, Save's snapshot construction, Filter's iteration) for consistency guarantees that single otter operations alone cannot provide. Just keep in mind to update both the weigher here and the note in Metadata if you add more fields so eviction behavior remains roughly proportional to actual memory use.

[dwisiswant0]

@coderabbitai wdym by "if you ever hit edge cases on low-resolution filesystems, you might consider allowing a small skew"?

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

@dwisiswant0, great question! Let me clarify that comment about low-resolution filesystems.

Low-resolution filesystems are filesystems that store file modification times with limited precision. For example:

• FAT32: 2-second resolution for modification times
• Some network filesystems (NFS, CIFS/SMB): Can have 1-2 second granularity depending on configuration
• Modern filesystems (ext4, NTFS, APFS): Nanosecond or microsecond precision

The potential edge case happens in IsValid() when comparing m.ModTime with the current file's modification time. If you:

1. Read a template and cache its metadata at time T
2. The file is modified at time T + 0.5 seconds
3. On a FAT32 filesystem, both timestamps might be rounded to the same 2-second boundary

The cache would incorrectly consider the entry valid even though the file changed.

"Small skew" means adding a tolerance window to the comparison, something like:

// Instead of exact equality
if !m.ModTime.Equal(fileInfo.ModTime()) {
    return false
}

// Allow ±1 second tolerance
timeDiff := m.ModTime.Sub(fileInfo.ModTime()).Abs()
if timeDiff > time.Second {
    return false
}

That said, your current implementation is perfectly fine for Nuclei's use case:

• Most users run on modern filesystems (ext4, NTFS, APFS) with high-resolution timestamps
• Template files are typically modified through explicit edits/updates, not rapid successive changes
• The rare false-negative (marking valid cache as invalid) just triggers a re-parse, which is harmless

This was just a theoretical edge case to be aware of—I wouldn't recommend changing anything unless you actually encounter issues in the field! 🙂

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (2)

pkg/catalog/loader/loader.go (2)

318-340: Consider surfacing or logging include/exclude resolution errors when building the index filter

buildIndexFilter currently discards errors from Catalog.GetTemplatesPath for IncludeTemplates / ExcludeTemplates, which means bad IDs/paths in those flags silently degrade to “no-op” behavior for the force-include/exclude semantics. Similarly, loadTemplatesIndex only logs creation/load failures and returns nil, which is fine for graceful fallback but makes metadataIndex == nil a special behavior case elsewhere.

If you want better debuggability without changing behavior, consider:

• Logging the errs from GetTemplatesPath here (reusing logErroredTemplates) so misconfigured include/exclude templates are visible even before any templates are loaded.
• Optionally documenting that NewDefaultIndex failures are the only reason metadataIndex is nil, so future uses don’t inadvertently assume “index always present”.

This is non-blocking but would make filter/debug behavior more transparent.

Also applies to: 342-356

---

475-495: Validation now ignores index-based filters; consider aligning with load-time behavior

ValidateTemplates now builds simple map[string]struct{} sets for all templatePaths and workflowPaths, then validates everything via areTemplatesValid / areWorkflowsValid. This is simpler than the previous path‑filtering, but it means validation no longer respects the index‑based filters (authors, tags, severities, protocols, include/exclude templates).

If users run validation with the same CLI filters they use for execution (e.g. -tags or -severity), they might expect the validated set to match the loaded set.

Not strictly required for correctness, but for consistency and performance you might want to:

• Apply indexFilter.Matches (and any path include/exclude logic) when building templatePathsMap / workflowPathsMap, using cached or on‑the‑fly metadata similarly to the loading paths; or
• Document that ValidateTemplates validates all resolved templates/workflows regardless of filter flags.

Given the cost of full validation, aligning it with the index filter would be a nice future improvement.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 93dd108 and 8e0a55d.

📒 Files selected for processing (1)

• pkg/catalog/loader/loader.go (9 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.go: Format Go code using go fmt ./...
Run static analysis using go vet ./...

Files:

• pkg/catalog/loader/loader.go

🧬 Code graph analysis (1)

pkg/catalog/loader/loader.go (3)

pkg/catalog/index/index.go (2)

• Index (32-37)
• NewDefaultIndex (101-103)

pkg/catalog/index/filter.go (1)

• Filter (20-56)

pkg/catalog/index/metadata.go (2)

• Metadata (14-55)
• NewMetadataFromTemplate (58-73)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)

• GitHub Check: Tests (ubuntu-latest)
• GitHub Check: Tests (windows-latest)
• GitHub Check: Tests (macOS-latest)

🔇 Additional comments (2)

pkg/catalog/loader/loader.go (2)

17-17: Metadata index wiring and save lifecycle look sound

The introduction of metadataIndex, indexFilter, and saveMetadataIndexOnce plus their initialization in New is cohesive: index/load happen once per store, and sync.OnceFunc prevents redundant saves even when multiple loading paths call defer saveMetadataIndexOnce(). Early-return in the save closure when metadataIndex is nil also ensures graceful fallback when the cache cannot be created. No functional issues spotted here.

Also applies to: 94-103, 141-143, 152-158, 172-185

---

611-629: Workflow loading preallocation is fine

Preallocating loadedWorkflows with len(includedWorkflows) is a small but sensible optimization; the control flow and error handling around LoadWorkflow/Parse remain unchanged. No issues here.

[coderabbitai]

Actionable comments posted: 0

♻️ Duplicate comments (1)

pkg/catalog/loader/loader.go (1)

360-367: Nil-pointer risk in LoadTemplatesOnlyMetadata when index is warm but parser cache is cold

When the metadata index is warm and tagFilter does not force a load, validPaths can contain entries for which no template has been parsed in this run:

if store.metadataIndex != nil {
    if metadata, found := store.metadataIndex.Get(templatePath); found {
        if !indexFilter.Matches(metadata) {
            continue
        }
        if store.tagFilter != nil {
            loaded, err := Parser.LoadTemplate(...)
            // ...
        }
        validPaths[templatePath] = struct{}{}
        continue
    }
}

Those paths are later processed via:

templatesCache := store.parserCacheOnce()
for templatePath := range validPaths {
    template, _, _ := templatesCache.Has(templatePath)

    if len(template.RequestsHeadless) > 0 && !store.config.ExecutorOptions.Options.Headless {
        // ...
    }
    // ...
    if template != nil {
        // ...
    }
}

If the parser cache has no entry for templatePath (a normal case when you rely solely on the persisted metadata cache), template is nil and the first len(template.RequestsHeadless) dereference will panic. This is exactly the “warm metadata / cold parser cache” scenario.

To keep the short‑circuiting benefit while avoiding the panic, ensure you load the template on demand when template is nil before dereferencing any fields. For example:

-   for templatePath := range validPaths {
-       template, _, _ := templatesCache.Has(templatePath)
+   for templatePath := range validPaths {
+       template, _, _ := templatesCache.Has(templatePath)
+
+       if template == nil {
+           loaded, err := store.config.ExecutorOptions.Parser.LoadTemplate(templatePath, store.tagFilter, nil, store.config.Catalog)
+           if !loaded {
+               if err != nil {
+                   if strings.Contains(err.Error(), templates.ErrExcluded.Error()) {
+                       stats.Increment(templates.TemplatesExcludedStats)
+                       if config.DefaultConfig.LogAllEvents {
+                           store.logger.Print().Msgf("[%v] %v\n", aurora.Yellow("WRN").String(), err.Error())
+                       }
+                   } else {
+                       store.logger.Warning().Msg(err.Error())
+                   }
+               }
+               continue
+           }
+
+           if template, _, _ = templatesCache.Has(templatePath); template == nil {
+               continue
+           }
+       }
 
-       if len(template.RequestsHeadless) > 0 && !store.config.ExecutorOptions.Options.Headless {
+       if len(template.RequestsHeadless) > 0 && !store.config.ExecutorOptions.Options.Headless {
            continue
        }
        // ... rest of gating and duplicate-ID logic unchanged ...

This keeps metadata‑based skipping when possible, but guarantees that any template you inspect for headless/code/DAST/self‑contained/file checks is non‑nil, preventing a panic and preserving gating semantics.

Also applies to: 368-413, 435-468

🧹 Nitpick comments (3)

pkg/catalog/loader/loader.go (3)

94-103: Metadata index lifecycle wiring looks sound; be aware of single-save semantics

The metadataIndex, indexFilter, and saveMetadataIndexOnce wiring is clean and guarded against nil indexes, so failures in creating/loading the index degrade gracefully.

One behavioral wrinkle: saveMetadataIndexOnce will only ever persist once per Store instance. If, in any workflow, you end up calling both LoadTemplatesOnlyMetadata and LoadTemplatesWithTags (or other future callers that mutate metadata) on the same Store, only the first completion will flush index changes. If that’s not intentional, consider either:

• moving persistence behind an explicit Close()/Flush() on Store, or
• dropping OnceFunc and saving at each high‑level load, letting the index implementation manage incremental writes or cheap no‑ops.

Also applies to: 172-185

---

318-340: Index filter construction is correct; consider surfacing include/exclude resolution errors

Mapping config fields into index.Filter looks consistent (authors, tags, (exclude|include) IDs/templates, severities, protocols). One thing you might want to improve for UX: GetTemplatesPath errors for IncludeTemplates/ExcludeTemplates are currently discarded.

If users typo a path in -include-templates/-exclude-templates, they’ll get silent misconfiguration. You could reuse logErroredTemplates here, for example:

 func (store *Store) buildIndexFilter() *index.Filter {
-    includeTemplates, _ := store.config.Catalog.GetTemplatesPath(store.config.IncludeTemplates)
-    excludeTemplates, _ := store.config.Catalog.GetTemplatesPath(store.config.ExcludeTemplates)
+    includeTemplates, includeErrs := store.config.Catalog.GetTemplatesPath(store.config.IncludeTemplates)
+    excludeTemplates, excludeErrs := store.config.Catalog.GetTemplatesPath(store.config.ExcludeTemplates)
+    store.logErroredTemplates(includeErrs)
+    store.logErroredTemplates(excludeErrs)

---

342-356: Graceful fallback on index creation/load; consider dropping a bad index instead of reusing it

Creating the default index and logging both creation and load failures gives you a clean fallback (no caching, normal behavior) when the cache directory or gob file is unusable.

If Load() returns an error due to a corrupt on-disk cache, you currently keep using metadataIdx. Depending on Index.Load semantics, that may leave you with partially initialized state. If Load() doesn’t already reset to an empty cache on failure, consider:

    idx, err := index.NewDefaultIndex()
    if err != nil {
        store.logger.Warning().Msgf("Could not create metadata cache: %v", err)
    } else {
        metadataIdx = idx
        if err := metadataIdx.Load(); err != nil {
-           store.logger.Warning().Msgf("Could not load metadata cache: %v", err)
+           store.logger.Warning().Msgf("Could not load metadata cache: %v", err)
+           metadataIdx = nil
        }
    }

so a corrupt file behaves like “no index” instead of “possibly inconsistent index”.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8e0a55d and 0002035.

📒 Files selected for processing (1)

• pkg/catalog/loader/loader.go (9 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.go: Format Go code using go fmt ./...
Run static analysis using go vet ./...

Files:

• pkg/catalog/loader/loader.go

🧬 Code graph analysis (1)

pkg/catalog/loader/loader.go (3)

pkg/catalog/index/index.go (1)

• Index (32-37)

pkg/catalog/index/filter.go (1)

• Filter (20-56)

pkg/catalog/index/metadata.go (2)

• Metadata (14-55)
• NewMetadataFromTemplate (58-73)

🔇 Additional comments (4)

pkg/catalog/loader/loader.go (4)

141-144: Separation between tagFilter (advanced) and indexFilter (basic) is clear and maintainable

The comments and initialization make the division of responsibility explicit: tagFilter for IncludeConditions, indexFilter for all basic include/exclude criteria. This makes the filtering pipeline easier to reason about and aligns with the new index package’s intent.

Also applies to: 151-158

---

481-490: Path map construction for validation is straightforward and efficient

Building templatePathsMap/workflowPathsMap once and passing them into areTemplatesValid/areWorkflowsValid is a clear way to remove the old pathFilter dependency while preserving uniqueness semantics. The map preallocation with len(...) is also a nice touch for avoiding extra allocations.

Also applies to: 491-494

---

614-615: Minor optimization in LoadWorkflows slice initialization

Preallocating loadedWorkflows with len(includedWorkflows) capacity is a sensible micro‑optimization on this hot path and doesn’t change behavior.

---

635-638: Index-backed fast path and fallback behavior in LoadTemplatesWithTags look correct

The new flow correctly:

• Uses cached metadata (metadataIndex.Get) to short‑circuit templates that obviously don’t match indexFilter.
• Still runs LoadTemplate when advanced IncludeConditions or per-call tags may affect inclusion.
• Recomputes metadata from parsed templates and re-applies indexFilter even when metadataIndex is nil, so basic CLI filters (tags, authors, severities, IDs, protocol types, include/exclude templates) continue to work without a usable cache.
• Only updates the index via SetFromTemplate when available, otherwise falls back to NewMetadataFromTemplate.

This preserves pre-index semantics while giving you the intended metadata-based pruning, and the concurrency pattern (shared read-only indexFilter, internally synchronized metadataIndex) is safe.

Also applies to: 684-707, 708-723

[dwisiswant0]

@Mzack9999 - heads up: I just added the memory comparison table from the gctrace results in the PR body. :)

Gonna merge this now.