cache, goroutine and unbounded workers management#6420
Conversation
…s to improve performance. Update template parsing to conditionally retain raw templates based on size constraints.
…chers to enhance performance. Introduce a buffer pool in raw requests to reduce memory allocations. Update template cache management for improved efficiency.
WalkthroughAdds bounded worker pools for template and HTTP execution, introduces shared LRU caches for compiled regex/DSL, applies buffer pooling for raw HTTP headers, adjusts payload/file-ref handling and validation, makes dedupe initialization conditional, extends template cache API, and adds unit tests across affected modules. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Engine
participant Pool as TemplateWorkerPool
participant Tpl as Template
participant Inp as Input
participant Exec as Executer
Engine->>Pool: enqueue(task(index, input, skip))
activate Pool
loop worker picks task
Pool->>Engine: executeTemplateOnInput(template, input)
activate Engine
alt workflow protocol
Engine->>Exec: executeWorkflow(ctx, scanCtx)
else executer path
Engine->>Exec: ExecuteWithResults or Execute
end
Exec-->>Engine: (ok?, results?, err)
Engine-->>Pool: update results / log / in-flight cleanup
deactivate Engine
end
Pool-->>Engine: all workers done
deactivate Pool
sequenceDiagram
autonumber
participant HTTP as executeParallelHTTP
participant Pool as HTTPWorkerPool
participant SPM as spmHandler
participant RL as RateLimiter
participant Net as Network
HTTP->>Pool: enqueue(req, updatedInput)
loop worker processes tasks
Pool->>SPM: Acquire (semaphore)
alt skip due to first-match/unresponsive
Pool->>SPM: Done (skip)
else
Pool->>RL: RateLimitTake
Pool->>Net: execute request
Net-->>Pool: response/result
Pool->>SPM: send result on ResultChan
Pool->>SPM: Done (release)
end
end
HTTP->>SPM: Wait (aggregate / early-stop)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests
Comment |
…ractors and matchers, along with tests and benchmarks for validation
knakul853
added
the
Type: Optimization
…cing test coverage and reliability
knakul853
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 11
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
pkg/protocols/common/generators/validate.go (1)
21-31: Validation now rejects all multiline string payloads — breaks inline payloads.ContainsRune('\n') should not error; inline multiline payloads are valid and handled in load.go. Also don’t return from the entire function when LoadHelperFileFunction is set; just skip file validation for this entry.
- // check if it's a multiline string list - if strings.ContainsRune(payloadType, '\n') { - return errors.New("invalid number of lines in payload") - } + // Inline multiline strings are allowed; loader will parse them. + // Skip filesystem validation for such payloads. + if strings.ContainsRune(payloadType, '\n') { + continue + } @@ - if g.options.LoadHelperFileFunction != nil { - return nil - } + if g.options.LoadHelperFileFunction != nil { + // Skip path validation; custom loader will handle it. + continue + }pkg/templates/templates.go (1)
408-413: Eliminate FileExists pre-check before invoking LoadHelperFile
Replace everyif !strings.ContainsRune(..., '\n') && fileutil.FileExists(...)(at lines 408–413, 419–424, 446–451, 460–465) with a lone newline check that always callsloadFile. InsideloadFile, filter out “not found” errors (do not append toerrs) so Catalog- or TemplatePath-resolved files still load, while inline snippets remain unaffected. Add unit tests for relative helper files via Catalog in code, JavaScript, flow, and multiprotocol branches.pkg/core/executors.go (1)
144-162: In-flight bookkeeping leak on host-errors skipIndex is added to InFlight before the HostErrorsCache check; on skip the entry is never cleaned, impacting resume/memory. Clean up before returning.
Apply:
if e.executerOpts.HostErrorsCache != nil && e.executerOpts.HostErrorsCache.Check(e.executerOpts.ProtocolType.String(), contextargs.NewWithMetaInput(ctx, scannedValue)) { + cleanupInFlight(index) skipEvent := &output.ResultEvent{ TemplateID: template.ID, TemplatePath: template.Path, Info: template.Info, Type: e.executerOpts.ProtocolType.String(), Host: scannedValue.Input, MatcherStatus: false, Error: "host was skipped as it was found unresponsive", Timestamp: time.Now(), }
🧹 Nitpick comments (23)
pkg/protocols/http/raw/raw.go (2)
21-22: Add guardrails to the global buffer pool to avoid retaining huge buffers.sync.Pool can retain very large buffers after spikes. Consider a small helper that resets and only pools buffers under a cap (e.g., 64 KiB). Replace direct Put calls with this helper.
Add near the pool declaration:
const maxPooledBuf = 64 << 10 // 64KiB func putBuf(b *bytes.Buffer) { // Prefer bytes.Buffer.Cap() if on Go ≥1.20; otherwise use cap(b.Bytes()). if cap(b.Bytes()) > maxPooledBuf { return } b.Reset() bufferPool.Put(b) }Then replace bufferPool.Put(...) call sites with putBuf(...).
309-321: Cookie buffer pooling is fine; tighten it and drop fmt overhead.String() creates a copy, so it’s safe post-Reset. Minor optimizations: build with WriteString/WriteByte, trim the trailing “; ” once, and reset before pooling.
Apply this diff:
- buff := bufferPool.Get().(*bytes.Buffer) - buff.Reset() + buff := bufferPool.Get().(*bytes.Buffer) + buff.Reset() for _, cookie := range s.Data.Cookies { - fmt.Fprintf(buff, "%s=%s; ", cookie.Key, cookie.Value) + buff.WriteString(cookie.Key) + buff.WriteByte('=') + buff.WriteString(cookie.Value) + buff.WriteString("; ") } if buff.Len() > 0 { + // remove trailing "; " + if buff.Len() >= 2 { + buff.Truncate(buff.Len() - 2) + } if val, ok := r.Headers["Cookie"]; ok { - r.Headers["Cookie"] = strings.TrimSuffix(strings.TrimSpace(val), ";") + "; " + buff.String() + r.Headers["Cookie"] = strings.TrimSuffix(strings.TrimSpace(val), ";") + "; " + buff.String() } else { r.Headers["Cookie"] = buff.String() } } - bufferPool.Put(buff) + buff.Reset() + bufferPool.Put(buff)pkg/operators/extractors/compile.go (2)
14-19: Consider TTL/eviction or configurability for these globals.The caches never evict once filled (up to 4096). Add TTL/LRU (aligned with matchers regex result cache) or make sizes configurable via options/env for long-running processes.
14-19: Nit: acronym casing.Prefer extractorDSLCache/extractorMaxDSLCacheSize for Go initialism style consistency.
pkg/operators/matchers/regexcache.go (4)
20-25: Avoid retaining large corpora as map keys.Keying by full corpus string risks prolonged retention of big buffers. Consider key = (pattern, xxhash64(corpus)) and store corpus length; confirm equality on hit to guard against rare collisions.
Also applies to: 27-33, 64-82
35-46: Eagerly drop expired entries on read (optional).get() returns “miss” for expired entries but leaves them in memory until the next set(). Optionally delete-on-read when expired to reclaim memory sooner.
84-101: Document non-mutability of returned slices.Callers must not mutate the returned slices (shared with cache). Add a comment or return copies if mutation is a concern.
Also applies to: 103-127
27-33: Expose light config/reset hooks for tests and tuning.Add methods to adjust ttl/maxSize and reset() under test to avoid direct map access.
I can wire a tiny config struct and rc.reset() if you want.
pkg/operators/matchers/regexcache_test.go (2)
34-37: Strengthen assertion or lower flakiness.Performance checks are noisy. Either assert cachedTime <= uncachedTime/2 on larger corpus or drop the timing assertion entirely and only validate cache hits.
47-65: Add submatch caching test and n-behavior test.Cover CachedFindAllStringSubmatch and verify that subsequent calls with larger n return more results (after applying the fix).
pkg/operators/matchers/match.go (1)
134-147: Guard against pathological match explosion.For inputs with many matches (e.g., “.” or “\w+”), appending all matches can bloat memory. Consider a soft cap (e.g., first N matches per regex) when !MatchAll, or make N configurable.
pkg/operators/matchers/compile.go (2)
15-21: Global caches lack eviction and compile de-duplication under load.
- Without eviction, the first 4096 unique entries persist forever; after that, no new items are cached.
- Concurrent identical compiles can still happen before Load hits; consider singleflight to coalesce.
Example (outline, not exhaustive):
+// optional: coalesce duplicate compiles +// import "golang.org/x/sync/singleflight" +var reGroup singleflight.Group +var dslGroup singleflight.Group…and wrap compile paths with group.Do(key, func() (any, error) { ... }).
15-21: Optional: bound and evict with a tiny LRU.If hit rate matters beyond the first 4096 entries, a simple size-bounded LRU (list+map) with a mutex will outperform sync.Map+size threshold for hot workloads.
I can provide a minimal lock-based LRU tailored for *regexp.Regexp and *govaluate.EvaluableExpression if desired.
pkg/protocols/javascript/js.go (1)
814-819: Skip whitespace-only dumps in prettyPrint (minor).Early-return is good. Also treat whitespace-only buffers as empty.
func prettyPrint(templateId string, buff string) { - if buff == "" { + if strings.TrimSpace(buff) == "" { return } lines := strings.Split(buff, "\n") - final := make([]string, 0, len(lines)) + final := make([]string, 0, len(lines))pkg/protocols/common/generators/load.go (2)
20-32: Avoid variable shadowing and improve readability in fast-path.Rename local
payloadsslice to avoid shadowing the outerpayloadsmap.- payloads, err := generator.loadPayloadsFromFile(file) + items, err := generator.loadPayloadsFromFile(file) if err != nil { return nil, errors.Wrap(err, "could not load payloads") } - loadedPayloads[name] = payloads + loadedPayloads[name] = items
34-38: Trim and drop empty lines for inline multiline payloads.Match file-loading behavior (which skips empty lines) to avoid inserting empty payloads when input ends with a newline.
- elements := strings.Split(pt, "\n") + raw := strings.Split(pt, "\n") + elements := make([]string, 0, len(raw)) + for _, e := range raw { + e = strings.TrimSpace(e) + if e != "" { + elements = append(elements, e) + } + }pkg/protocols/http/request.go (1)
299-305: Optional: consider shrinking workers when concurrency decreases.You grow the pool when PayloadConcurrency increases but never downscale. Not critical, but you could stop excess workers by closing a separate “quit” channel or by checking spmHandler.Size() periodically to reduce goroutines under sustained downsizing.
pkg/protocols/http/request_test.go (1)
275-323: Potential test flakiness: StopAtFirstMatch under parallelismWith Threads=2 both payloads may start before the stop signal propagates, producing 2 matches intermittently. To deflake, gate the second payload (e.g., server delay for x=2) or increase payload count and assert matches <= 1 by enforcing cancellation earlier in the request execution.
pkg/templates/compile.go (1)
577-580: Good: bound RawTemplate retention by omit flag and sizeConditionally keeping RawTemplate when OmitTemplate is false and size <= MaxTemplateFileSizeForEncoding reduces memory bloat. Consider documenting the 1 MiB threshold in user-facing docs and a debug log path when large templates are omitted.
pkg/core/executors.go (1)
91-113: Optional: buffer tasks channelA small buffer (e.g., workerCount) reduces producer/consumer lockstep without changing ordering or bounds.
Apply:
- tasks := make(chan task) + tasks := make(chan task, workerCount)pkg/core/executors_test.go (3)
3-12: Harden stub conformance, make callback assertions deterministic, and validate event contents.
- Add a compile-time assertion that fakeExecuter implements protocols.Executer.
- Use atomic counter to avoid potential races if Callback is invoked from another goroutine.
- Assert the content of the emitted event to tighten expectations.
Run: go test -race ./pkg/core -run Test_executeTemplateOnInput_CallbackPath
import ( "context" "testing" + "sync/atomic" "github.com/projectdiscovery/nuclei/v3/pkg/output" + "github.com/projectdiscovery/nuclei/v3/pkg/protocols" "github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs" "github.com/projectdiscovery/nuclei/v3/pkg/scan" "github.com/projectdiscovery/nuclei/v3/pkg/templates" "github.com/projectdiscovery/nuclei/v3/pkg/types" ) // fakeExecuter is a simple stub for protocols.Executer used to test executeTemplateOnInput type fakeExecuter struct { withResults bool } +var _ protocols.Executer = (*fakeExecuter)(nil) + func (f *fakeExecuter) Compile() error { return nil } func (f *fakeExecuter) Requests() int { return 1 } func (f *fakeExecuter) Execute(ctx *scan.ScanContext) (bool, error) { return !f.withResults, nil } func (f *fakeExecuter) ExecuteWithResults(ctx *scan.ScanContext) ([]*output.ResultEvent, error) { if !f.withResults { return nil, nil } return []*output.ResultEvent{{Host: "h"}}, nil } @@ func Test_executeTemplateOnInput_CallbackPath(t *testing.T) { e := newTestEngine() - called := 0 - e.Callback = func(*output.ResultEvent) { called++ } + var called int32 + got := make([]*output.ResultEvent, 0, 1) + e.Callback = func(ev *output.ResultEvent) { + atomic.AddInt32(&called, 1) + got = append(got, ev) + } @@ if !ok { t.Fatalf("expected match true") } - if called == 0 { + if len(got) != 1 { + t.Fatalf("expected 1 result, got %d", len(got)) + } + if got[0].Host != "h" { + t.Fatalf("unexpected event: %+v", got[0]) + } + if atomic.LoadInt32(&called) == 0 { t.Fatalf("expected callback to be called") } }Also applies to: 14-17, 34-52
34-34: Enable parallelism in tests.Safe here and speeds up CI a bit.
func Test_executeTemplateOnInput_CallbackPath(t *testing.T) { + t.Parallel() @@ func Test_executeTemplateOnInput_ExecutePath(t *testing.T) { + t.Parallel()Also applies to: 54-54
67-67: Add a “no match” branch test to cover ok=false.Covers the path where neither ExecuteWithResults nor Execute produce a match.
+// noMatchExecuter returns no results and no match +type noMatchExecuter struct{} + +func (*noMatchExecuter) Compile() error { return nil } +func (*noMatchExecuter) Requests() int { return 1 } +func (*noMatchExecuter) Execute(ctx *scan.ScanContext) (bool, error) { return false, nil } +func (*noMatchExecuter) ExecuteWithResults(ctx *scan.ScanContext) ([]*output.ResultEvent, error) { + return nil, nil +} + +func Test_executeTemplateOnInput_NoMatch(t *testing.T) { + t.Parallel() + e := newTestEngine() + tpl := &templates.Template{} + tpl.Executer = &noMatchExecuter{} + + ok, err := e.executeTemplateOnInput(context.Background(), tpl, &contextargs.MetaInput{Input: "x"}) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if ok { + t.Fatalf("expected match false") + } +}
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (20)
pkg/core/executors.go(4 hunks)pkg/core/executors_test.go(1 hunks)pkg/operators/extractors/compile.go(3 hunks)pkg/operators/extractors/extract.go(2 hunks)pkg/operators/matchers/compile.go(3 hunks)pkg/operators/matchers/match.go(3 hunks)pkg/operators/matchers/match_test.go(1 hunks)pkg/operators/matchers/regexcache.go(1 hunks)pkg/operators/matchers/regexcache_test.go(1 hunks)pkg/protocols/common/generators/load.go(1 hunks)pkg/protocols/common/generators/validate.go(1 hunks)pkg/protocols/http/build_request.go(1 hunks)pkg/protocols/http/raw/raw.go(5 hunks)pkg/protocols/http/request.go(3 hunks)pkg/protocols/http/request_test.go(2 hunks)pkg/protocols/javascript/js.go(1 hunks)pkg/reporting/reporting.go(1 hunks)pkg/templates/cache.go(2 hunks)pkg/templates/compile.go(2 hunks)pkg/templates/templates.go(4 hunks)
🧰 Additional context used
🧠 Learnings (5)
📚 Learning: 2025-07-16T21:27:14.937Z
Learnt from: hdm
PR: projectdiscovery/nuclei#6322
File: pkg/templates/compile.go:79-81
Timestamp: 2025-07-16T21:27:14.937Z
Learning: To make the template caching mechanism in pkg/templates/compile.go production-ready, DSLs need to be updated to use runtime options instead of cached variables, rather than restoring the Compile() calls on each request.
Applied to files:
pkg/templates/compile.gopkg/operators/extractors/compile.gopkg/operators/matchers/compile.gopkg/templates/cache.go
📚 Learning: 2025-07-16T21:27:14.937Z
Learnt from: hdm
PR: projectdiscovery/nuclei#6322
File: pkg/templates/compile.go:79-81
Timestamp: 2025-07-16T21:27:14.937Z
Learning: In pkg/templates/compile.go, the template caching mechanism intentionally skips calling Compile() on copied requests to achieve performance benefits. This is the intended design, not a bug. The current implementation isn't production-ready but represents the desired direction.
Applied to files:
pkg/templates/compile.gopkg/operators/extractors/compile.gopkg/operators/matchers/compile.go
📚 Learning: 2025-07-16T21:28:08.073Z
Learnt from: hdm
PR: projectdiscovery/nuclei#6322
File: pkg/templates/compile.go:59-78
Timestamp: 2025-07-16T21:28:08.073Z
Learning: The shallow copy behavior (`tplCopy := *value`) in pkg/templates/compile.go is intentional design for the template caching mechanism. The partial-copy approach is part of the performance optimization strategy, not a bug requiring deep copying.
Applied to files:
pkg/templates/compile.gopkg/templates/cache.go
📚 Learning: 2025-06-30T16:34:42.125Z
Learnt from: dwisiswant0
PR: projectdiscovery/nuclei#6290
File: pkg/protocols/http/build_request.go:457-464
Timestamp: 2025-06-30T16:34:42.125Z
Learning: In the projectdiscovery/retryablehttp-go package, the Request struct embeds URL fields directly, making req.Scheme, req.Host, and other URL fields accessible directly on the Request object instead of requiring req.URL.Scheme, req.URL.Host, etc.
Applied to files:
pkg/protocols/http/raw/raw.gopkg/protocols/http/build_request.go
📚 Learning: 2024-12-13T08:42:57.506Z
Learnt from: dwisiswant0
PR: projectdiscovery/nuclei#5772
File: pkg/authprovider/authx/cookies_auth.go:36-38
Timestamp: 2024-12-13T08:42:57.506Z
Learning: In the `ApplyOnRR` method of `CookiesAuthStrategy` in `pkg/authprovider/authx/cookies_auth.go`, when setting cookies, only remove the cookies that are about to be set to avoid removing essential session cookies.
Applied to files:
pkg/protocols/http/raw/raw.go
🧬 Code graph analysis (10)
pkg/templates/compile.go (3)
pkg/protocols/headless/engine/page.go (1)
Options(50-54)pkg/types/types.go (1)
Options(33-465)pkg/protocols/protocols.go (1)
MaxTemplateFileSizeForEncoding(45-45)
pkg/core/executors_test.go (5)
pkg/scan/scan_context.go (1)
ScanContext(22-44)pkg/output/output.go (1)
ResultEvent(146-221)pkg/core/engine.go (1)
Engine(18-24)pkg/protocols/protocols.go (1)
Executer(49-58)pkg/protocols/common/contextargs/metainput.go (1)
MetaInput(18-30)
pkg/operators/extractors/extract.go (1)
pkg/operators/matchers/regexcache.go (1)
CachedFindAllStringSubmatch(103-127)
pkg/operators/extractors/compile.go (1)
pkg/operators/common/dsl/dsl.go (2)
HelperFunctions(18-18)CompilationError(126-129)
pkg/reporting/reporting.go (1)
internal/runner/runner.go (1)
New(109-398)
pkg/protocols/http/request_test.go (3)
pkg/protocols/common/contextargs/contextargs.go (2)
Context(22-33)NewWithInput(48-64)pkg/testutils/testutils.go (2)
NewMockExecuterOptions(89-106)TemplateInfo(82-86)pkg/output/output.go (2)
InternalEvent(93-93)InternalWrappedEvent(100-113)
pkg/operators/matchers/compile.go (1)
pkg/operators/common/dsl/dsl.go (2)
HelperFunctions(18-18)CompilationError(126-129)
pkg/protocols/http/request.go (2)
pkg/protocols/common/contextargs/contextargs.go (1)
Context(22-33)pkg/progress/progress.go (1)
Progress(17-35)
pkg/operators/matchers/match.go (1)
pkg/operators/matchers/matchers.go (2)
ANDCondition(145-145)ORCondition(147-147)
pkg/core/executors.go (6)
pkg/protocols/common/contextargs/metainput.go (1)
MetaInput(18-30)pkg/core/engine.go (2)
Engine(18-24)New(27-34)pkg/protocols/common/contextargs/contextargs.go (2)
Context(22-33)New(36-38)pkg/scan/scan_context.go (1)
NewScanContext(47-49)pkg/templates/types/types.go (1)
WorkflowProtocol(33-33)pkg/protocols/protocols.go (1)
Executer(49-58)
🔇 Additional comments (18)
pkg/protocols/http/raw/raw.go (1)
11-11: Import of sync is appropriate.Needed for the new buffer pool. No concerns here.
pkg/reporting/reporting.go (2)
195-201: Initialize dedupe only when it’s actually used — good optimization.Gates dedupe storage behind presence of DB path or at least one tracker/exporter. This aligns with memory/IO reduction goals; CreateIssue already guards on c.dedupe != nil. LGTM.
301-307: Align ShouldFilter documentation with its semantics
The loop and all tracker implementations treat a true return value as “include this event,” so the interface comment is inverted. In pkg/reporting/reporting.go replace:// ShouldFilter determines if the event should be filtered outwith something like:
// ShouldFilter returns true if the event meets this tracker’s allow/deny criteria and should be processedLikely an incorrect or invalid review comment.
pkg/operators/extractors/compile.go (1)
71-76: Ensure dsl.HelperFunctions is immutable
Caching compiled expressions by DSL string alone assumes the global HelperFunctions map never changes. Confirm it can’t be modified at runtime—otherwise include its version in the cache key or freeze a copy.pkg/operators/matchers/regexcache.go (1)
84-101: Audit call sites passing non–-1limits
No occurrences were found wherecachedFindAllStringorCachedFindAllStringSubmatchare invoked with a literal or variable third argument other than-1—only their own definitions matched. If you retain this partial‐caching behavior, ensure all callers explicitly passn = -1to avoid truncated results.pkg/operators/matchers/match_test.go (1)
87-87: LGTM: range now iterates values, not indices.This fixes the DSL test to feed actual strings instead of 0/1.
pkg/operators/extractors/extract.go (2)
21-27: Nice short-circuit with LiteralPrefix.Good skip on impossible matches and switch to cached submatches for reuse.
27-27: Downstream dependency on cache semantics.This calls CachedFindAllStringSubmatch with n = -1, so it relies on the cache storing full results. Ensure the regexcache fix lands to avoid future surprises if other callers pass n != -1.
pkg/operators/matchers/match.go (2)
134-147: LGTM: single-scan with cachedFindAllString.Good consolidation around a single cached scan and correct AND/OR fall-through handling.
23-25: No action required: maxRegexScanBytes is referenced in regexcache.go
The constant is used in pkg/operators/matchers/regexcache.go (lines 85 and 105), so it won’t be “declared and not used.”pkg/operators/matchers/compile.go (2)
8-8: Import sync/atomic if you adopt the atomic counters.- "sync" + "sync" + "sync/atomic"
89-108: Atomic DSL cache size implemented; no further action needed. EvaluableExpression is safe for concurrent Evaluate calls, and HelperFunctions is only initialized once (no post-init mutations), so freezing or namespacing by function map isn’t required.pkg/protocols/http/build_request.go (1)
71-73: LGTM: index-based first-line trim and unconditional single-line trim.This tightens parsing without extra allocations and handles single-line inputs consistently.
Also applies to: 81-82
pkg/protocols/http/request_test.go (2)
262-274: Interface conformance looks fineMinimal fakeHostErrorsCache correctly covers the methods typically used in execution paths; signatures match Context usage. No action needed here.
324-373: HTTP ExecuteWithResults correctly checks HostErrorsCache
Verified that Request.ExecuteWithResults invokes isUnresponsiveAddress (which calls HostErrorsCache.Check) at multiple points to skip unresponsive hosts, so this test covers the HTTP-layer skip logic.pkg/templates/compile.go (1)
83-87: In-place UpdateOptions mutates cached requests (data race/cross-run pollution)tplCopy is a shallow copy; requests inside are shared with the cached template. Updating r in place overwrites cached state, causing cross-run option bleed and potential races under concurrent Parse(). Restore per-request struct copy before UpdateOptions.
Apply (pattern shown for one slice; replicate for all request slices):
- for i, r := range tplCopy.RequestsHTTP { - if r != nil { - r.UpdateOptions(tplCopy.Options) - tplCopy.RequestsHTTP[i] = r - } - } + for i, r := range tplCopy.RequestsHTTP { + if r != nil { + rCopy := *r + rCopy.UpdateOptions(tplCopy.Options) + tplCopy.RequestsHTTP[i] = &rCopy + } + }Also applies to: 89-93, 95-99, 101-105, 107-111, 113-117, 119-123, 125-129, 131-135, 137-141
⛔ Skipped due to learnings
Learnt from: hdm PR: projectdiscovery/nuclei#6322 File: pkg/templates/compile.go:59-78 Timestamp: 2025-07-16T21:28:08.073Z Learning: The shallow copy behavior (`tplCopy := *value`) in pkg/templates/compile.go is intentional design for the template caching mechanism. The partial-copy approach is part of the performance optimization strategy, not a bug requiring deep copying.Learnt from: hdm PR: projectdiscovery/nuclei#6322 File: pkg/templates/compile.go:79-81 Timestamp: 2025-07-16T21:27:14.937Z Learning: To make the template caching mechanism in pkg/templates/compile.go production-ready, DSLs need to be updated to use runtime options instead of cached variables, rather than restoring the Compile() calls on each request.pkg/core/executors.go (1)
84-112: Worker-pool structure LGTMSwitch to bounded workers removes unbounded goroutine fanout and simplifies lifecycle. No concerns here.
pkg/core/executors_test.go (1)
34-66: Overall: solid coverage of both primary branches.Tests exercise the Callback path and the direct Execute path and use a minimal Engine scaffold appropriately.
…caching logic for better performance
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
pkg/operators/extractors/compile.go (1)
57-67: Shadowing bug in JSON query handling leads to wrong/error-prone messages.You shadow the loop variable
querywith the parsed AST viaquery, err := gojq.Parse(query). On parse error, the%sformats a non-string (nil AST), and even on success error messages show the AST, not the original string. Fix by using distinct names and quoting the original string.- for _, query := range e.JSON { - query, err := gojq.Parse(query) - if err != nil { - return fmt.Errorf("could not parse json: %s", query) - } - compiled, err := gojq.Compile(query) - if err != nil { - return fmt.Errorf("could not compile json: %s", query) - } - e.jsonCompiled = append(e.jsonCompiled, compiled) - } + for _, expr := range e.JSON { + parsed, err := gojq.Parse(expr) + if err != nil { + return fmt.Errorf("could not parse json: %q", expr) + } + compiled, err := gojq.Compile(parsed) + if err != nil { + return fmt.Errorf("could not compile json: %q", expr) + } + e.jsonCompiled = append(e.jsonCompiled, compiled) + }pkg/core/executors.go (1)
143-161: In-flight leak on host-error skip: index added to InFlight but never cleaned upWhen HostErrorsCache triggers, no task is enqueued, so cleanupInFlight is never called. This corrupts resume state.
Apply either approach A (minimal) or B (preferable):
A) Minimal cleanup before early return:
if e.executerOpts.HostErrorsCache != nil && e.executerOpts.HostErrorsCache.Check(e.executerOpts.ProtocolType.String(), contextargs.NewWithMetaInput(ctx, scannedValue)) { skipEvent := &output.ResultEvent{ TemplateID: template.ID, TemplatePath: template.Path, Info: template.Info, Type: e.executerOpts.ProtocolType.String(), Host: scannedValue.Input, MatcherStatus: false, Error: "host was skipped as it was found unresponsive", Timestamp: time.Now(), } if e.Callback != nil { e.Callback(skipEvent) } else if e.executerOpts.Output != nil { _ = e.executerOpts.Output.Write(skipEvent) } + cleanupInFlight(index) return true }B) Preferable: move InFlight insertion to after the host-error check so skipped hosts are never tracked as in-flight.
♻️ Duplicate comments (4)
pkg/operators/matchers/match.go (2)
113-126: Case-insensitive LiteralPrefix short-circuit — fixed as suggested.Skipping the prefix fast-path when "(?i" exists avoids false negatives. LGTM.
128-136: OR fast-path now uses cachedFindAllString — good optimization.Reusing the cache avoids double scans and unifies behavior. LGTM.
pkg/core/executors.go (2)
217-239: Good: callback path now propagates errors and derives match from resultsThis addresses the prior “don’t swallow ExecuteWithResults error” concern.
29-40: Self-contained path still swallows ExecuteWithResults errors and forces match=trueMirror executeTemplateOnInput: propagate errors and compute match from len(results).
Apply:
- if e.Callback != nil { - if results, err := template.Executer.ExecuteWithResults(ctx); err == nil { - for _, result := range results { - e.Callback(result) - } - } - - match = true - } else { - match, err = template.Executer.Execute(ctx) - } + if e.Callback != nil { + var res []*output.ResultEvent + res, err = template.Executer.ExecuteWithResults(ctx) + if err == nil { + for _, r := range res { + e.Callback(r) + } + match = len(res) > 0 + } + } else { + match, err = template.Executer.Execute(ctx) + }
🧹 Nitpick comments (7)
pkg/templates/templates_test.go (2)
12-12: Run test in parallel to keep suite fast.t.Parallel() is safe here and improves throughput. Included in the main diff.
12-36: Consider a simple concurrency/race test for the cache.To validate thread-safety under PR’s worker-pool changes, add a targeted race test:
// pkg/templates/templates_race_test.go package templates import ( "fmt" "sync" "testing" ) func TestCacheConcurrentStoreAndGet(t *testing.T) { t.Parallel() c := NewCache() tpl := &Template{ID: "x"} var wg sync.WaitGroup for i := 0; i < 256; i++ { wg.Add(1) go func(i int) { defer wg.Done() id := fmt.Sprintf("id-%d", i) c.Store(id, tpl, nil, nil) if _, err := c.Get(id); err != nil { t.Errorf("get %s: %v", id, err) } }(i) } wg.Wait() }pkg/operators/matchers/match.go (1)
148-149: Comment correctness nit.The comment mentions “OR (and MatchAll true)” but the code appends regardless of condition when aggregating. Clarify to avoid confusion.
- // If the condition was an OR (and MatchAll true), we still need to gather all + // When aggregating (e.g., MatchAll), gather all matches regardless of conditionpkg/operators/matchers/compile.go (1)
16-23: Cache structure and counters — sane defaults.4096 entries for both caches with atomic size tracking is a reasonable starting point. Consider future configurability via env/opts if needed.
pkg/operators/matchers/match_test.go (1)
213-220: New test for case-insensitive prefix skip — valuable regression coverage.Validates the optimized path doesn’t cause false negatives. LGTM.
Consider adding a scoped flag case to cover inline groups:
+func TestMatchRegex_ScopedCaseInsensitiveGroup(t *testing.T) { + m := &Matcher{Type: MatcherTypeHolder{MatcherType: RegexMatcher}, Condition: "or", Regex: []string{"a(?i)b"}} + require.NoError(t, m.CompileMatchers()) + ok, got := m.MatchRegex("zzz aB yyy") + require.True(t, ok) + require.NotEmpty(t, got) +}pkg/core/executors.go (2)
91-93: Optional: buffer tasks channel to decouple producer/consumersA small buffer reduces scheduler contention without changing bounds.
Apply:
-tasks := make(chan task) +tasks := make(chan task, workerCount)
208-214: Nit: set results only on true to avoid no-op CASAvoid CAS(false,false) in the hot path.
Apply:
- results.CompareAndSwap(false, match) + if match { + results.Store(true) + }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (13)
pkg/core/executors.go(4 hunks)pkg/core/executors_test.go(1 hunks)pkg/operators/extractors/compile.go(3 hunks)pkg/operators/matchers/compile.go(3 hunks)pkg/operators/matchers/match.go(3 hunks)pkg/operators/matchers/match_test.go(2 hunks)pkg/operators/matchers/regexcache.go(1 hunks)pkg/operators/matchers/regexcache_test.go(1 hunks)pkg/protocols/http/raw/raw.go(5 hunks)pkg/protocols/http/raw/raw_test.go(1 hunks)pkg/protocols/http/request.go(3 hunks)pkg/templates/cache.go(2 hunks)pkg/templates/templates_test.go(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (6)
- pkg/operators/matchers/regexcache_test.go
- pkg/operators/matchers/regexcache.go
- pkg/core/executors_test.go
- pkg/protocols/http/request.go
- pkg/protocols/http/raw/raw.go
- pkg/templates/cache.go
🧰 Additional context used
🧠 Learnings (3)
📓 Common learnings
Learnt from: hdm
PR: projectdiscovery/nuclei#6322
File: pkg/templates/compile.go:79-81
Timestamp: 2025-07-16T21:27:14.937Z
Learning: To make the template caching mechanism in pkg/templates/compile.go production-ready, DSLs need to be updated to use runtime options instead of cached variables, rather than restoring the Compile() calls on each request.
Learnt from: hdm
PR: projectdiscovery/nuclei#6322
File: pkg/templates/compile.go:79-81
Timestamp: 2025-07-16T21:27:14.937Z
Learning: In pkg/templates/compile.go, the template caching mechanism intentionally skips calling Compile() on copied requests to achieve performance benefits. This is the intended design, not a bug. The current implementation isn't production-ready but represents the desired direction.
📚 Learning: 2025-07-16T21:27:14.937Z
Learnt from: hdm
PR: projectdiscovery/nuclei#6322
File: pkg/templates/compile.go:79-81
Timestamp: 2025-07-16T21:27:14.937Z
Learning: To make the template caching mechanism in pkg/templates/compile.go production-ready, DSLs need to be updated to use runtime options instead of cached variables, rather than restoring the Compile() calls on each request.
Applied to files:
pkg/operators/matchers/compile.gopkg/operators/extractors/compile.go
📚 Learning: 2025-07-16T21:27:14.937Z
Learnt from: hdm
PR: projectdiscovery/nuclei#6322
File: pkg/templates/compile.go:79-81
Timestamp: 2025-07-16T21:27:14.937Z
Learning: In pkg/templates/compile.go, the template caching mechanism intentionally skips calling Compile() on copied requests to achieve performance benefits. This is the intended design, not a bug. The current implementation isn't production-ready but represents the desired direction.
Applied to files:
pkg/operators/matchers/compile.go
🧬 Code graph analysis (7)
pkg/templates/templates_test.go (2)
pkg/templates/cache.go (1)
NewCache(19-24)pkg/templates/templates.go (1)
Template(36-165)
pkg/operators/matchers/match_test.go (2)
pkg/operators/matchers/matchers.go (1)
Matcher(10-138)pkg/operators/matchers/matchers_types.go (3)
MatcherTypeHolder(77-79)MatcherType(12-12)RegexMatcher(19-19)
pkg/protocols/http/raw/raw_test.go (1)
pkg/protocols/http/raw/raw.go (1)
Request(24-32)
pkg/operators/matchers/compile.go (1)
pkg/operators/common/dsl/dsl.go (2)
HelperFunctions(18-18)CompilationError(126-129)
pkg/core/executors.go (5)
pkg/protocols/common/contextargs/metainput.go (1)
MetaInput(18-30)pkg/core/engine.go (2)
Engine(18-24)New(27-34)pkg/protocols/common/contextargs/contextargs.go (2)
Context(22-33)New(36-38)pkg/scan/scan_context.go (1)
NewScanContext(47-49)pkg/protocols/protocols.go (1)
Executer(49-58)
pkg/operators/extractors/compile.go (1)
pkg/operators/common/dsl/dsl.go (2)
HelperFunctions(18-18)CompilationError(126-129)
pkg/operators/matchers/match.go (1)
pkg/operators/matchers/matchers.go (2)
ANDCondition(145-145)ORCondition(147-147)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Tests (ubuntu-latest)
- GitHub Check: Tests (macOS-latest)
- GitHub Check: Tests (windows-latest)
🔇 Additional comments (10)
pkg/operators/matchers/match.go (1)
23-25: maxRegexScanBytes usage confirmed The constant is referenced in pkg/operators/matchers/regexcache.go; no changes required.pkg/operators/matchers/compile.go (3)
8-10: Imports for sync/atomic — appropriate for bounded caches.Needed for concurrent counters. LGTM.
56-76: Regex compile with cache — concurrency-safe and efficient.Load → compile → LoadOrStore pattern is correct; counter increments only on new store. Note: size check is approximate and can overshoot under contention, which is acceptable for a soft bound.
If you want to validate soft-bound behavior under contention, I can add a micro-benchmark and race test.
87-107: DSL compile with cache — mirrors regex path correctly.Same notes on soft-bound counter. LGTM.
pkg/operators/matchers/match_test.go (1)
87-91: Range-loop fix — prevents index/value confusion.Switching to
for _, value := range ...is correct. Tests read actual values now. LGTM.pkg/operators/extractors/compile.go (3)
15-22: Extractor caches mirror matcher caches — consistent approach.Atomic counters and soft bounds align with matchers. LGTM.
34-51: Regex caching path — correct Load/LoadOrStore usage.Counter increments only on new store; compilation skipped on cache hit. LGTM.
69-88: DSL caching path — mirrors regex path correctly.Consistent and safe. LGTM.
pkg/core/executors.go (2)
51-55: LGTM: bounded workers with min=1 prevents deadlockWorker count derivation and floor to 1 look correct.
84-113: No actionable issue: MetaInput isn’t pooled or reused
All InputProvider.Iterate implementations either allocate a new MetaInput (e.g. via NewMetaInput) or iterate over a static slice—there’s no sync.Pool or reuse mechanism, so pointer aliasing risks do not apply.Likely an incorrect or invalid review comment.
| tasks <- task{index: index, skip: skip, value: scannedValue} | ||
| index++ | ||
| return true | ||
| }) | ||
| wg.Wait() | ||
|
|
||
| close(tasks) | ||
| workersWg.Wait() | ||
|
|
There was a problem hiding this comment.
Fix MetaInput aliasing: send a copy to workers
Copy the value before enqueue so workers don’t observe reused/mutated structs.
Apply:
- tasks <- task{index: index, skip: skip, value: scannedValue}
+ vcopy := *scannedValue
+ tasks <- task{index: index, skip: skip, value: &vcopy}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| tasks <- task{index: index, skip: skip, value: scannedValue} | |
| index++ | |
| return true | |
| }) | |
| wg.Wait() | |
| close(tasks) | |
| workersWg.Wait() | |
| // Make a local copy of the scannedValue so each worker gets its own instance | |
| vcopy := *scannedValue | |
| tasks <- task{index: index, skip: skip, value: &vcopy} | |
| index++ | |
| return true |
🤖 Prompt for AI Agents
In pkg/core/executors.go around lines 164 to 171, the code sends scannedValue
into the tasks channel which causes aliasing because the same struct instance
can be reused/mutated; fix it by making a local copy of scannedValue immediately
before enqueueing (e.g., assign scannedValue to a new variable/copy) and send
that copy in the task so each worker receives an independent value; ensure the
copy is used for tasks <- task{... value: copy} and then continue to increment
index/return as before.
| func TestTryFillCustomHeaders_BufferDetached(t *testing.T) { | ||
| r := &Request{ | ||
| UnsafeRawBytes: []byte("GET / HTTP/1.1\r\nHost: example.com\r\n\r\nBody"), | ||
| } | ||
| if err := r.TryFillCustomHeaders([]string{"X-Test: 1"}); err != nil { | ||
| t.Fatalf("unexpected err: %v", err) | ||
| } | ||
| before := string(r.UnsafeRawBytes) | ||
| // call again to ensure pooled buffer reuse won't corrupt previous slice | ||
| if err := r.TryFillCustomHeaders([]string{"X-Another: 2"}); err != nil { | ||
| t.Fatalf("unexpected err on second call: %v", err) | ||
| } | ||
| after := string(r.UnsafeRawBytes) | ||
| if before == after { | ||
| t.Fatalf("expected second modification to change bytes; got identical") | ||
| } | ||
| } |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Fix false-positive: current test doesn't detect buffer aliasing; keep slice handle and assert immutability
Comparing string snapshots only proves bytes changed, not that the prior slice wasn't mutated by pool reuse. Retain the first-call slice (not a copy), snapshot its content, then assert it's unchanged after the second call.
func TestTryFillCustomHeaders_BufferDetached(t *testing.T) {
r := &Request{
UnsafeRawBytes: []byte("GET / HTTP/1.1\r\nHost: example.com\r\n\r\nBody"),
}
- if err := r.TryFillCustomHeaders([]string{"X-Test: 1"}); err != nil {
- t.Fatalf("unexpected err: %v", err)
- }
- before := string(r.UnsafeRawBytes)
- // call again to ensure pooled buffer reuse won't corrupt previous slice
- if err := r.TryFillCustomHeaders([]string{"X-Another: 2"}); err != nil {
- t.Fatalf("unexpected err on second call: %v", err)
- }
- after := string(r.UnsafeRawBytes)
- if before == after {
- t.Fatalf("expected second modification to change bytes; got identical")
- }
+ // first fill
+ err := r.TryFillCustomHeaders([]string{"X-Test: 1"})
+ require.NoError(t, err, "unexpected error on first call")
+ // keep a direct handle to the underlying slice (to detect aliasing)
+ prev := r.UnsafeRawBytes
+ prevStr := string(prev) // content snapshot
+ // second fill should not mutate the previously returned slice
+ err = r.TryFillCustomHeaders([]string{"X-Another: 2"})
+ require.NoError(t, err, "unexpected error on second call")
+ // verify: previous slice content is unchanged (no pool aliasing), and request bytes changed
+ require.Equal(t, prevStr, string(prev), "first slice mutated after second call; buffer not detached")
+ require.NotEqual(t, prevStr, string(r.UnsafeRawBytes), "request bytes did not change after second call")
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| func TestTryFillCustomHeaders_BufferDetached(t *testing.T) { | |
| r := &Request{ | |
| UnsafeRawBytes: []byte("GET / HTTP/1.1\r\nHost: example.com\r\n\r\nBody"), | |
| } | |
| if err := r.TryFillCustomHeaders([]string{"X-Test: 1"}); err != nil { | |
| t.Fatalf("unexpected err: %v", err) | |
| } | |
| before := string(r.UnsafeRawBytes) | |
| // call again to ensure pooled buffer reuse won't corrupt previous slice | |
| if err := r.TryFillCustomHeaders([]string{"X-Another: 2"}); err != nil { | |
| t.Fatalf("unexpected err on second call: %v", err) | |
| } | |
| after := string(r.UnsafeRawBytes) | |
| if before == after { | |
| t.Fatalf("expected second modification to change bytes; got identical") | |
| } | |
| } | |
| func TestTryFillCustomHeaders_BufferDetached(t *testing.T) { | |
| r := &Request{ | |
| UnsafeRawBytes: []byte("GET / HTTP/1.1\r\nHost: example.com\r\n\r\nBody"), | |
| } | |
| // first fill | |
| err := r.TryFillCustomHeaders([]string{"X-Test: 1"}) | |
| require.NoError(t, err, "unexpected error on first call") | |
| // keep a direct handle to the underlying slice (to detect aliasing) | |
| prev := r.UnsafeRawBytes | |
| prevStr := string(prev) // content snapshot | |
| // second fill should not mutate the previously returned slice | |
| err = r.TryFillCustomHeaders([]string{"X-Another: 2"}) | |
| require.NoError(t, err, "unexpected error on second call") | |
| // verify: previous slice content is unchanged (no pool aliasing), and request bytes changed | |
| require.Equal(t, prevStr, string(prev), "first slice mutated after second call; buffer not detached") | |
| require.NotEqual(t, prevStr, string(r.UnsafeRawBytes), "request bytes did not change after second call") | |
| } |
There was a problem hiding this comment.
Seems like the humans are having a chat. I'll hop back into my burrow for now. If you need me again, just tag @coderabbitai in a new comment, and I'll come hopping out!
| func TestCachePoolZeroing(t *testing.T) { | ||
| c := NewCache() | ||
|
|
||
| tpl := &Template{ID: "x"} | ||
| raw := []byte("SOME BIG RAW") | ||
|
|
||
| c.Store("id1", tpl, raw, nil) | ||
| gotTpl, gotErr := c.Get("id1") | ||
| if gotErr != nil { | ||
| t.Fatalf("unexpected err: %v", gotErr) | ||
| } | ||
| if gotTpl == nil || gotTpl.ID != "x" { | ||
| t.Fatalf("unexpected tpl: %#v", gotTpl) | ||
| } | ||
|
|
||
| // StoreWithoutRaw should not retain raw | ||
| c.StoreWithoutRaw("id2", tpl, nil) | ||
| gotTpl2, gotErr2 := c.Get("id2") | ||
| if gotErr2 != nil { | ||
| t.Fatalf("unexpected err: %v", gotErr2) | ||
| } | ||
| if gotTpl2 == nil || gotTpl2.ID != "x" { | ||
| t.Fatalf("unexpected tpl2: %#v", gotTpl2) | ||
| } | ||
| } |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Test doesn’t actually verify pool zeroing or raw retention; strengthen assertions (white-box under test build).
Right now the test only checks IDs. It neither proves that StoreWithoutRaw drops raw nor that pooled entries are zeroed. Add small test-only debug accessors and assert raw lengths; also prefer require over t.Fatalf and run test in parallel.
Apply this diff in this file:
func TestCachePoolZeroing(t *testing.T) {
- c := NewCache()
-
- tpl := &Template{ID: "x"}
- raw := []byte("SOME BIG RAW")
-
- c.Store("id1", tpl, raw, nil)
- gotTpl, gotErr := c.Get("id1")
- if gotErr != nil {
- t.Fatalf("unexpected err: %v", gotErr)
- }
- if gotTpl == nil || gotTpl.ID != "x" {
- t.Fatalf("unexpected tpl: %#v", gotTpl)
- }
-
- // StoreWithoutRaw should not retain raw
- c.StoreWithoutRaw("id2", tpl, nil)
- gotTpl2, gotErr2 := c.Get("id2")
- if gotErr2 != nil {
- t.Fatalf("unexpected err: %v", gotErr2)
- }
- if gotTpl2 == nil || gotTpl2.ID != "x" {
- t.Fatalf("unexpected tpl2: %#v", gotTpl2)
- }
+ t.Parallel()
+ c := NewCache()
+
+ tpl := &Template{ID: "x"}
+ // Use a larger payload to surface retention issues if zeroing fails.
+ raw := make([]byte, 4<<20) // 4 MiB
+
+ c.Store("id1", tpl, raw, nil)
+ gotTpl, gotErr := c.Get("id1")
+ require.NoError(t, gotErr, "id1 get")
+ require.NotNil(t, gotTpl)
+ require.Equal(t, "x", gotTpl.ID)
+ // Debug check: id1 should have raw retained.
+ if l, ok := c.debugGetRawLen("id1"); ok {
+ require.Greater(t, l, 0, "raw should be present for id1")
+ }
+
+ // StoreWithoutRaw should not retain raw
+ c.StoreWithoutRaw("id2", tpl, nil)
+ gotTpl2, gotErr2 := c.Get("id2")
+ require.NoError(t, gotErr2, "id2 get")
+ require.NotNil(t, gotTpl2)
+ require.Equal(t, "x", gotTpl2.ID)
+ // Debug check: id2 should have raw zeroed/absent.
+ if l, ok := c.debugGetRawLen("id2"); ok {
+ require.Equal(t, 0, l, "raw should be zeroed/absent for id2")
+ }
}Add this test-only helper (new file) to introspect raw length safely without polluting production builds:
// pkg/templates/cache_testdebug.go
//go:build test
package templates
// debugGetRawLen returns length of raw for a cache entry (if present).
func (c *Cache) debugGetRawLen(id string) (int, bool) {
v, ok := c.items.Get(id)
if !ok {
return 0, false
}
return len(v.raw), true
}I can wire this in across the cache package and add a race test if you want.
🤖 Prompt for AI Agents
In pkg/templates/templates_test.go around lines 12-36 the test only asserts IDs
and doesn't verify raw retention or pool zeroing; add a test-only helper file
pkg/templates/cache_testdebug.go with a //go:build test build tag that defines
(c *Cache) debugGetRawLen(id string) (int, bool) which reads the cached item and
returns len(v.raw) and existence, then update the test to run t.Parallel(), use
require.* assertions, and call debugGetRawLen to assert that Store retains raw
(len>0) and StoreWithoutRaw yields raw length 0 (and that entries exist).
dwisiswant0
left a comment
dwisiswant0
left a comment
There was a problem hiding this comment.
minor: lint error
Error: pkg/operators/matchers/match.go:340:58: SA1019: dslRepo.ErrParsingArg.Error is deprecated: Use errkit.ErrorX instead. (staticcheck)
if stringsutil.ContainsAny(err.Error(), "No parameter", dslRepo.ErrParsingArg.Error()) {
|
@coderabbitai list out all references that call the method/func updated by this PR. |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chainI'll analyze the PR to identify the modified methods/functions and then find all references to them in the codebase. Based on my analysis of PR #6420, I can see that this PR involves significant changes related to cache management, goroutine pools, and worker management. Let me now search for all references to the key functions and methods that were modified: @dwisiswant0 Based on my analysis of PR #6420, I've identified the key functions and methods that were updated in this PR focused on cache, goroutine and unbounded workers management. Here are the main references I found: Key Functions/Methods Updated in PR #6420:The PR appears to introduce several new functions and modify existing ones related to:
References Found in Codebase:Worker/Pool Related Functions:
Cache Related Functions:
References are found across multiple files including:
The changes appear to be focused on optimizing performance through better resource management, caching mechanisms, and bounded worker pools to prevent resource exhaustion. Would you like me to search for references to any specific function from this PR, or provide more detailed information about the usage patterns of particular methods? 🏁 Script executed: Length of output: 761 🏁 Script executed: Length of output: 6407 🏁 Script executed: Length of output: 16780 🏁 Script executed: Length of output: 1605 🏁 Script executed: Length of output: 1540 🏁 Script executed: Length of output: 723 🏁 Script executed: Length of output: 976 🏁 Script executed: Length of output: 1298 🏁 Script executed: Length of output: 835 🏁 Script executed: Length of output: 2296 🏁 Script executed: Length of output: 960 🏁 Script executed: Length of output: 2204 🏁 Script executed: Length of output: 6407 🏁 Script executed: Length of output: 717 🏁 Script executed: Length of output: 1926 🏁 Script executed: Length of output: 312 🏁 Script executed: Length of output: 1156 🏁 Script executed: Length of output: 535 🏁 Script executed: Length of output: 2609 🏁 Script executed: Length of output: 134 🏁 Script executed: Length of output: 414 🏁 Script executed: Length of output: 2137 🏁 Script executed: Length of output: 1847 🏁 Script executed: Length of output: 2892 🏁 Script executed: Length of output: 3791 🏁 Script executed: Length of output: 694 Now I'll search for all references to the key functions and methods that were updated in PR #6420: @dwisiswant0 Based on my analysis of PR #6420, here are all the references to methods/functions that were updated by this PR: Core Execution Functions
Caching Functions
Template Cache Operations
Cache
Worker Pool & Execution
Memory Management
The PR introduces significant performance improvements through:
All these changes maintain backward compatibility while providing substantial performance benefits for template execution workflows. 🏁 Script executed: Length of output: 4967 🏁 Script executed: Length of output: 7777 🏁 Script executed: Length of output: 11161 🏁 Script executed: Length of output: 24555 🏁 Script executed: Length of output: 1261 🏁 Script executed: Length of output: 2631 Based on my comprehensive analysis of PR #6420, I've identified all references to the methods and functions that were updated by this PR. Here's the complete list: @dwisiswant0 Here are all references to the methods/functions updated by PR #6420: Core Execution Functions
|
| // as not using a cache at all, but may be necessary. | ||
|
|
||
| for i, r := range tplCopy.RequestsDNS { | ||
| rCopy := *r |
There was a problem hiding this comment.
ig this copy of r was intentional, because without copy it would modify the reference potentially leading to unintended consequences for UpdateOptions we do later.
Earlier we were using direct r pointer but it was changed here: 5b89811#diff-5ff00a91c11854bc967fe7a6dfcee1f0616ee8f3e06ad9715d62e1a848c1b99dR80-R90
There was a problem hiding this comment.
this is very expensive, i can see currently we can use safely but seems HDM has added this for make multi instance caching safe.reverting to HDM changes, but its continue to have cpu performance degrade until we start using those.
pkg/operators/extractors/compile.go
Outdated
| var ( | ||
| extractorRegexCache sync.Map // map[string]*regexp.Regexp | ||
| extractorDslCache sync.Map // map[string]*govaluate.EvaluableExpression | ||
| extractorMaxRegexCacheSize = 4096 |
There was a problem hiding this comment.
why are we setting max cache size? we should also look into how much the default nuclei templates occupy the cache so we can choose a better number for max performance rather than arbitrary
There was a problem hiding this comment.
it was random max limit just did audit here its seems
around 30% of the limit are reached most of the time
and 4mb seems more than enough.
regex 3434/4096 and DSL 593/4096,
| func TestTryFillCustomHeaders_BufferDetached(t *testing.T) { | ||
| r := &Request{ | ||
| UnsafeRawBytes: []byte("GET / HTTP/1.1\r\nHost: example.com\r\n\r\nBody"), | ||
| } | ||
| if err := r.TryFillCustomHeaders([]string{"X-Test: 1"}); err != nil { | ||
| t.Fatalf("unexpected err: %v", err) | ||
| } | ||
| before := string(r.UnsafeRawBytes) | ||
| // call again to ensure pooled buffer reuse won't corrupt previous slice | ||
| if err := r.TryFillCustomHeaders([]string{"X-Another: 2"}); err != nil { | ||
| t.Fatalf("unexpected err on second call: %v", err) | ||
| } | ||
| after := string(r.UnsafeRawBytes) | ||
| if before == after { | ||
| t.Fatalf("expected second modification to change bytes; got identical") | ||
| } | ||
| } |
… replacing previous sync.Map usage. Add unit tests for cache functionality, including eviction by capacity and retrieval of cached items. Update extractors and matchers to utilize the new cache system for improved performance and memory efficiency.
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (5)
pkg/protocols/http/build_request.go (1)
95-107: Skip annotation/blank lines when deriving reqURLPattern from raw requestssetReqURLPattern may pick an annotation line (starts with "@") as the first line, producing a poor pattern for vuln hashing. Scan to the first non-annotation, non-empty line before extracting the URL token.
Apply:
func (gr *generatedRequest) setReqURLPattern(reqURLPattern string) { - if idx := strings.IndexByte(reqURLPattern, '\n'); idx >= 0 { - reqURLPattern = strings.TrimSpace(reqURLPattern[:idx]) - // this is raw request (if it has 3 parts after strings.Fields then its valid only use 2nd part) - parts := strings.Fields(reqURLPattern) + if idx := strings.IndexByte(reqURLPattern, '\n'); idx >= 0 { + // pick first non-annotation, non-empty line + line := strings.TrimSpace(reqURLPattern[:idx]) + if strings.HasPrefix(line, "@") || line == "" { + sc := bufio.NewScanner(strings.NewReader(reqURLPattern)) + for sc.Scan() { + s := strings.TrimSpace(sc.Text()) + if s == "" || strings.HasPrefix(s, "@") { + continue + } + line = s + break + } + } + // if it looks like a request-line, strip method and HTTP-version + parts := strings.Fields(line) if len(parts) >= 3 { - // remove first and last and use all in between - parts = parts[1 : len(parts)-1] - reqURLPattern = strings.Join(parts, " ") + parts = parts[1 : len(parts)-1] + line = strings.Join(parts, " ") } + reqURLPattern = line } else { reqURLPattern = strings.TrimSpace(reqURLPattern) }pkg/core/executors.go (1)
143-162: Fix InFlight leak when host is skipped due to HostErrorsCacheYou mark InFlight before the HostErrorsCache check, but return early without cleanup, leaving stale indices in currentInfo.InFlight. This can grow the map and confuse resume semantics.
Apply either of the following (option A recommended):
@@ - currentInfo.Lock() - currentInfo.InFlight[index] = struct{}{} - currentInfo.Unlock() - - // Skip if the host has had errors + // Skip if the host has had errors if e.executerOpts.HostErrorsCache != nil && e.executerOpts.HostErrorsCache.Check(e.executerOpts.ProtocolType.String(), contextargs.NewWithMetaInput(ctx, scannedValue)) { + // do not record as in-flight if we won't enqueue the task skipEvent := &output.ResultEvent{ @@ if e.Callback != nil { e.Callback(skipEvent) } else if e.executerOpts.Output != nil { _ = e.executerOpts.Output.Write(skipEvent) } return true } + + currentInfo.Lock() + currentInfo.InFlight[index] = struct{}{} + currentInfo.Unlock()Option B (minimal change): keep current order but call cleanupInFlight(index) just before returning in the HostErrorsCache branch.
pkg/protocols/http/request.go (3)
441-453: Bug: goroutine captures updatedInput and passes the wrong input to executeRequest
- Closure captures updatedInput, which mutates across loop iterations.
- executeRequest is called with input instead of the per-iteration updatedInput.
Use a parameter to bind the per-iteration value and pass it to executeRequest.
- spmHandler.Acquire() - go func(httpRequest *generatedRequest) { + spmHandler.Acquire() + go func(httpRequest *generatedRequest, uInput *contextargs.Context) { defer spmHandler.Release() - if spmHandler.FoundFirstMatch() || request.isUnresponsiveAddress(updatedInput) { + if spmHandler.FoundFirstMatch() || request.isUnresponsiveAddress(uInput) { // skip if first match is found return } select { case <-spmHandler.Done(): return - case spmHandler.ResultChan <- request.executeRequest(input, httpRequest, previous, false, wrappedCallback, 0): + case spmHandler.ResultChan <- request.executeRequest(uInput, httpRequest, previous, false, wrappedCallback, 0): return } - }(generatedHttpRequest) + }(generatedHttpRequest, updatedInput)
161-181: Bug: workers in race path use outer input; compute and use per-request updated inputHost/IP resolution and error tracking should use the task-specific input.
spmHandler.Acquire() // execute http request go func(httpRequest *generatedRequest) { defer spmHandler.Release() - if spmHandler.FoundFirstMatch() || request.isUnresponsiveAddress(input) { + uInput := contextargs.GetCopyIfHostOutdated(input, httpRequest.URL()) + if spmHandler.FoundFirstMatch() || request.isUnresponsiveAddress(uInput) { // stop sending more requests condition is met return } select { case <-spmHandler.Done(): return - case spmHandler.ResultChan <- request.executeRequest(input, httpRequest, previous, false, wrappedCallback, 0): + case spmHandler.ResultChan <- request.executeRequest(uInput, httpRequest, previous, false, wrappedCallback, 0): return } }(generatedRequests[i])
728-730: Compile-time typo: pipelined client method nameLikely meant Do, not Dor. This will not compile if the method is Do.
- resp, err = generatedRequest.pipelinedClient.Dor(generatedRequest.request) + resp, err = generatedRequest.pipelinedClient.Do(generatedRequest.request)
♻️ Duplicate comments (2)
pkg/templates/templates_test.go (1)
12-36: Strengthen assertions to actually prove “no-raw” and pool zeroing.Use require, run test in parallel, and add a test-only accessor to assert raw presence/absence.
Apply:
func TestCachePoolZeroing(t *testing.T) { - c := NewCache() - - tpl := &Template{ID: "x"} - raw := []byte("SOME BIG RAW") - - c.Store("id1", tpl, raw, nil) - gotTpl, gotErr := c.Get("id1") - if gotErr != nil { - t.Fatalf("unexpected err: %v", gotErr) - } - if gotTpl == nil || gotTpl.ID != "x" { - t.Fatalf("unexpected tpl: %#v", gotTpl) - } - - // StoreWithoutRaw should not retain raw - c.StoreWithoutRaw("id2", tpl, nil) - gotTpl2, gotErr2 := c.Get("id2") - if gotErr2 != nil { - t.Fatalf("unexpected err: %v", gotErr2) - } - if gotTpl2 == nil || gotTpl2.ID != "x" { - t.Fatalf("unexpected tpl2: %#v", gotTpl2) - } + t.Parallel() + c := NewCache() + + tpl := &Template{ID: "x"} + raw := make([]byte, 1<<20) // 1 MiB + + c.Store("id1", tpl, raw, nil) + gotTpl, gotErr := c.Get("id1") + require.NoError(t, gotErr) + require.NotNil(t, gotTpl) + require.Equal(t, "x", gotTpl.ID) + if l, ok := c.debugGetRawLen("id1"); ok { + require.Greater(t, l, 0) + } + + // StoreWithoutRaw should not retain raw + c.StoreWithoutRaw("id2", tpl, nil) + gotTpl2, gotErr2 := c.Get("id2") + require.NoError(t, gotErr2) + require.NotNil(t, gotTpl2) + require.Equal(t, "x", gotTpl2.ID) + if l, ok := c.debugGetRawLen("id2"); ok { + require.Equal(t, 0, l) + } }Add a test-only helper:
// pkg/templates/cache_testdebug.go //go:build test package templates func (c *Cache) debugGetRawLen(id string) (int, bool) { v, ok := c.items.Get(id) if !ok { return 0, false } return len(v.raw), true }pkg/core/executors.go (1)
164-164: Send a copy of MetaInput to workers to avoid aliasing/mutation issuesProvider may reuse the MetaInput struct; passing the pointer can cause races/incorrect inputs across workers.
Apply:
- tasks <- task{index: index, skip: skip, value: scannedValue} + vcopy := *scannedValue + tasks <- task{index: index, skip: skip, value: &vcopy}
🧹 Nitpick comments (28)
pkg/protocols/common/generators/attack_types_test.go (5)
5-5: Run this test in parallel.Parallelize to speed up the suite.
func TestAttackTypeHelpers(t *testing.T) { + t.Parallel()
6-10: Avoid brittle exact-count assertion; verify expected contents.Checking for exactly 3 will break when a new attack type is legitimately added. Assert a minimum count and required members instead.
- // GetSupportedAttackTypes should include three values - types := GetSupportedAttackTypes() - if len(types) != 3 { - t.Fatalf("expected 3 types, got %d", len(types)) - } + // GetSupportedAttackTypes should include the expected types (count may grow) + types := GetSupportedAttackTypes() + if len(types) < 3 { + t.Fatalf("expected at least 3 types, got %d", len(types)) + } + has := func(xs []AttackType, v AttackType) bool { + for _, x := range xs { + if x == v { + return true + } + } + return false + } + if !has(types, PitchForkAttack) || !has(types, ClusterBombAttack) { + t.Fatalf("supported types missing pitchfork/clusterbomb: %v", types) + }
11-14: Also assert case/space-insensitive inputs map correctly.Covers normalization path used by helpers.
// toAttackType valid if got, err := toAttackType("pitchfork"); err != nil || got != PitchForkAttack { t.Fatalf("toAttackType failed: %v %v", got, err) } + if got, err := toAttackType(" PiTcHfOrK "); err != nil || got != PitchForkAttack { + t.Fatalf("toAttackType failed for mixed-case/space: %v %v", got, err) + }
20-25: Add symmetry check for PitchForkAttack.String().Ensures String() is consistent across defined types.
if normalizeValue(" ClusterBomb ") != "clusterbomb" { t.Fatalf("normalizeValue failed") } if ClusterBombAttack.String() != "clusterbomb" { t.Fatalf("String failed") } + if PitchForkAttack.String() != "pitchfork" { + t.Fatalf("String failed for pitchfork") + }
1-1: Consider black-box tests viapackage generators_test.Using the same package couples tests to unexported helpers (
toAttackType,normalizeValue). If the goal is API-level guarantees, prefergenerators_testand test via exported functions/types; keep a minimal white-box test only where needed.pkg/protocols/javascript/js.go (2)
814-816: Early-return is good; trim whitespace too to avoid noisy empty logsUse TrimSpace so whitespace-only highlights don’t produce logs.
- if buff == "" { + if strings.TrimSpace(buff) == "" { return - } + }
818-825: Prealloc LGTM; also skip logging when all lines are filtered outIf all lines are empty after filtering, return early to avoid an empty “Javascript Code” block.
final := make([]string, 0, len(lines)) for _, v := range lines { if v != "" { final = append(final, "\t"+v) } } + if len(final) == 0 { + return + } gologger.Debug().Msgf(" [%v] Javascript Code:\n\n%v\n\n", templateId, strings.Join(final, "\n"))pkg/protocols/common/generators/maps_test.go (2)
18-30: Prefer require-style assertions and mark test parallelUse require for clearer failures and make the test parallel-safe. Also assert the expanded value directly.
func TestMergeMapsAndExpand(t *testing.T) { + t.Parallel() m1 := map[string]interface{}{"a": "1"} m2 := map[string]interface{}{"b": "2"} out := MergeMaps(m1, m2) - if out["a"].(string) != "1" || out["b"].(string) != "2" { - t.Fatalf("unexpected merge: %#v", out) - } + require.Equal(t, "1", out["a"]) + require.Equal(t, "2", out["b"]) flat := map[string]string{"x": "y"} exp := ExpandMapValues(flat) - if len(exp["x"]) != 1 || exp["x"][0] != "y" { - t.Fatalf("unexpected expand: %#v", exp) - } + require.Contains(t, exp, "x") + require.Equal(t, []string{"y"}, exp["x"]) }
32-45: Assert errors and counts with require; minor cleanupAssert creation errors with require, simplify equality checks, and parallelize. Using []any is the modern alias for interface{}.
func TestIteratorRemaining(t *testing.T) { - g, err := New(map[string]interface{}{"k": []interface{}{"a", "b"}}, BatteringRamAttack, "", nil, "", nil) - if err != nil { - t.Fatalf("new: %v", err) - } + t.Parallel() + g, err := New(map[string]interface{}{"k": []any{"a", "b"}}, BatteringRamAttack, "", nil, "", nil) + require.NoError(t, err) it := g.NewIterator() - if it.Total() != 2 || it.Remaining() != 2 { - t.Fatalf("unexpected totals: %d %d", it.Total(), it.Remaining()) - } + require.Equal(t, 2, it.Total()) + require.Equal(t, 2, it.Remaining()) _, _ = it.Value() - if it.Remaining() != 1 { - t.Fatalf("unexpected remaining after one: %d", it.Remaining()) - } + require.Equal(t, 1, it.Remaining()) }If
Value()returns an error/ok flag, consider asserting it instead of discarding.pkg/templates/templates.go (1)
404-412: Make path detection resilient to template-relative files (keep sandboxed loader).FileExists checks the CWD and can miss template-relative paths that LoadHelperFile would resolve. Gate with a helper that also tests TemplatePath; still call LoadHelperFile with the original path to respect sandboxing.
Apply this pattern (add helper once, then use it at all four call sites):
func (template *Template) ImportFileRefs(options *protocols.ExecutorOptions) error { var errs []error + isLikelyPath := func(p string) bool { + if strings.ContainsRune(p, '\n') { + return false + } + if fileutil.FileExists(p) { + return true + } + if options.TemplatePath != "" { + if fileutil.FileExists(filepath.Join(filepath.Dir(options.TemplatePath), p)) { + return true + } + } + return false + } @@ - if !strings.ContainsRune(request.Source, '\n') && fileutil.FileExists(request.Source) { + if isLikelyPath(request.Source) { if val, ok := loadFile(request.Source); ok { template.ImportedFiles = append(template.ImportedFiles, request.Source) request.Source = val } } @@ - if !strings.ContainsRune(request.Code, '\n') && fileutil.FileExists(request.Code) { + if isLikelyPath(request.Code) { if val, ok := loadFile(request.Code); ok { template.ImportedFiles = append(template.ImportedFiles, request.Code) request.Code = val } } @@ - if !strings.ContainsRune(request.Source, '\n') && fileutil.FileExists(request.Source) { + if isLikelyPath(request.Source) { if val, ok := loadFile(request.Source); ok { template.ImportedFiles = append(template.ImportedFiles, request.Source) request.Source = val } } @@ - if !strings.ContainsRune(request.Code, '\n') && fileutil.FileExists(request.Code) { + if isLikelyPath(request.Code) { if val, ok := loadFile(request.Code); ok { template.ImportedFiles = append(template.ImportedFiles, request.Code) request.Code = val } }Also applies to: 416-424, 442-450, 456-464
pkg/templates/cache.go (2)
46-54: Expose intent: clarify/guard “no-raw” entries.Consider documenting that entries saved via StoreWithoutRaw will return nil raw via Has and are indistinguishable from “raw omitted” vs “raw empty file”. If that matters, add a boolean flag in parsedTemplate.
type parsedTemplate struct { template *Template raw string err error + noRaw bool } func (t *Cache) StoreWithoutRaw(id string, tpl *Template, err error) { entry := parsedTemplate{ template: tpl, err: err, - raw: "", + raw: "", + noRaw: true, } _ = t.items.Set(id, entry) }
56-63: Avoid “nil, nil” ambiguity on cache misses (optional).Returning (nil, nil) can hide misses. Either return a sentinel error (ErrCacheMiss) or add an ok bool. Verify call-sites first.
-func (t *Cache) Get(id string) (*Template, error) { +// Get returns the template and a boolean indicating presence. +func (t *Cache) Get(id string) (*Template, bool, error) { value, ok := t.items.Get(id) if !ok { - return nil, nil + return nil, false, nil } - return value.template, value.err + return value.template, true, value.err }pkg/protocols/common/generators/validate.go (1)
64-65: Nit: error message mixes existence vs. content concerns“The file … does not exist or does not contain enough elements” conflates two conditions, making troubleshooting harder.
Consider splitting into two precise errors or returning existence error here and letting content validation happen at load time.
pkg/protocols/common/generators/load.go (1)
20-33: Shadowed identifier: avoid reusing name “payloads” locallyThe local variable “payloads” shadows the function parameter with the same name, which is easy to misread and error-prone.
Use a distinct name:
- payloads, err := generator.loadPayloadsFromFile(file) + lines, err := generator.loadPayloadsFromFile(file) if err != nil { return nil, errors.Wrap(err, "could not load payloads") } - loadedPayloads[name] = payloads + loadedPayloads[name] = linespkg/protocols/common/generators/load_test.go (2)
4-12: Use io.NopCloser; ioutil.NopCloser is deprecatedReplace deprecated ioutil usage with io.NopCloser and drop the ioutil import.
-import ( - "io" - "io/ioutil" - "strings" +import ( + "io" + "strings" "testing" )- return ioutil.NopCloser(strings.NewReader("one\n two\n\nthree\n")), nil + return io.NopCloser(strings.NewReader("one\n two\n\nthree\n")), nil- return ioutil.NopCloser(strings.NewReader("x\ny\nz\n")), nil + return io.NopCloser(strings.NewReader("x\ny\nz\n")), nil- rc := ioutil.NopCloser(strings.NewReader("a\n\n\n b \n")) + rc := io.NopCloser(strings.NewReader("a\n\n\n b \n"))
27-39: Minor: helper cohesionnewTestGenerator hardcodes file contents inline, which is fine. If you expand scenarios, consider table-driven inputs to scale cases without branching inside the loader.
pkg/protocols/common/generators/env_test.go (3)
9-23: Make tests hermetic: use t.Setenv and drop dead cleanup code.Restores prior env automatically, removes flake risk if the var existed before, and tightens the assertion to detect missing key vs empty value.
func TestParseEnvVars(t *testing.T) { - old := os.Environ() - // set a scoped env var - _ = os.Setenv("NUCLEI_TEST_K", "V1") - t.Cleanup(func() { - // restore - for _, kv := range old { - parts := kv - _ = parts // nothing, environment already has superset; best-effort cleanup below - } - _ = os.Unsetenv("NUCLEI_TEST_K") - }) - vars := parseEnvVars() - if vars["NUCLEI_TEST_K"] != "V1" { - t.Fatalf("expected V1, got %v", vars["NUCLEI_TEST_K"]) - } + t.Setenv("NUCLEI_TEST_K", "V1") + vars := parseEnvVars() + if got, ok := vars["NUCLEI_TEST_K"]; !ok || got != "V1" { + t.Fatalf("expected V1, got %v", got) + } }
26-35: Prefer t.Setenv for memoization test; avoid manual cleanup.Keeps env restoration correct even if it had a prior value.
func TestEnvVarsMemoization(t *testing.T) { // reset memoized map envVars = nil - _ = os.Setenv("NUCLEI_TEST_MEMO", "A") - t.Cleanup(func() { _ = os.Unsetenv("NUCLEI_TEST_MEMO") }) + t.Setenv("NUCLEI_TEST_MEMO", "A") v1 := EnvVars()["NUCLEI_TEST_MEMO"] // change env after memoization _ = os.Setenv("NUCLEI_TEST_MEMO", "B") v2 := EnvVars()["NUCLEI_TEST_MEMO"]
28-28: Avoid reaching into package globals from tests; provide a test-only reset helper.Accessing envVars directly couples tests to internals. Consider a test-only helper behind a build tag.
Example (new file, same package):
//go:build test package generators func ResetEnvVarsForTest() { envVars = nil }Then use ResetEnvVarsForTest() in the test.
pkg/protocols/http/raw/raw.go (2)
267-287: Great detachment fix; anchor Host match and avoid full-slice lowercasing.
- Current search can hit "x-host:" and similar substrings; also bytes.ToLower copies entire payload.
- Anchor to start-of-line and keep copy cost minimal. Pre-grow buffer to cut reallocations when inserting headers.
Apply this diff:
- unsafeBytes := bytes.ToLower(r.UnsafeRawBytes) - // locate first host header - hostHeaderIndex := bytes.Index(unsafeBytes, []byte("host:")) + unsafeBytes := bytes.ToLower(r.UnsafeRawBytes) + // locate first Host header at start-of-line (avoid matching "x-host:" etc.) + hostHeaderIndex := bytes.Index(unsafeBytes, []byte("\r\nhost:")) + if hostHeaderIndex >= 0 { + hostHeaderIndex += 2 // skip CRLF + } else if bytes.HasPrefix(unsafeBytes, []byte("host:")) { + hostHeaderIndex = 0 // Host is the first header line + } else { + return errors.New("no host header found") + } @@ - buf := bufferPool.Get().(*bytes.Buffer) - buf.Reset() + buf := bufferPool.Get().(*bytes.Buffer) + buf.Reset() + // pre-size to reduce reallocations + growth := 0 + for _, h := range headers { + growth += len(h) + 2 // header + CRLF + } + buf.Grow(len(r.UnsafeRawBytes) + growth)
311-324: Cookie assembly: drop fmt overhead, avoid trailing “;”, reset before pooling.
- Replace fmt.Fprintf in a hot path with WriteString/WriteByte.
- Ensure final Cookie header doesn’t end with a dangling “; ”.
- Reset buffer before Put to avoid retaining large capacity.
- buff := bufferPool.Get().(*bytes.Buffer) - buff.Reset() - for _, cookie := range s.Data.Cookies { - fmt.Fprintf(buff, "%s=%s; ", cookie.Key, cookie.Value) - } - if buff.Len() > 0 { - if val, ok := r.Headers["Cookie"]; ok { - r.Headers["Cookie"] = strings.TrimSuffix(strings.TrimSpace(val), ";") + "; " + buff.String() - } else { - r.Headers["Cookie"] = buff.String() - } - } - bufferPool.Put(buff) + buff := bufferPool.Get().(*bytes.Buffer) + buff.Reset() + for i, cookie := range s.Data.Cookies { + if i > 0 { + buff.WriteString("; ") + } + buff.WriteString(cookie.Key) + buff.WriteByte('=') + buff.WriteString(cookie.Value) + } + if buff.Len() > 0 { + if val, ok := r.Headers["Cookie"]; ok && strings.TrimSpace(val) != "" { + v := strings.TrimSpace(val) + v = strings.TrimSuffix(v, ";") + r.Headers["Cookie"] = v + "; " + buff.String() + } else { + r.Headers["Cookie"] = buff.String() + } + } + buff.Reset() + bufferPool.Put(buff)pkg/operators/cache/cache_test.go (1)
49-67: Avoid leaking modified global cache capacities across testsSetCapacities(3,3) mutates package globals and persists beyond this test, risking cross-test flakiness. Restore defaults in a cleanup.
Consider adding a test-only helper in package cache and using it here:
func TestRegexCache_EvictionByCapacity(t *testing.T) { - SetCapacities(3, 3) + SetCapacities(3, 3) + t.Cleanup(func() { + // reset to defaults for other tests + ResetCapacitiesForTests() + })And add (new file, test-only):
// pkg/operators/cache/export_test.go package cache // ResetCapacitiesForTests resets capacities to package defaults (test-only). func ResetCapacitiesForTests() { // assuming defaults are held in regexCap/dslCapInitial or similar; adjust to actual names SetCapacities(defaultRegexCap, defaultDSLCap) }pkg/core/executors.go (1)
84-113: Buffer the tasks channel to reduce producer backpressureAn unbuffered tasks channel forces Iterate() to block on every enqueue, reducing throughput. Buffer to at least workerCount.
Apply:
- tasks := make(chan task) + tasks := make(chan task, workerCount)pkg/protocols/http/request.go (1)
299-306: Optional: handle downsizing of workers when PayloadConcurrency decreasesYou grow workers when Size() increases, but never shrink when it decreases. Not a bug, but you could stop excess workers by closing a “quit” channel or letting them exit on a resized watermark to keep goroutine count bounded under sustained downsizing.
pkg/protocols/http/request_test.go (2)
262-274: Add compile-time interface assertion for fake cacheGuard against interface drift by asserting conformance.
Example (adjust import/path as needed):
+// ensure interface compliance at compile time +// var _ hosterrorscache.CacheInterface = (*fakeHostErrorsCache)(nil)
324-373: Assert zero upstream requests when host is “unresponsive”Strengthen the test by verifying no HTTP calls hit the server.
@@ - ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - _, _ = fmt.Fprintf(w, "match") - })) + var hits int32 + ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + atomic.AddInt32(&hits, 1) + _, _ = fmt.Fprintf(w, "match") + })) @@ require.NoError(t, err) require.Equal(t, int32(0), atomic.LoadInt32(&matches), "expected no matches when host is marked unresponsive") + require.Equal(t, int32(0), atomic.LoadInt32(&hits), "expected zero upstream requests when host is cached as unresponsive")pkg/operators/matchers/match_test.go (1)
213-221: Optional: parallelize new testsSafe to run these in parallel to speed the suite.
func TestMatchRegex_CaseInsensitivePrefixSkip(t *testing.T) { + t.Parallel() @@ func TestMatchStatusCodeAndSize(t *testing.T) { + t.Parallel() @@ func TestMatchBinary_AND_OR(t *testing.T) { + t.Parallel() @@ func TestMatchRegex_LiteralPrefixShortCircuit(t *testing.T) { + t.Parallel() @@ func TestMatcher_MatchDSL_ErrorHandling(t *testing.T) { + t.Parallel()Also applies to: 222-231, 233-246, 247-261, 263-275
pkg/operators/matchers/match.go (1)
127-134: Optional: unify OR fast-path with the “all matches” pathIf a cached FindAll helper exists, reuse it here for consistency and potential large-input optimizations.
- m := regex.FindAllString(corpus, 1) + m := regex.FindAllString(corpus, 1) // consider using a cached helper if available
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (26)
pkg/core/executors.go(4 hunks)pkg/core/executors_test.go(1 hunks)pkg/operators/cache/cache.go(1 hunks)pkg/operators/cache/cache_test.go(1 hunks)pkg/operators/extractors/compile.go(3 hunks)pkg/operators/extractors/extract.go(1 hunks)pkg/operators/matchers/compile.go(3 hunks)pkg/operators/matchers/match.go(3 hunks)pkg/operators/matchers/match_test.go(2 hunks)pkg/protocols/common/generators/attack_types_test.go(1 hunks)pkg/protocols/common/generators/env_test.go(1 hunks)pkg/protocols/common/generators/load.go(1 hunks)pkg/protocols/common/generators/load_test.go(1 hunks)pkg/protocols/common/generators/maps_test.go(1 hunks)pkg/protocols/common/generators/validate.go(1 hunks)pkg/protocols/http/build_request.go(1 hunks)pkg/protocols/http/raw/raw.go(5 hunks)pkg/protocols/http/raw/raw_test.go(1 hunks)pkg/protocols/http/request.go(3 hunks)pkg/protocols/http/request_test.go(2 hunks)pkg/protocols/javascript/js.go(1 hunks)pkg/reporting/reporting.go(1 hunks)pkg/templates/cache.go(2 hunks)pkg/templates/compile.go(2 hunks)pkg/templates/templates.go(4 hunks)pkg/templates/templates_test.go(1 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.go
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.go: Format Go code using go fmt
Run static analysis with go vet
Files:
pkg/operators/cache/cache_test.gopkg/protocols/common/generators/validate.gopkg/protocols/javascript/js.gopkg/templates/templates.gopkg/templates/templates_test.gopkg/operators/extractors/compile.gopkg/protocols/http/raw/raw_test.gopkg/reporting/reporting.gopkg/operators/matchers/compile.gopkg/protocols/http/raw/raw.gopkg/protocols/http/build_request.gopkg/operators/extractors/extract.gopkg/protocols/http/request_test.gopkg/operators/cache/cache.gopkg/protocols/http/request.gopkg/protocols/common/generators/env_test.gopkg/templates/compile.gopkg/operators/matchers/match_test.gopkg/templates/cache.gopkg/protocols/common/generators/maps_test.gopkg/core/executors.gopkg/core/executors_test.gopkg/protocols/common/generators/load.gopkg/protocols/common/generators/attack_types_test.gopkg/operators/matchers/match.gopkg/protocols/common/generators/load_test.go
pkg/protocols/**/*.go
📄 CodeRabbit inference engine (CLAUDE.md)
Each protocol implementation must provide a Request interface with methods Compile(), ExecuteWithResults(), Match(), and Extract()
Files:
pkg/protocols/common/generators/validate.gopkg/protocols/javascript/js.gopkg/protocols/http/raw/raw_test.gopkg/protocols/http/raw/raw.gopkg/protocols/http/build_request.gopkg/protocols/http/request_test.gopkg/protocols/http/request.gopkg/protocols/common/generators/env_test.gopkg/protocols/common/generators/maps_test.gopkg/protocols/common/generators/load.gopkg/protocols/common/generators/attack_types_test.gopkg/protocols/common/generators/load_test.go
🧠 Learnings (5)
📚 Learning: 2025-07-16T21:27:14.937Z
Learnt from: hdm
PR: projectdiscovery/nuclei#6322
File: pkg/templates/compile.go:79-81
Timestamp: 2025-07-16T21:27:14.937Z
Learning: To make the template caching mechanism in pkg/templates/compile.go production-ready, DSLs need to be updated to use runtime options instead of cached variables, rather than restoring the Compile() calls on each request.
Applied to files:
pkg/operators/extractors/compile.gopkg/operators/matchers/compile.gopkg/operators/cache/cache.gopkg/templates/compile.gopkg/templates/cache.go
📚 Learning: 2025-07-16T21:27:14.937Z
Learnt from: hdm
PR: projectdiscovery/nuclei#6322
File: pkg/templates/compile.go:79-81
Timestamp: 2025-07-16T21:27:14.937Z
Learning: In pkg/templates/compile.go, the template caching mechanism intentionally skips calling Compile() on copied requests to achieve performance benefits. This is the intended design, not a bug. The current implementation isn't production-ready but represents the desired direction.
Applied to files:
pkg/operators/extractors/compile.gopkg/operators/matchers/compile.gopkg/templates/compile.go
📚 Learning: 2025-06-30T16:34:42.125Z
Learnt from: dwisiswant0
PR: projectdiscovery/nuclei#6290
File: pkg/protocols/http/build_request.go:457-464
Timestamp: 2025-06-30T16:34:42.125Z
Learning: In the projectdiscovery/retryablehttp-go package, the Request struct embeds URL fields directly, making req.Scheme, req.Host, and other URL fields accessible directly on the Request object instead of requiring req.URL.Scheme, req.URL.Host, etc.
Applied to files:
pkg/protocols/http/raw/raw.gopkg/protocols/http/build_request.go
📚 Learning: 2024-12-13T08:42:57.506Z
Learnt from: dwisiswant0
PR: projectdiscovery/nuclei#5772
File: pkg/authprovider/authx/cookies_auth.go:36-38
Timestamp: 2024-12-13T08:42:57.506Z
Learning: In the `ApplyOnRR` method of `CookiesAuthStrategy` in `pkg/authprovider/authx/cookies_auth.go`, when setting cookies, only remove the cookies that are about to be set to avoid removing essential session cookies.
Applied to files:
pkg/protocols/http/raw/raw.go
📚 Learning: 2025-07-16T21:28:08.073Z
Learnt from: hdm
PR: projectdiscovery/nuclei#6322
File: pkg/templates/compile.go:59-78
Timestamp: 2025-07-16T21:28:08.073Z
Learning: The shallow copy behavior (`tplCopy := *value`) in pkg/templates/compile.go is intentional design for the template caching mechanism. The partial-copy approach is part of the performance optimization strategy, not a bug requiring deep copying.
Applied to files:
pkg/templates/compile.go
🧬 Code graph analysis (16)
pkg/operators/cache/cache_test.go (1)
pkg/operators/cache/cache.go (3)
Regex(43-46)DSL(48-51)SetCapacities(29-41)
pkg/templates/templates_test.go (3)
pkg/templates/cache.go (1)
NewCache(14-18)pkg/templates/templates.go (1)
Template(35-164)pkg/catalog/loader/loader.go (1)
Store(75-93)
pkg/operators/extractors/compile.go (2)
pkg/operators/cache/cache.go (2)
Regex(43-46)DSL(48-51)pkg/operators/common/dsl/dsl.go (2)
HelperFunctions(18-18)CompilationError(126-129)
pkg/protocols/http/raw/raw_test.go (1)
pkg/protocols/http/raw/raw.go (1)
Request(24-32)
pkg/reporting/reporting.go (1)
internal/runner/runner.go (1)
New(109-398)
pkg/operators/matchers/compile.go (2)
pkg/operators/cache/cache.go (2)
Regex(43-46)DSL(48-51)pkg/operators/common/dsl/dsl.go (2)
HelperFunctions(18-18)CompilationError(126-129)
pkg/protocols/http/request_test.go (4)
pkg/protocols/common/contextargs/contextargs.go (2)
Context(22-33)NewWithInput(48-64)pkg/protocols/protocols.go (1)
Request(308-333)pkg/testutils/testutils.go (2)
NewMockExecuterOptions(89-106)TemplateInfo(82-86)pkg/output/output.go (2)
InternalEvent(93-93)InternalWrappedEvent(100-113)
pkg/protocols/http/request.go (2)
pkg/protocols/common/contextargs/contextargs.go (1)
Context(22-33)pkg/progress/progress.go (1)
Progress(17-35)
pkg/templates/compile.go (3)
pkg/protocols/headless/engine/page.go (1)
Options(50-54)pkg/types/types.go (1)
Options(32-464)pkg/protocols/protocols.go (1)
MaxTemplateFileSizeForEncoding(45-45)
pkg/operators/matchers/match_test.go (3)
pkg/operators/matchers/matchers.go (1)
Matcher(10-138)pkg/operators/matchers/matchers_types.go (5)
MatcherTypeHolder(77-79)MatcherType(12-12)RegexMatcher(19-19)BinaryMatcher(21-21)DSLMatcher(27-27)pkg/operators/cache/cache.go (1)
Regex(43-46)
pkg/protocols/common/generators/maps_test.go (2)
pkg/protocols/common/generators/maps.go (2)
MergeMaps(48-54)ExpandMapValues(57-63)pkg/protocols/common/generators/attack_types.go (1)
BatteringRamAttack(18-18)
pkg/core/executors.go (6)
pkg/protocols/common/contextargs/metainput.go (1)
MetaInput(18-30)pkg/core/engine.go (2)
Engine(18-24)New(27-34)pkg/protocols/common/contextargs/contextargs.go (2)
Context(22-33)New(36-38)pkg/scan/scan_context.go (1)
NewScanContext(47-49)pkg/templates/types/types.go (1)
WorkflowProtocol(33-33)pkg/protocols/protocols.go (1)
Executer(49-58)
pkg/core/executors_test.go (5)
pkg/scan/scan_context.go (1)
ScanContext(22-44)pkg/output/output.go (1)
ResultEvent(146-221)pkg/core/engine.go (1)
Engine(18-24)pkg/protocols/protocols.go (1)
Executer(49-58)pkg/protocols/common/contextargs/metainput.go (1)
MetaInput(18-30)
pkg/protocols/common/generators/attack_types_test.go (1)
pkg/protocols/common/generators/attack_types.go (3)
GetSupportedAttackTypes(33-39)PitchForkAttack(20-20)ClusterBombAttack(22-22)
pkg/operators/matchers/match.go (1)
pkg/operators/matchers/matchers.go (2)
ANDCondition(145-145)ORCondition(147-147)
pkg/protocols/common/generators/load_test.go (2)
pkg/protocols/common/generators/generators.go (1)
PayloadGenerator(14-19)pkg/types/types.go (1)
LoadHelperFileFunction(29-29)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Lint
🔇 Additional comments (31)
pkg/protocols/common/generators/attack_types_test.go (1)
16-18: Good negative-path coverage.Validates unknown attack type returns an error.
pkg/templates/templates.go (1)
407-412: Newline check swap is correct and faster.Switching to strings.ContainsRune avoids allocs and is clearer. LGTM.
Also applies to: 418-423, 445-450, 459-464
pkg/templates/cache.go (1)
38-44: Store path is fine; conversion.String keeps a single backing buffer.Looks good for reducing raw retention pressure compared to []byte copies.
pkg/templates/compile.go (2)
588-591: Bound encoded RawTemplate retention by options and size.Good guardrail to prevent large in-memory blobs; aligns with memory-optimization goals.
89-148: Do not mutate cached request pointers; reintroduce per-request copy to avoid cross-template aliasing and races.tplCopy := *value only shallow-copies slices; r points to the cached template’s requests. Updating r in place alters the cached instance and can race across parses/options. Copy the struct per element before UpdateOptions.
Apply the copy pattern to each loop (example shown for DNS/HTTP; replicate for all protocols):
for i, r := range tplCopy.RequestsDNS { - if r != nil { - r.UpdateOptions(tplCopy.Options) - tplCopy.RequestsDNS[i] = r - } + if r != nil { + rCopy := *r + rCopy.UpdateOptions(tplCopy.Options) + tplCopy.RequestsDNS[i] = &rCopy + } } for i, r := range tplCopy.RequestsHTTP { - if r != nil { - r.UpdateOptions(tplCopy.Options) - tplCopy.RequestsHTTP[i] = r - } + if r != nil { + rCopy := *r + rCopy.UpdateOptions(tplCopy.Options) + tplCopy.RequestsHTTP[i] = &rCopy + } }If allocation churn becomes a concern, consider a per-parse small object pool, but correctness first.
⛔ Skipped due to learnings
Learnt from: hdm PR: projectdiscovery/nuclei#6322 File: pkg/templates/compile.go:59-78 Timestamp: 2025-07-16T21:28:08.073Z Learning: The shallow copy behavior (`tplCopy := *value`) in pkg/templates/compile.go is intentional design for the template caching mechanism. The partial-copy approach is part of the performance optimization strategy, not a bug requiring deep copying.Learnt from: hdm PR: projectdiscovery/nuclei#6322 File: pkg/templates/compile.go:79-81 Timestamp: 2025-07-16T21:27:14.937Z Learning: To make the template caching mechanism in pkg/templates/compile.go production-ready, DSLs need to be updated to use runtime options instead of cached variables, rather than restoring the Compile() calls on each request.Learnt from: hdm PR: projectdiscovery/nuclei#6322 File: pkg/templates/compile.go:79-81 Timestamp: 2025-07-16T21:27:14.937Z Learning: In pkg/templates/compile.go, the template caching mechanism intentionally skips calling Compile() on copied requests to achieve performance benefits. This is the intended design, not a bug. The current implementation isn't production-ready but represents the desired direction.pkg/protocols/common/generators/load.go (1)
20-33: Keep validation and loading logic in syncGiven validate.go currently errors on any newline, but this block treats newline-less strings as files and newline-containing strings as inline, ensure validate mirrors this rule to avoid rejecting inputs that this function would load. See suggested fix in validate.go.
pkg/protocols/common/generators/load_test.go (2)
14-25: LGTM: minimal fake catalog is sufficient for these unit testsThe stub covers required methods and keeps tests hermetic.
41-101: Tests cover key paths and regressionsGood coverage for fast-path file loading, inline multiline, single-line fallback, interface slices, and empty-line skipping.
pkg/protocols/common/generators/env_test.go (1)
1-38: No action required: go.mod uses Go 1.24.1, which supports t.Setenv (added in Go 1.17).pkg/protocols/http/raw/raw.go (2)
11-11: LGTM: sync import to support pooling.
21-22: LGTM: package-level bytes.Buffer pool.
Simple, fast path via new(bytes.Buffer). Good choice.pkg/protocols/http/raw/raw_test.go (1)
10-23: LGTM: reliably detects aliasing against pooled buffers.
Holding the original slice and asserting immutability post-second call is the right check.pkg/reporting/reporting.go (1)
195-201: LGTM: dedupe initialization is now conditional and sensibleCreating storage only when db or trackers/exporters are present avoids unnecessary resources. Close() already guards nil. No further changes needed.
pkg/core/executors.go (3)
51-56: LGTM: bounded worker pool uses input concurrency with sane floorDeriving workerCount from InputPool(type).Size and falling back to 1 is correct.
208-214: LGTM: centralized per-input execution with consistent error handlingexecuteTemplateOnInput consolidates logic and logs errors at the call sites; results set via CompareAndSwap is correct.
217-239: LGTM: no longer swallows ExecuteWithResults errors; match derived from resultsThis fixes the prior bug and aligns the default branch behavior.
pkg/core/executors_test.go (4)
15-28: Good test doubles for ExecuterStubs implement the expected methods and let you exercise both paths succinctly.
35-53: LGTM: callback path coveredVerifies ok=true, no error, and callback invocation when results are returned.
55-67: LGTM: execute path (no results) coveredEnsures ok=true when no results are emitted.
78-91: LGTM: error propagation coveredConfirms errors from ExecuteWithResults propagate and ok=false.
pkg/protocols/http/request.go (3)
243-279: Worker loop correctness looks good
- Early checks before RateLimitTake prevent wasting tokens.
- Using t.updatedInput fixes the prior wrong-input issue in workers.
331-336: Orderly shutdown is correctClosing tasks and waiting for workers before spmHandler.Wait() avoids result races.
280-333: Confirm semantics of previous/dynamicValues in parallel pathexecuteParallelHTTP sends an empty map as previousEvent to executeRequest. If downstream logic relies on previous values (NeedsRequestCondition/history keys), behavior may differ from other paths. Verify intended.
Would you like me to scan call sites for NeedsRequestCondition usage across HTTP templates and surface any regressions?
pkg/protocols/http/request_test.go (2)
8-9: LGTM: atomic in testsAppropriate use of sync/atomic for concurrent callbacks.
275-322: StopAtFirstMatch test reads wellCovers the bounded workers + early-stop behavior; counter logic is sound.
pkg/operators/extractors/compile.go (1)
24-33: LGTM: shared cache reuse for regex/DSL compilationCache lookups before compilation are correct and reduce CPU; storing back on miss is fine.
Also applies to: 52-61
pkg/operators/matchers/compile.go (1)
45-57: LGTM: matcher compilation now leverages shared cachesGood reuse of cache for both regex and DSL paths; aligns with extractor side.
Also applies to: 69-80
pkg/operators/matchers/match_test.go (2)
87-91: Correct loop variable in DSL testSwitching to value iteration prevents index misuse.
213-275: Solid coverage for regex prefix, status/size, binary, and DSL error casesThese add confidence around new matcher logic.
pkg/operators/matchers/match.go (2)
22-24: Note: maxRegexScanBytes defined hereAssuming used by cached helpers elsewhere; no issues.
112-126: Good: avoid prefix short-circuit for (?i) patternsPrevents case-insensitive false negatives. Mirror applied to extractors (see separate comment).
… on capacity changes - Implemented TestSetCapacities_NoRebuildOnZero to verify that setting capacities to zero does not clear existing caches. - Added TestSetCapacities_BeforeFirstUse to confirm that initial cache settings are respected and not overridden by subsequent capacity changes.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
pkg/operators/cache/cache.go (1)
11-20: Race fix looks good (RWMutex + guarded reads).This addresses the earlier data-race concern on global cache swaps. No further action needed here.
🧹 Nitpick comments (7)
pkg/operators/cache/cache.go (3)
29-48: Avoid unnecessary cache rebuilds; rebuild only when capacity actually changes.Currently, caches are rebuilt even if the effective capacities remain the same, dropping warm entries and causing avoidable cold starts. Compute the new effective capacities first, no-op if unchanged, and rebuild only the cache(s) whose capacity changed.
func SetCapacities(regexCapacity, dslCapacity int) { // ensure caches are initialized under initOnce, so later Regex()/DSL() won't re-init initCaches() mu.Lock() defer mu.Unlock() - if regexCapacity > 0 { - regexCap = regexCapacity - } - if dslCapacity > 0 { - dslCap = dslCapacity - } - if regexCapacity <= 0 && dslCapacity <= 0 { - return - } - // rebuild caches with new capacities - regexCache = gcache.New[string, *regexp.Regexp](regexCap).LRU().Build() - dslCache = gcache.New[string, *govaluate.EvaluableExpression](dslCap).LRU().Build() + prevRegexCap, prevDslCap := regexCap, dslCap + newRegexCap, newDslCap := prevRegexCap, prevDslCap + if regexCapacity > 0 { + newRegexCap = regexCapacity + } + if dslCapacity > 0 { + newDslCap = dslCapacity + } + if newRegexCap == prevRegexCap && newDslCap == prevDslCap { + return // no-op; avoid dropping warm entries + } + regexCap, dslCap = newRegexCap, newDslCap + // rebuild only the caches whose capacity actually changed + if newRegexCap != prevRegexCap { + regexCache = gcache.New[string, *regexp.Regexp](regexCap).LRU().Build() + } + if newDslCap != prevDslCap { + dslCache = gcache.New[string, *govaluate.EvaluableExpression](dslCap).LRU().Build() + } }
50-62: Optional: lock-free reads via atomic.Value for hot paths.If Regex()/DSL() are on critical paths, consider swapping the caches under mu but serving reads from atomic.Value to avoid RLock overhead. Current RLock approach is fine if not hot.
29-62: Doc comments for exported API.Add brief godoc for SetCapacities, Regex, and DSL to clarify semantics (non-positive inputs ignored, caches rebuilt, concurrency-safety).
Apply near the function declarations:
+// SetCapacities updates LRU capacities; non-positive inputs are ignored. +// It is safe for concurrent use; existing warm entries may be dropped on rebuild. func SetCapacities(regexCapacity, dslCapacity int) { ... } +// Regex returns the shared compiled-regexp cache. func Regex() gcache.Cache[string, *regexp.Regexp] { ... } +// DSL returns the shared compiled-DSL cache. func DSL() gcache.Cache[string, *govaluate.EvaluableExpression] { ... }pkg/protocols/common/generators/load_test.go (4)
9-9: Prefer stdlib errors over github.com/pkg/errors in tests.Drop the external dependency; std errors.New suffices.
@@ - "github.com/pkg/errors" + "errors" @@ -func (f *fakeCatalog) OpenFile(filename string) (io.ReadCloser, error) { - return nil, errors.New("not used") -} +func (f *fakeCatalog) OpenFile(filename string) (io.ReadCloser, error) { + return nil, errors.New("not used") +}Also applies to: 16-18
23-25: Silence unused param warning by naming it_.Cleaner for linters and intent.
-func (f *fakeCatalog) ResolvePath(templateName, second string) (string, error) { +func (f *fakeCatalog) ResolvePath(templateName, _ string) (string, error) { return templateName, nil }
14-15: Simplify fakeCatalog and add compile-time conformance check.No need to embed the interface; assert conformance explicitly.
-type fakeCatalog struct{ catalog.Catalog } +type fakeCatalog struct{} + +var _ catalog.Catalog = (*fakeCatalog)(nil)
41-51: Optional: run tests in parallel.If PayloadGenerator/newTestGenerator have no shared mutable global state, mark tests parallel for faster CI.
func TestLoadPayloads_FastPathFile(t *testing.T) { + t.Parallel() @@ func TestLoadPayloads_InlineMultiline(t *testing.T) { + t.Parallel() @@ func TestValidate_AllowsInlineMultiline(t *testing.T) { + t.Parallel() @@ func TestLoadPayloads_SingleLineFallsBackToFile(t *testing.T) { + t.Parallel() @@ func TestLoadPayloads_InterfaceSlice(t *testing.T) { + t.Parallel() @@ func TestLoadPayloadsFromFile_SkipsEmpty(t *testing.T) { + t.Parallel()Also applies to: 53-64, 66-72, 74-85, 87-97, 99-109
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
pkg/operators/cache/cache.go(1 hunks)pkg/operators/cache/cache_test.go(1 hunks)pkg/operators/extractors/extract.go(1 hunks)pkg/protocols/common/generators/load_test.go(1 hunks)pkg/protocols/common/generators/validate.go(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
- pkg/operators/cache/cache_test.go
- pkg/operators/extractors/extract.go
- pkg/protocols/common/generators/validate.go
🧰 Additional context used
📓 Path-based instructions (2)
**/*.go
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.go: Format Go code using go fmt
Run static analysis with go vet
Files:
pkg/protocols/common/generators/load_test.gopkg/operators/cache/cache.go
pkg/protocols/**/*.go
📄 CodeRabbit inference engine (CLAUDE.md)
Each protocol implementation must provide a Request interface with methods Compile(), ExecuteWithResults(), Match(), and Extract()
Files:
pkg/protocols/common/generators/load_test.go
🧬 Code graph analysis (1)
pkg/protocols/common/generators/load_test.go (2)
pkg/protocols/common/generators/generators.go (1)
PayloadGenerator(14-19)pkg/types/types.go (1)
LoadHelperFileFunction(29-29)
🪛 GitHub Check: Lint
pkg/protocols/common/generators/load_test.go
[failure] 5-5:
SA1019: "io/ioutil" has been deprecated since Go 1.19: As of Go 1.16, the same functionality is now provided by package [io] or package [os], and those implementations should be preferred in new code. See the specific function documentation for details. (staticcheck)
🪛 GitHub Actions: 🔨 Tests
pkg/protocols/common/generators/load_test.go
[error] 5-5: golangci-lint: SA1019: 'io/ioutil' is deprecated since Go 1.19. Use io or os instead. (staticcheck)
🔇 Additional comments (3)
pkg/operators/cache/cache.go (2)
22-27: Lazy init via sync.Once is correct.Single-build, no lock held during construction. LGTM.
29-48: SetCapacities is only invoked in tests, not at runtime
Repo-wide search shows SetCapacities is only called in pkg/operators/cache/cache_test.go and has no production call sites, so there’s no hot-path or runtime rebuild risk.pkg/protocols/common/generators/load_test.go (1)
41-51: Nice: self-contained, filesystem-free tests.In-memory loader + fake catalog keeps tests fast and hermetic. Assertions cover whitespace and empty-line handling well.
Also applies to: 53-64, 74-85, 87-97, 99-109, 66-72
- Removed maxRegexScanBytes constant from match.go. - Replaced ioutil with io package in load_test.go for NopCloser usage. - Restored TestValidate_AllowsInlineMultiline in load_test.go to ensure inline validation functionality.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (2)
pkg/protocols/common/generators/load_test.go (1)
4-5: Good fix: dropped deprecated io/ioutil and used io.NopCloser.Resolves SA1019 and modernizes the tests.
pkg/operators/matchers/match.go (1)
124-135: Unify OR fast-path with cached scan to avoid double regex passes.Current flow does two scans on no-match paths (FindAllString with 1, then -1) and bypasses the cache/large-input guard. Reuse the cached path with a limit to cover both OR fast-path and full scan.
Apply:
- // Fast OR-path: return first match without full scan - if matcher.condition == ORCondition && !matcher.MatchAll { - m := regex.FindAllString(corpus, 1) - if len(m) == 0 { - continue - } - return true, m - } - - // Single scan: get all matches directly - currentMatches := regex.FindAllString(corpus, -1) + // Single cached path: use limit=1 for OR fast-path, else -1 + limit := -1 + if matcher.condition == ORCondition && !matcher.MatchAll { + limit = 1 + } + currentMatches := cachedFindAllString(regex, corpus, limit) if len(currentMatches) == 0 { switch matcher.condition { case ANDCondition: return false, []string{} case ORCondition: continue } } + if matcher.condition == ORCondition && !matcher.MatchAll { + return true, currentMatches + }Run to confirm cache helper exists and no stray direct calls remain:
#!/bin/bash set -euo pipefail # 1) Verify cache helper exists rg -nP --type=go '\bcachedFindAllString\s*\(' -C2 # 2) Ensure no direct OR fast-path calls bypass cache in matchers rg -nP --type=go -g 'pkg/operators/matchers/**' 'FindAllString\s*\(\s*corpus\s*,\s*1\s*\)' -C2
🧹 Nitpick comments (5)
pkg/protocols/common/generators/load_test.go (5)
8-11: Prefer stdlib errors over github.com/pkg/errors in tests.No wrapping/stack is used; stdlib keeps dependencies lean.
Apply:
-import ( - "io" - "strings" - "testing" - - "github.com/pkg/errors" +import ( + "errors" + "io" + "strings" + "testing" @@ -func (f *fakeCatalog) OpenFile(filename string) (io.ReadCloser, error) { - return nil, errors.New("not used") +func (f *fakeCatalog) OpenFile(filename string) (io.ReadCloser, error) { + return nil, errors.New("not used")
13-24: Stub catalog is fine; consider narrowing to only needed methods.Reduces maintenance if catalog interface evolves. Optional.
65-76: Strengthen assertion by checking actual payload content, not only length.Prevents false positives if loader returns wrong lines with correct count.
- got := out["C"] - if len(got) != 3 { - t.Fatalf("unexpected len: %d", len(got)) - } + got := out["C"] + if len(got) != 3 || got[0] != "one" || got[1] != " two" || got[2] != "three" { + t.Fatalf("unexpected: %#v", got) + }
78-88: Add a case for non-string elements in []interface{}.E.g., []interface{}{"p", 2, true} to verify coercion/error behavior is intentional.
90-99: Consider CRLF coverage.Add a variant with "a\r\n\r\n b \r\n" to ensure Windows line endings are handled.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
pkg/operators/matchers/match.go(2 hunks)pkg/protocols/common/generators/load_test.go(1 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.go
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.go: Format Go code using go fmt
Run static analysis with go vet
Files:
pkg/operators/matchers/match.gopkg/protocols/common/generators/load_test.go
pkg/protocols/**/*.go
📄 CodeRabbit inference engine (CLAUDE.md)
Each protocol implementation must provide a Request interface with methods Compile(), ExecuteWithResults(), Match(), and Extract()
Files:
pkg/protocols/common/generators/load_test.go
🧬 Code graph analysis (2)
pkg/operators/matchers/match.go (1)
pkg/operators/matchers/matchers.go (2)
ANDCondition(145-145)ORCondition(147-147)
pkg/protocols/common/generators/load_test.go (2)
pkg/protocols/common/generators/generators.go (1)
PayloadGenerator(14-19)pkg/types/types.go (1)
LoadHelperFileFunction(29-29)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Tests (ubuntu-latest)
- GitHub Check: Tests (windows-latest)
- GitHub Check: Tests (macOS-latest)
🔇 Additional comments (6)
pkg/protocols/common/generators/load_test.go (4)
40-50: LGTM: covers file fast-path and blank-line skipping with whitespace preservation.
52-63: LGTM: inline multiline behavior validated, including trailing empty line.
102-107: LGTM: validate allows inline multiline inputs.Solid guard against over-restrictive validation.
26-38: No changes required: loadPayloadsFromFile already defers closing the returned ReadCloser.pkg/operators/matchers/match.go (2)
109-122: Literal-prefix short-circuit fix looks good.Skipping the prefix check when the pattern contains (?i) avoids case-insensitive false negatives. LGTM.
144-145: Comment clarity is fine.The note about OR with MatchAll helps future readers.
- Updated executeTemplateWithTargets to respect context cancellation. - Introduced fakeTargetProvider and slowExecuter for testing. - Added Test_executeTemplateWithTargets_RespectsCancellation to validate cancellation behavior during template execution.
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
pkg/core/executors.go (2)
31-41: Do not swallow ExecuteWithResults errors; derive match from results lengthSelf-contained path unconditionally sets match=true and shadows err, so failures become silent and results are forced true. Reuse executeTemplateOnInput to unify behavior and propagate errors.
Apply:
- var err error - var match bool - ctx := scan.NewScanContext(ctx, contextargs.New(ctx)) - if e.Callback != nil { - if results, err := template.Executer.ExecuteWithResults(ctx); err == nil { - for _, result := range results { - e.Callback(result) - } - } - - match = true - } else { - match, err = template.Executer.Execute(ctx) - } + var err error + var match bool + match, err = e.executeTemplateOnInput(ctx, template, &contextargs.MetaInput{})
152-171: Leak/incorrect resume state: InFlight not cleared on host-skipWhen a host is skipped via HostErrorsCache, index was added to currentInfo.InFlight but never removed, skewing resume state and leaking entries.
if e.executerOpts.HostErrorsCache != nil && e.executerOpts.HostErrorsCache.Check(e.executerOpts.ProtocolType.String(), contextargs.NewWithMetaInput(ctx, scannedValue)) { + cleanupInFlight(index) skipEvent := &output.ResultEvent{pkg/protocols/http/request.go (1)
455-466: Bug: pass updatedInput to executeRequest in turbo pipelineWorker checks use updatedInput, but executeRequest still receives the original input, leading to wrong host/Error tracking and event fields.
- case spmHandler.ResultChan <- request.executeRequest(input, httpRequest, previous, false, wrappedCallback, 0): + case spmHandler.ResultChan <- request.executeRequest(updatedInput, httpRequest, previous, false, wrappedCallback, 0):
♻️ Duplicate comments (1)
pkg/core/executors.go (1)
173-173: Avoid MetaInput aliasing; enqueue a copyThe same MetaInput instance may be reused/mutated by the iterator; workers could observe races or inconsistent fields. Send a value copy.
- tasks <- task{index: index, skip: skip, value: scannedValue} + vcopy := *scannedValue + tasks <- task{index: index, skip: skip, value: &vcopy}
🧹 Nitpick comments (8)
pkg/core/executors.go (3)
88-121: Minor: buffer tasks channel to workerCountUnbuffered tasks can add avoidable scheduler contention. A small buffer equal to workerCount brings smoother throughput without changing semantics.
- tasks := make(chan task) + tasks := make(chan task, workerCount)
178-179: Optional: recover in workersA panic in a single task would crash the whole scan. Add a lightweight recover to confine blast radius and log.
- go func() { + go func() { + defer func() { + if r := recover(); r != nil { + e.options.Logger.Warning().Msgf("[%s] worker panicked: %v", template.ID, r) + } + }() defer workersWg.Done()
226-248: Nil-safety: don't overwrite MetaInput when value is nilKeep default MetaInput from contextargs.New(ctx) if caller passes nil (e.g., self-contained paths).
- ctxArgs := contextargs.New(ctx) - ctxArgs.MetaInput = value + ctxArgs := contextargs.New(ctx) + if value != nil { + ctxArgs.MetaInput = value + }pkg/core/executors_test.go (1)
134-148: Add an assertion to ensure no deadlock and quick exitThe test only exercises the path; assert that it returns promptly and doesn’t set matched.
- e.executeTemplateWithTargets(ctx, tpl, targets, &matched) + start := time.Now() + e.executeTemplateWithTargets(ctx, tpl, targets, &matched) + if time.Since(start) > 500*time.Millisecond { + t.Fatalf("expected early return on cancelled context") + } + if matched.Load() { + t.Fatalf("expected no matches on cancelled context") + }pkg/protocols/http/request.go (3)
299-311: Optional: support shrinking worker count on downsizeWhen PayloadConcurrency decreases, consider allowing workers to retire to match spm size; keeps footprint tight.
- if spmHandler.Size() > currentWorkers { + switch { + case spmHandler.Size() > currentWorkers: for i := 0; i < spmHandler.Size()-currentWorkers; i++ { spawnWorker(ctx) } currentWorkers = spmHandler.Size() + case spmHandler.Size() < currentWorkers: + // let extra workers exit by closing a separate quit chan or by checking a shared atomic size + currentWorkers = spmHandler.Size() }
336-346: Minor: avoid per-task map allocationpreviousEvent is read-only in executeRequest; reuse a shared empty map to reduce allocations.
- case tasks <- task{req: generatedHttpRequest, updatedInput: updatedInput}: + case tasks <- task{req: generatedHttpRequest, updatedInput: updatedInput}:And define once near workers:
emptyPrev := make(map[string]interface{}) // ... case spmHandler.ResultChan <- request.executeRequest(t.updatedInput, t.req, emptyPrev, false, wrappedCallback, 0):
161-181: Parity: race path should use task-specific input and late rate-limit (if applicable)executeRaceRequest launches per-request goroutines with outer input and no ratelimit. If generator can alter host/URL, consider using updatedInput like parallel/pipeline paths and aligning ratelimit placement.
- go func(httpRequest *generatedRequest) { + go func(httpRequest *generatedRequest) { defer spmHandler.Release() - if spmHandler.FoundFirstMatch() || request.isUnresponsiveAddress(input) { + updatedInput := contextargs.GetCopyIfHostOutdated(input, httpRequest.URL()) + if spmHandler.FoundFirstMatch() || request.isUnresponsiveAddress(updatedInput) { return } select { case <-spmHandler.Done(): return - case spmHandler.ResultChan <- request.executeRequest(input, httpRequest, previous, false, wrappedCallback, 0): + case spmHandler.ResultChan <- request.executeRequest(updatedInput, httpRequest, previous, false, wrappedCallback, 0): return } }(generatedRequests[i])Optionally add request.options.RateLimitTake() after re-checks if rate limiting should apply to race mode.
pkg/operators/cache/cache_test.go (1)
97-114: Consider adding synchronization verification for the concurrent test.While this test exercises concurrent access patterns, it doesn't explicitly verify that the operations complete successfully or that data integrity is maintained. The test could be enhanced to verify that no panics occur and that cache operations maintain consistency.
Consider adding assertions to verify successful operations:
func TestSetCapacities_ConcurrentAccess(t *testing.T) { SetCapacities(64, 64) stop := make(chan struct{}) + errChan := make(chan error, 1) go func() { + defer func() { + if r := recover(); r != nil { + errChan <- fmt.Errorf("panic in goroutine: %v", r) + } + close(stop) + }() for i := 0; i < 5000; i++ { - _ = Regex().Set("k"+string(rune('a'+(i%26))), regexp.MustCompile("a")) + key := "k" + string(rune('a'+(i%26))) + if err := Regex().Set(key, regexp.MustCompile("a")); err != nil { + errChan <- fmt.Errorf("failed to set regex: %v", err) + return + } _, _ = Regex().GetIFPresent("k" + string(rune('a'+(i%26)))) _, _ = DSL().GetIFPresent("1+2==3") } - close(stop) }() for i := 0; i < 200; i++ { SetCapacities(64+(i%5), 64+((i+1)%5)) } <-stop + + select { + case err := <-errChan: + t.Fatalf("concurrent access error: %v", err) + default: + // No errors, test passed + } }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
pkg/core/executors.go(4 hunks)pkg/core/executors_test.go(1 hunks)pkg/operators/cache/cache_test.go(1 hunks)pkg/protocols/http/request.go(3 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.go
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.go: Format Go code using go fmt
Run static analysis with go vet
Files:
pkg/core/executors_test.gopkg/operators/cache/cache_test.gopkg/protocols/http/request.gopkg/core/executors.go
pkg/protocols/**/*.go
📄 CodeRabbit inference engine (CLAUDE.md)
Each protocol implementation must provide a Request interface with methods Compile(), ExecuteWithResults(), Match(), and Extract()
Files:
pkg/protocols/http/request.go
🧬 Code graph analysis (4)
pkg/core/executors_test.go (6)
pkg/scan/scan_context.go (1)
ScanContext(22-44)pkg/output/output.go (1)
ResultEvent(146-221)pkg/core/engine.go (1)
Engine(18-24)pkg/protocols/protocols.go (2)
Executer(49-58)ExecutorOptions(61-141)pkg/protocols/common/contextargs/metainput.go (1)
MetaInput(18-30)pkg/input/types/probe.go (1)
InputLivenessProbe(4-9)
pkg/operators/cache/cache_test.go (1)
pkg/operators/cache/cache.go (3)
Regex(50-55)DSL(57-62)SetCapacities(29-48)
pkg/protocols/http/request.go (2)
pkg/protocols/common/contextargs/contextargs.go (1)
Context(22-33)pkg/progress/progress.go (1)
Progress(17-35)
pkg/core/executors.go (6)
pkg/protocols/common/contextargs/metainput.go (1)
MetaInput(18-30)pkg/core/engine.go (2)
Engine(18-24)New(27-34)pkg/protocols/common/contextargs/contextargs.go (2)
Context(22-33)New(36-38)pkg/scan/scan_context.go (1)
NewScanContext(47-49)pkg/templates/types/types.go (1)
WorkflowProtocol(33-33)pkg/protocols/protocols.go (1)
Executer(49-58)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Lint
🔇 Additional comments (11)
pkg/core/executors.go (1)
51-59: LGTM: bounded workers derived from poolPool lookup + sane default reads well.
pkg/core/executors_test.go (3)
40-58: LGTM: callback path verifies propagation and ok derivationGood coverage for ExecuteWithResults + callback behavior.
60-72: LGTM: non-callback Execute pathAsserts ok=true when Execute returns true.
83-96: LGTM: error propagation testConfirms error bubbles up and ok=false.
pkg/protocols/http/request.go (1)
243-280: LGTM: bounded worker-pool with early checks and proper acquire/releaseGood placement of cancellation checks and ratelimit take after guards.
pkg/operators/cache/cache_test.go (6)
10-28: LGTM! Comprehensive test for basic regex cache operations.The test properly validates the Set/Get operations for the regex cache, including error handling and pattern matching verification.
30-47: LGTM! Well-structured test for DSL cache operations.The test correctly validates the DSL cache functionality with proper error handling and expression comparison.
49-67: LGTM! Effective LRU eviction test.The test properly validates that the LRU eviction policy works as expected when the cache reaches its capacity limit.
69-83: LGTM! Important edge case test for zero capacities.This test validates the critical behavior that passing zero capacities to
SetCapacitiesdoesn't rebuild or clear existing caches, which is essential for maintaining cache state.
85-95: LGTM! Validates capacity configuration before first use.The test ensures that early capacity configuration is properly honored and not overridden by subsequent lazy initialization.
1-114: Excellent test coverage for the new caching layer!The test suite comprehensively covers the critical aspects of the caching implementation:
- Basic cache operations (Set/Get) for both regex and DSL caches
- LRU eviction behavior
- Edge cases with zero capacities
- Pre-initialization capacity configuration
- Concurrent access patterns
The tests are well-structured, focused, and provide good coverage for ensuring the reliability of the caching layer that will be used across the matchers and extractors modules.
* CheckRDPEncryption
* feat(templating): add vars templating into yaml inputs
* fix: enhance code rabbit
* fix: change gologger runner version
* feat(ytt): add ytt files var + add vars from cli and config
* feat: send struct from var file
* fix code rabbit
* fix(main.go): add errcheck
* retain required empty spaces
* fixing path
* fixing test
* use bytes slice
* Add option to control number of concurrent templates loaded on startup
* adding vnc auth
* gen go+js
* lint
* no changes custom template message should be INF not ERR
* Path-Based Fuzzing SQL fix (#6400)
* setup claude
* migrate to using errkit
* fix unused imports + lint errors
* update settings.json
* fix url encoding issue
* fix lint error
* fix the path fuzzing component
* fix lint error
* fix(fuzz): handles duplicate multipart form field names (#6404)
* fix: handle duplicate field names in multipart form encoding
* fix(fuzz): handles `[]any` type in `*MultiPartForm.Encode`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(fuzz): adds panic recovery & display encoded out
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(fuzz): incorrectly treated mixed type field
in `*MultiPartForm.Encode`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(fuzz): refactor compare w decoded instead
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(fuzz): prealloc for `[]any` type
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(fuzz): treats nil value as empty string
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(fuzz): rm early error return for non-array file
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(fuzz): adds `TestMultiPartFormFileUpload` test
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
Co-authored-by: yusei-wy <31252054+yusei-wy@users.noreply.github.com>
* limited test, instead of all
* lint
* integration test
* lint
* Update pkg/external/customtemplates/github.go
Co-authored-by: Dwi Siswanto <25837540+dwisiswant0@users.noreply.github.com>
* fix for error.Is false return
* bump httpx version
* chore(deps): bump github.com/go-viper/mapstructure/v2
Bumps the go_modules group with 1 update in the / directory: [github.com/go-viper/mapstructure/v2](https://github.com/go-viper/mapstructure).
Updates `github.com/go-viper/mapstructure/v2` from 2.3.0 to 2.4.0
- [Release notes](https://github.com/go-viper/mapstructure/releases)
- [Changelog](https://github.com/go-viper/mapstructure/blob/main/CHANGELOG.md)
- [Commits](https://github.com/go-viper/mapstructure/compare/v2.3.0...v2.4.0)
---
updated-dependencies:
- dependency-name: github.com/go-viper/mapstructure/v2
dependency-version: 2.4.0
dependency-type: indirect
dependency-group: go_modules
...
Signed-off-by: dependabot[bot] <support@github.com>
* test(reporting/exporters/mongo): add mongo integration test with test… (#6237)
* test(reporting/exporters/mongo): add mongo integration test with testcontainer-go module
Signed-off-by: Lorenzo Susini <susinilorenzo1@gmail.com>
* execute exportes only on linux
---------
Signed-off-by: Lorenzo Susini <susinilorenzo1@gmail.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
* Refactor to use reflect.TypeFor (#6428)
* issue / discussion template update
* misc hyperlink update
* link update
* chore(deps): bump the modules group across 1 directory with 11 updates (#6438)
* chore(deps): bump the modules group across 1 directory with 11 updates
Bumps the modules group with 10 updates in the / directory:
| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.6` | `0.4.7` |
| [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.92` | `0.0.93` |
| [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.105` | `1.0.106` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.120` | `1.0.121` |
| [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.5.0` | `0.5.1` |
| [github.com/projectdiscovery/gozero](https://github.com/projectdiscovery/gozero) | `0.0.3` | `0.1.0` |
| [github.com/projectdiscovery/ratelimit](https://github.com/projectdiscovery/ratelimit) | `0.0.81` | `0.0.82` |
| [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx) | `1.1.9` | `1.2.0` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.37` | `0.2.43` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.1.27` | `1.1.33` |
Updates `github.com/projectdiscovery/fastdialer` from 0.4.6 to 0.4.7
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.6...v0.4.7)
Updates `github.com/projectdiscovery/hmap` from 0.0.92 to 0.0.93
- [Release notes](https://github.com/projectdiscovery/hmap/releases)
- [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.92...v0.0.93)
Updates `github.com/projectdiscovery/retryabledns` from 1.0.105 to 1.0.106
- [Release notes](https://github.com/projectdiscovery/retryabledns/releases)
- [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.105...v1.0.106)
Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.120 to 1.0.121
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.120...v1.0.121)
Updates `github.com/projectdiscovery/dsl` from 0.5.0 to 0.5.1
- [Release notes](https://github.com/projectdiscovery/dsl/releases)
- [Commits](https://github.com/projectdiscovery/dsl/compare/v0.5.0...v0.5.1)
Updates `github.com/projectdiscovery/gozero` from 0.0.3 to 0.1.0
- [Release notes](https://github.com/projectdiscovery/gozero/releases)
- [Commits](https://github.com/projectdiscovery/gozero/compare/v0.0.3...v0.1.0)
Updates `github.com/projectdiscovery/networkpolicy` from 0.1.20 to 0.1.21
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.20...v0.1.21)
Updates `github.com/projectdiscovery/ratelimit` from 0.0.81 to 0.0.82
- [Release notes](https://github.com/projectdiscovery/ratelimit/releases)
- [Commits](https://github.com/projectdiscovery/ratelimit/compare/v0.0.81...v0.0.82)
Updates `github.com/projectdiscovery/tlsx` from 1.1.9 to 1.2.0
- [Release notes](https://github.com/projectdiscovery/tlsx/releases)
- [Changelog](https://github.com/projectdiscovery/tlsx/blob/main/.goreleaser.yml)
- [Commits](https://github.com/projectdiscovery/tlsx/compare/v1.1.9...v1.2.0)
Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.37 to 0.2.43
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.37...v0.2.43)
Updates `github.com/projectdiscovery/cdncheck` from 1.1.27 to 1.1.33
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.1.27...v1.1.33)
---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/fastdialer
dependency-version: 0.4.7
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/hmap
dependency-version: 0.0.93
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryabledns
dependency-version: 1.0.106
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
dependency-version: 1.0.121
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/dsl
dependency-version: 0.5.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/gozero
dependency-version: 0.1.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
dependency-version: 0.1.21
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/ratelimit
dependency-version: 0.0.82
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/tlsx
dependency-version: 1.2.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
dependency-version: 0.2.43
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
dependency-version: 1.1.33
dependency-type: indirect
update-type: version-update:semver-patch
dependency-group: modules
...
Signed-off-by: dependabot[bot] <support@github.com>
* bump
* httpx dev
* mod tidy
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
* Reporting validation (#6456)
* add custom validator for reporting issues
* use httpx dev branch
* remove yaml marshal/unmarshal for validator callback
* chore(deps): bump the workflows group across 1 directory with 2 updates (#6462)
Bumps the workflows group with 2 updates in the / directory: [actions/checkout](https://github.com/actions/checkout) and [actions/stale](https://github.com/actions/stale).
Updates `actions/checkout` from 4 to 5
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v5)
Updates `actions/stale` from 9 to 10
- [Release notes](https://github.com/actions/stale/releases)
- [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/stale/compare/v9...v10)
---
updated-dependencies:
- dependency-name: actions/checkout
dependency-version: '5'
dependency-type: direct:production
update-type: version-update:semver-major
dependency-group: workflows
- dependency-name: actions/stale
dependency-version: '10'
dependency-type: direct:production
update-type: version-update:semver-major
dependency-group: workflows
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* feat: added new text/template syntax to jira custom fields
* feat: added additional text/template helpers
* dont load templates with the same ID
* using synclockmap
* lint
* lint
* chore(deps): bump the modules group with 9 updates
Bumps the modules group with 9 updates:
| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.7` | `0.4.9` |
| [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.106` | `1.0.107` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.121` | `1.0.123` |
| [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.5.1` | `0.6.0` |
| [github.com/projectdiscovery/httpx](https://github.com/projectdiscovery/httpx) | `1.7.1-0.20250902174407-8d6c2658663f` | `1.7.1` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.21` | `0.1.23` |
| [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.4.24-0.20250823123502-bd7f2849ddb4` | `0.5.0` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.43` | `0.2.45` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.1.33` | `1.1.35` |
Updates `github.com/projectdiscovery/fastdialer` from 0.4.7 to 0.4.9
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.7...v0.4.9)
Updates `github.com/projectdiscovery/retryabledns` from 1.0.106 to 1.0.107
- [Release notes](https://github.com/projectdiscovery/retryabledns/releases)
- [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.106...v1.0.107)
Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.121 to 1.0.123
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.121...v1.0.123)
Updates `github.com/projectdiscovery/dsl` from 0.5.1 to 0.6.0
- [Release notes](https://github.com/projectdiscovery/dsl/releases)
- [Commits](https://github.com/projectdiscovery/dsl/compare/v0.5.1...v0.6.0)
Updates `github.com/projectdiscovery/httpx` from 1.7.1-0.20250902174407-8d6c2658663f to 1.7.1
- [Release notes](https://github.com/projectdiscovery/httpx/releases)
- [Changelog](https://github.com/projectdiscovery/httpx/blob/dev/.goreleaser.yml)
- [Commits](https://github.com/projectdiscovery/httpx/commits/v1.7.1)
Updates `github.com/projectdiscovery/networkpolicy` from 0.1.21 to 0.1.23
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.21...v0.1.23)
Updates `github.com/projectdiscovery/utils` from 0.4.24-0.20250823123502-bd7f2849ddb4 to 0.5.0
- [Release notes](https://github.com/projectdiscovery/utils/releases)
- [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md)
- [Commits](https://github.com/projectdiscovery/utils/commits/v0.5.0)
Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.43 to 0.2.45
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.43...v0.2.45)
Updates `github.com/projectdiscovery/cdncheck` from 1.1.33 to 1.1.35
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.1.33...v1.1.35)
---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/fastdialer
dependency-version: 0.4.9
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryabledns
dependency-version: 1.0.107
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
dependency-version: 1.0.123
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/dsl
dependency-version: 0.6.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: modules
- dependency-name: github.com/projectdiscovery/httpx
dependency-version: 1.7.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
dependency-version: 0.1.23
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/utils
dependency-version: 0.5.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
dependency-version: 0.2.45
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
dependency-version: 1.1.35
dependency-type: indirect
update-type: version-update:semver-patch
dependency-group: modules
...
Signed-off-by: dependabot[bot] <support@github.com>
* httpx fix
* release fix
* code from https://github.com/projectdiscovery/nuclei/pull/6427
* lint
* centralizing ratelimiter logic
* adding me
* refactor
* Remove the stack trace when the nuclei-ignore file does not exist (#6455)
* remove the stack trace when the nuclei-ignore file does not exist
* removing useless debug stack
---------
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
* chore(deps): bump the modules group with 7 updates
Bumps the modules group with 7 updates:
| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.9` | `0.4.10` |
| [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.93` | `0.0.94` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.123` | `1.0.124` |
| [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.6.0` | `0.7.0` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.23` | `0.1.24` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.45` | `0.2.46` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.1.35` | `1.1.36` |
Updates `github.com/projectdiscovery/fastdialer` from 0.4.9 to 0.4.10
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.9...v0.4.10)
Updates `github.com/projectdiscovery/hmap` from 0.0.93 to 0.0.94
- [Release notes](https://github.com/projectdiscovery/hmap/releases)
- [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.93...v0.0.94)
Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.123 to 1.0.124
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.123...v1.0.124)
Updates `github.com/projectdiscovery/dsl` from 0.6.0 to 0.7.0
- [Release notes](https://github.com/projectdiscovery/dsl/releases)
- [Commits](https://github.com/projectdiscovery/dsl/compare/v0.6.0...v0.7.0)
Updates `github.com/projectdiscovery/networkpolicy` from 0.1.23 to 0.1.24
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.23...v0.1.24)
Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.45 to 0.2.46
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.45...v0.2.46)
Updates `github.com/projectdiscovery/cdncheck` from 1.1.35 to 1.1.36
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.1.35...v1.1.36)
---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/fastdialer
dependency-version: 0.4.10
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/hmap
dependency-version: 0.0.94
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
dependency-version: 1.0.124
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/dsl
dependency-version: 0.7.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
dependency-version: 0.1.24
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
dependency-version: 0.2.46
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
dependency-version: 1.1.36
dependency-type: indirect
update-type: version-update:semver-patch
dependency-group: modules
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix: update go jira deps (#6475)
* fix: handle jira deprecated endpoint
* refactor: update Jira issue search result structure to include 'Self' field
* Revert "refactor: update Jira issue search result structure to include 'Self' field"
This reverts commit b0953419d33dff3fb61f1bcdcddab0ae759379b8.
* Revert "fix: handle jira deprecated endpoint"
This reverts commit 1fc05076cdb31906f403d80455b2e1609a66e2ae.
* chore(deps): bump github.com/andygrunwald/go-jira to v1.16.1 and tidy
* fix(jira): migrate Issue.Search to SearchV2JQL with explicit Fields
* cache, goroutine and unbounded workers management (#6420)
* Enhance matcher compilation with caching for regex and DSL expressions to improve performance. Update template parsing to conditionally retain raw templates based on size constraints.
* Implement caching for regex and DSL expressions in extractors and matchers to enhance performance. Introduce a buffer pool in raw requests to reduce memory allocations. Update template cache management for improved efficiency.
* feat: improve concurrency to be bound
* refactor: replace fmt.Sprintf with fmt.Fprintf for improved performance in header handling
* feat: add regex matching tests and benchmarks for performance evaluation
* feat: add prefix check in regex extraction to optimize matching process
* feat: implement regex caching mechanism to enhance performance in extractors and matchers, along with tests and benchmarks for validation
* feat: add unit tests for template execution in the core engine, enhancing test coverage and reliability
* feat: enhance error handling in template execution and improve regex caching logic for better performance
* Implement caching for regex and DSL expressions in the cache package, replacing previous sync.Map usage. Add unit tests for cache functionality, including eviction by capacity and retrieval of cached items. Update extractors and matchers to utilize the new cache system for improved performance and memory efficiency.
* Add tests for SetCapacities in cache package to ensure cache behavior on capacity changes
- Implemented TestSetCapacities_NoRebuildOnZero to verify that setting capacities to zero does not clear existing caches.
- Added TestSetCapacities_BeforeFirstUse to confirm that initial cache settings are respected and not overridden by subsequent capacity changes.
* Refactor matchers and update load test generator to use io package
- Removed maxRegexScanBytes constant from match.go.
- Replaced ioutil with io package in load_test.go for NopCloser usage.
- Restored TestValidate_AllowsInlineMultiline in load_test.go to ensure inline validation functionality.
* Add cancellation support in template execution and enhance test coverage
- Updated executeTemplateWithTargets to respect context cancellation.
- Introduced fakeTargetProvider and slowExecuter for testing.
- Added Test_executeTemplateWithTargets_RespectsCancellation to validate cancellation behavior during template execution.
* Refactored header-based auth scans not to normalize the header names. (#6479)
* Refactored header-based auth scans not to normalize the header names.
* Removed the header validation as it's not really useful here.
* adding docs
---------
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
* docs: update syntax & JSON schema 🤖
* chore(deps): bump the modules group with 6 updates
Bumps the modules group with 6 updates:
| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.10` | `0.4.11` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.124` | `1.0.125` |
| [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.54` | `1.1.55` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.24` | `0.1.25` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.46` | `0.2.47` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.1.36` | `1.2.0` |
Updates `github.com/projectdiscovery/fastdialer` from 0.4.10 to 0.4.11
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.10...v0.4.11)
Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.124 to 1.0.125
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.124...v1.0.125)
Updates `github.com/projectdiscovery/gologger` from 1.1.54 to 1.1.55
- [Release notes](https://github.com/projectdiscovery/gologger/releases)
- [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.54...v1.1.55)
Updates `github.com/projectdiscovery/networkpolicy` from 0.1.24 to 0.1.25
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.24...v0.1.25)
Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.46 to 0.2.47
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.46...v0.2.47)
Updates `github.com/projectdiscovery/cdncheck` from 1.1.36 to 1.2.0
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.1.36...v1.2.0)
---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/fastdialer
dependency-version: 0.4.11
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
dependency-version: 1.0.125
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/gologger
dependency-version: 1.1.55
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
dependency-version: 0.1.25
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
dependency-version: 0.2.47
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
dependency-version: 1.2.0
dependency-type: indirect
update-type: version-update:semver-minor
dependency-group: modules
...
Signed-off-by: dependabot[bot] <support@github.com>
* Feat 6231 deadlock (#6469)
* fixing recursive deadlock
* using atomics
* fixing init
* feat(fuzz): enhance `MultiPartForm` with metadata APIs (#6486)
* feat(fuzz): enhance `MultiPartForm` with metadata APIs
* add `SetFileMetadata`/`GetFileMetadata` APIs for
file metadata management.
* implement RFC-2046 boundary validation
(max 70 chars).
* add boundary validation in `Decode` method.
* fix `filesMetadata` initialization.
* fix mem leak by removing defer from file reading
loop.
* fix file metadata overwriting by storing first
file's metadata instead of last.
Closes #6405, #6406
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(fuzz): satisfy lint errs
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
* jira: hotfix for Cloud to use /rest/api/3/search/jql (#6489)
* jira: hotfix for Cloud to use /rest/api/3/search/jql in FindExistingIssue; add live test verifying v3 endpoint
* jira: fix Cloud v3 search response handling (no total); set Self from base
* fix lint error
* tests(jira): apply De Morgan to satisfy staticcheck QF1001
* fix headless template loading logic when `-dast` option is enabled
* fix: improve cleanup in parallel execution (#6490)
* fixing logic
* fix(templates): suppress warn code flag not found
on validate.
fixes #6498
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(config): adds known misc directories
and excludes em in IsTemplate func.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(disk): uses `config.IsTemplate` instead
fixes #6499
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(make): rm unnecessary flag on template-validate
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(confif): update known misc dirs & improve IsTemplate func
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(deps): bump the modules group with 7 updates (#6505)
Bumps the modules group with 7 updates:
| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.125` | `1.0.126` |
| [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.7.0` | `0.7.1` |
| [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.55` | `1.1.56` |
| [github.com/projectdiscovery/mapcidr](https://github.com/projectdiscovery/mapcidr) | `1.1.34` | `1.1.95` |
| [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.5.0` | `0.6.0` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.47` | `0.2.48` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.0` | `1.2.3` |
Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.125 to 1.0.126
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.125...v1.0.126)
Updates `github.com/projectdiscovery/dsl` from 0.7.0 to 0.7.1
- [Release notes](https://github.com/projectdiscovery/dsl/releases)
- [Commits](https://github.com/projectdiscovery/dsl/compare/v0.7.0...v0.7.1)
Updates `github.com/projectdiscovery/gologger` from 1.1.55 to 1.1.56
- [Release notes](https://github.com/projectdiscovery/gologger/releases)
- [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.55...v1.1.56)
Updates `github.com/projectdiscovery/mapcidr` from 1.1.34 to 1.1.95
- [Release notes](https://github.com/projectdiscovery/mapcidr/releases)
- [Changelog](https://github.com/projectdiscovery/mapcidr/blob/main/.goreleaser.yml)
- [Commits](https://github.com/projectdiscovery/mapcidr/compare/v1.1.34...v1.1.95)
Updates `github.com/projectdiscovery/utils` from 0.5.0 to 0.6.0
- [Release notes](https://github.com/projectdiscovery/utils/releases)
- [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md)
- [Commits](https://github.com/projectdiscovery/utils/compare/v0.5.0...v0.6.0)
Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.47 to 0.2.48
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.47...v0.2.48)
Updates `github.com/projectdiscovery/cdncheck` from 1.2.0 to 1.2.3
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.0...v1.2.3)
---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/retryablehttp-go
dependency-version: 1.0.126
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/dsl
dependency-version: 0.7.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/gologger
dependency-version: 1.1.56
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/mapcidr
dependency-version: 1.1.95
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/utils
dependency-version: 0.6.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
dependency-version: 0.2.48
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
dependency-version: 1.2.3
dependency-type: indirect
update-type: version-update:semver-patch
dependency-group: modules
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(config): normalize `fpath` in `IsTemplate`
* normalize file `fpath` in `IsTemplate` using
filepath.FromSlash to ensure consistent matching
across platforms.
* update `GetKnownMiscDirectories` docs to clarify
that trailing slashes prevent false positives,
since `IsTemplate` compares against normalized
full paths.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* ai recommendations
* chore(deps): bump the modules group with 10 updates
Bumps the modules group with 10 updates:
| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.11` | `0.4.12` |
| [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.94` | `0.0.95` |
| [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.107` | `1.0.108` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.126` | `1.0.127` |
| [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.7.1` | `0.7.2` |
| [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.56` | `1.1.57` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.25` | `0.1.26` |
| [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.101` | `0.0.102` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.48` | `0.2.49` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.3` | `1.2.4` |
Updates `github.com/projectdiscovery/fastdialer` from 0.4.11 to 0.4.12
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.11...v0.4.12)
Updates `github.com/projectdiscovery/hmap` from 0.0.94 to 0.0.95
- [Release notes](https://github.com/projectdiscovery/hmap/releases)
- [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.94...v0.0.95)
Updates `github.com/projectdiscovery/retryabledns` from 1.0.107 to 1.0.108
- [Release notes](https://github.com/projectdiscovery/retryabledns/releases)
- [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.107...v1.0.108)
Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.126 to 1.0.127
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.126...v1.0.127)
Updates `github.com/projectdiscovery/dsl` from 0.7.1 to 0.7.2
- [Release notes](https://github.com/projectdiscovery/dsl/releases)
- [Commits](https://github.com/projectdiscovery/dsl/compare/v0.7.1...v0.7.2)
Updates `github.com/projectdiscovery/gologger` from 1.1.56 to 1.1.57
- [Release notes](https://github.com/projectdiscovery/gologger/releases)
- [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.56...v1.1.57)
Updates `github.com/projectdiscovery/networkpolicy` from 0.1.25 to 0.1.26
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.25...v0.1.26)
Updates `github.com/projectdiscovery/useragent` from 0.0.101 to 0.0.102
- [Release notes](https://github.com/projectdiscovery/useragent/releases)
- [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.101...v0.0.102)
Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.48 to 0.2.49
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.48...v0.2.49)
Updates `github.com/projectdiscovery/cdncheck` from 1.2.3 to 1.2.4
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.3...v1.2.4)
---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/fastdialer
dependency-version: 0.4.12
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/hmap
dependency-version: 0.0.95
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryabledns
dependency-version: 1.0.108
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
dependency-version: 1.0.127
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/dsl
dependency-version: 0.7.2
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/gologger
dependency-version: 1.1.57
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
dependency-version: 0.1.26
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/useragent
dependency-version: 0.0.102
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
dependency-version: 0.2.49
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
dependency-version: 1.2.4
dependency-type: indirect
update-type: version-update:semver-patch
dependency-group: modules
...
Signed-off-by: dependabot[bot] <support@github.com>
* feat: http(s) probing optimization
* small changes
* updating docs
* chore(typos): fix typos
* log failed expr compilations
* Update Go version badge in README
update accordingly
* Update README.md
edit correct version of go
* Update Go version requirement in README (#6529)
need to update required go version from 1.23 to >=1.24.1
* fix(variable): global variable not same between two request in flow mode (#6395)
* fix(variable): global variable not same between two request in flow mode(#6337)
* update gitignore
---------
Co-authored-by: chuu <7704684+lizhi3n@user.noreply.gitee.com>
Co-authored-by: PDTeamX <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
* chore: add typos check into tests CI
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(deps): bump github/codeql-action in the workflows group
Bumps the workflows group with 1 update: [github/codeql-action](https://github.com/github/codeql-action).
Updates `github/codeql-action` from 3 to 4
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v3...v4)
---
updated-dependencies:
- dependency-name: github/codeql-action
dependency-version: '4'
dependency-type: direct:production
update-type: version-update:semver-major
dependency-group: workflows
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): bump the modules group with 7 updates
Bumps the modules group with 7 updates:
| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.12` | `0.4.13` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.127` | `1.0.128` |
| [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.7.2` | `0.8.0` |
| [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.57` | `1.1.58` |
| [github.com/projectdiscovery/mapcidr](https://github.com/projectdiscovery/mapcidr) | `1.1.95` | `1.1.96` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.49` | `0.2.50` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.4` | `1.2.5` |
Updates `github.com/projectdiscovery/fastdialer` from 0.4.12 to 0.4.13
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.12...v0.4.13)
Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.127 to 1.0.128
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.127...v1.0.128)
Updates `github.com/projectdiscovery/dsl` from 0.7.2 to 0.8.0
- [Release notes](https://github.com/projectdiscovery/dsl/releases)
- [Commits](https://github.com/projectdiscovery/dsl/compare/v0.7.2...v0.8.0)
Updates `github.com/projectdiscovery/gologger` from 1.1.57 to 1.1.58
- [Release notes](https://github.com/projectdiscovery/gologger/releases)
- [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.57...v1.1.58)
Updates `github.com/projectdiscovery/mapcidr` from 1.1.95 to 1.1.96
- [Release notes](https://github.com/projectdiscovery/mapcidr/releases)
- [Changelog](https://github.com/projectdiscovery/mapcidr/blob/main/.goreleaser.yml)
- [Commits](https://github.com/projectdiscovery/mapcidr/compare/v1.1.95...v1.1.96)
Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.49 to 0.2.50
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.49...v0.2.50)
Updates `github.com/projectdiscovery/cdncheck` from 1.2.4 to 1.2.5
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.4...v1.2.5)
---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/fastdialer
dependency-version: 0.4.13
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
dependency-version: 1.0.128
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/dsl
dependency-version: 0.8.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: modules
- dependency-name: github.com/projectdiscovery/gologger
dependency-version: 1.1.58
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/mapcidr
dependency-version: 1.1.96
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
dependency-version: 0.2.50
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
dependency-version: 1.2.5
dependency-type: indirect
update-type: version-update:semver-patch
dependency-group: modules
...
Signed-off-by: dependabot[bot] <support@github.com>
* docs: update syntax & JSON schema 🤖
* Revert "chore: add typos check into tests CI"
This reverts commit 73e70ea49d18faee311be47a4207de8e476ee3a3.
* chore: preserve issue report w/ issue form (#6531)
Signed-off-by: Dwi Siswanto <git@dw1.io>
* perf(loader): reuse cached parsed templates (#6504)
* perf(loader): reuse cached parsed templates
in `(*Store).areWorkflowOrTemplatesValid`, which
is being called during template `-validate`-ion.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(testutils): optionally assign template info
in `NewMockExecuterOptions`, which is not
required for specific case, like when we want to
`(*Store).ValidateTemplates`.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(loader): adds `(*Store).ValidateTemplates` bench
Signed-off-by: Dwi Siswanto <git@dw1.io>
* refactor(templates): adds fast read parser
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test(templates): adds `Parser*` benchs
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(templates): satisfy lints
Signed-off-by: Dwi Siswanto <git@dw1.io>
* revert(templates): rm fast read parser
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix: Add important context to `tl` flag option
* feat: template list alphabetical order
* fix: Implement coderabbit suggestion
* Http probing optimizations high ports (#6538)
* feat: Assume HTTP(S) server on high port is HTTP
* feat: enhance http probing tests
* improving issue description
---------
Co-authored-by: Matej Smycka <smycka@ics.muni.cz>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
* chore(deps): bump the modules group with 5 updates (#6543)
Bumps the modules group with 5 updates:
| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.128` | `1.0.129` |
| [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.0` | `0.8.1` |
| [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.58` | `1.1.59` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.50` | `0.2.51` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.5` | `1.2.6` |
Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.128 to 1.0.129
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.128...v1.0.129)
Updates `github.com/projectdiscovery/dsl` from 0.8.0 to 0.8.1
- [Release notes](https://github.com/projectdiscovery/dsl/releases)
- [Commits](https://github.com/projectdiscovery/dsl/compare/v0.8.0...v0.8.1)
Updates `github.com/projectdiscovery/gologger` from 1.1.58 to 1.1.59
- [Release notes](https://github.com/projectdiscovery/gologger/releases)
- [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.58...v1.1.59)
Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.50 to 0.2.51
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.50...v0.2.51)
Updates `github.com/projectdiscovery/cdncheck` from 1.2.5 to 1.2.6
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.5...v1.2.6)
---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/retryablehttp-go
dependency-version: 1.0.129
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/dsl
dependency-version: 0.8.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/gologger
dependency-version: 1.1.59
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
dependency-version: 0.2.51
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
dependency-version: 1.2.6
dependency-type: indirect
update-type: version-update:semver-patch
dependency-group: modules
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fixing failing integration tests
* clean up pools after 24hours inactivity
* fixing lint
* fixing go routine leak
* bump utils
* fixing leak
* fixing syntax
* removing go logo
* fix: populate req_url_pattern before event creation (#6547)
* refactor(disk): templates catalog (#5914)
* refactor(disk): templates catalog
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(disk): drying err
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(disk): simplify `DiskCatalog.OpenFile` method
since `BackwardsCompatiblePaths` func is already
deprecated.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* test: update functional test cases
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat: reuse error
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(disk): handle glob errors consistently
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(disk): use forward slashes for fs.FS path ops
to fix Windows compat.
The io/fs package requires forward slashes ("/")
as path separators regardless of the OS. Using
[filepath.Separator] or [os.PathSeparator] breaks
[fs.Open] and [fs.Glob] ops on Windows where the
separator is backslash ("\").
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
* adding support for execution in docker
* adding test for virtual code
* executing virtual only on linux
* chore(deps): bump actions/upload-artifact in the workflows group
Bumps the workflows group with 1 update: [actions/upload-artifact](https://github.com/actions/upload-artifact).
Updates `actions/upload-artifact` from 4 to 5
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v4...v5)
---
updated-dependencies:
- dependency-name: actions/upload-artifact
dependency-version: '5'
dependency-type: direct:production
update-type: version-update:semver-major
dependency-group: workflows
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): bump the modules group with 5 updates (#6551)
Bumps the modules group with 5 updates:
| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.13` | `0.4.14` |
| [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.1` | `0.8.2` |
| [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.26` | `0.1.27` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.51` | `0.2.52` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.6` | `1.2.7` |
Updates `github.com/projectdiscovery/fastdialer` from 0.4.13 to 0.4.14
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.13...v0.4.14)
Updates `github.com/projectdiscovery/dsl` from 0.8.1 to 0.8.2
- [Release notes](https://github.com/projectdiscovery/dsl/releases)
- [Commits](https://github.com/projectdiscovery/dsl/compare/v0.8.1...v0.8.2)
Updates `github.com/projectdiscovery/networkpolicy` from 0.1.26 to 0.1.27
- [Release notes](https://github.com/projectdiscovery/networkpolicy/releases)
- [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.26...v0.1.27)
Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.51 to 0.2.52
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.51...v0.2.52)
Updates `github.com/projectdiscovery/cdncheck` from 1.2.6 to 1.2.7
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.6...v1.2.7)
---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/fastdialer
dependency-version: 0.4.14
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/dsl
dependency-version: 0.8.2
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/networkpolicy
dependency-version: 0.1.27
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
dependency-version: 0.2.52
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
dependency-version: 1.2.7
dependency-type: indirect
update-type: version-update:semver-patch
dependency-group: modules
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fixing tests
* adding virtual python
* adding xpath + json extractors
* adding tests
* chore: satisfy lints
Signed-off-by: Dwi Siswanto <git@dw1.io>
* using clone options for auth store
* fix(headless): fixed memory leak issue during page initialization (#6569)
* fix(headless): fixed memory leak issue during page initialization
* fix(headless): typo fix and added comment
* fix(headless): one more typo fix
* feat: best-effort keyboard-interactive support for SSH
* fix: provide answer only when asked for
* fix: add logging
* feat(js): enhance SSH keyboard interactive auth
by:
* implement regex-based prompt matching for
password variants.
* add support for filling username prompts in
keyboard interactive challenges.
* improve debug logging with structured output.
this addresses issues with servers using
non-standard prompt formats and provides better
visibility into auth failures.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(js): migrate `github.com/go-pg/pg` => `github.com/go-pg/pg/v10`
Signed-off-by: Dwi Siswanto <git@dw1.io>
* feat(templates): add file metadata fields to `parsedTemplate` (#6534)
* feat(templates): add file metadata fields to `parsedTemplate`
to track template file information for cache
validation purposes.
closes #6515.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(templates): satisfy lints
Signed-off-by: Dwi Siswanto <git@dw1.io>
---------
Signed-off-by: Dwi Siswanto <git@dw1.io>
* chore(deps): bump the modules group with 7 updates
Bumps the modules group with 7 updates:
| Package | From | To |
| --- | --- | --- |
| [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.14` | `0.4.15` |
| [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.129` | `1.0.130` |
| [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.2` | `0.8.3` |
| [github.com/projectdiscovery/mapcidr](https://github.com/projectdiscovery/mapcidr) | `1.1.96` | `1.1.97` |
| [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1-0.20251022145046-e013dc9c5bed` | `0.6.1-0.20251030144701-ce5c4b44e1e6` |
| [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.52` | `0.2.53` |
| [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.7` | `1.2.8` |
Updates `github.com/projectdiscovery/fastdialer` from 0.4.14 to 0.4.15
- [Release notes](https://github.com/projectdiscovery/fastdialer/releases)
- [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.14...v0.4.15)
Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.129 to 1.0.130
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.129...v1.0.130)
Updates `github.com/projectdiscovery/dsl` from 0.8.2 to 0.8.3
- [Release notes](https://github.com/projectdiscovery/dsl/releases)
- [Commits](https://github.com/projectdiscovery/dsl/compare/v0.8.2...v0.8.3)
Updates `github.com/projectdiscovery/mapcidr` from 1.1.96 to 1.1.97
- [Release notes](https://github.com/projectdiscovery/mapcidr/releases)
- [Changelog](https://github.com/projectdiscovery/mapcidr/blob/main/.goreleaser.yml)
- [Commits](https://github.com/projectdiscovery/mapcidr/compare/v1.1.96...v1.1.97)
Updates `github.com/projectdiscovery/utils` from 0.6.1-0.20251022145046-e013dc9c5bed to 0.6.1-0.20251030144701-ce5c4b44e1e6
- [Release notes](https://github.com/projectdiscovery/utils/releases)
- [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md)
- [Commits](https://github.com/projectdiscovery/utils/commits)
Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.52 to 0.2.53
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.52...v0.2.53)
Updates `github.com/projectdiscovery/cdncheck` from 1.2.7 to 1.2.8
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.7...v1.2.8)
---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/fastdialer
dependency-version: 0.4.15
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/retryablehttp-go
dependency-version: 1.0.130
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/dsl
dependency-version: 0.8.3
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/mapcidr
dependency-version: 1.1.97
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/utils
dependency-version: 0.6.1-0.20251030144701-ce5c4b44e1e6
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/wappalyzergo
dependency-version: 0.2.53
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/cdncheck
dependency-version: 1.2.8
dependency-type: indirect
update-type: version-update:semver-patch
dependency-group: modules
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix(templates): mem leaks in parser cache
Fixes duplicate template storage & removes
unnecessary raw bytes caching.
Mem usage reduced by ~30%.
> 423MB => 299MB heap alloc.
* Use `StoreWithoutRaw()` to avoid storing raw
bytes.
* Remove duplicate storage in both caches.
* Remove ineffective raw bytes retrieval logic.
Benchmarks show 45% perf improvement with no
regressions.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* fix(http): resolve timeout config issues (#6562)
across multiple layers
Fixes timeout configuration conflicts where HTTP
requests would timeout prematurely despite
configured values in `@timeout` annotations or
`-timeout` flags.
RCA:
* `retryablehttp` pkg overriding with default
30s timeout.
* Custom timeouts not propagating to
`retryablehttp` layer.
* Multiple timeout layers not sync properly.
Changes:
* Propagate custom timeouts from `@timeout`
annotations to `retryablehttp` layer.
* Adjust 5-minute maximum cap to prevent DoS via
extremely large timeouts.
* Ensure `retryableHttpOptions.Timeout` respects
`ResponseHeaderTimeout`.
* Add comprehensive tests for timeout capping
behavior.
This allows templates to override global timeout
via `@timeout` annotations while preventing abuse
thru unreasonably large timeout values.
Fixes #6560.
Signed-off-by: Dwi Siswanto <git@dw1.io>
* add env variable for nuclei tempaltes dir
* chore(deps): bump github.com/opencontainers/runc
Bumps the go_modules group with 1 update in the / directory: [github.com/opencontainers/runc](https://github.com/opencontainers/runc).
Updates `github.com/opencontainers/runc` from 1.2.3 to 1.2.8
- [Release notes](https://github.com/opencontainers/runc/releases)
- [Changelog](https://github.com/opencontainers/runc/blob/v1.2.8/CHANGELOG.md)
- [Commits](https://github.com/opencontainers/runc/compare/v1.2.3...v1.2.8)
---
updated-dependencies:
- dependency-name: github.com/opencontainers/runc
dependency-version: 1.2.8
dependency-type: indirect
dependency-group: go_modules
...
Signed-off-by: dependabot[bot] <support@github.com>
* adding env tests on linux
* docs: update syntax & JSON schema 🤖
* chore(deps): bump the modules group with 4 updates
Bumps the modules group with 4 updates: [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go), [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl), [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) and [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck).
Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.130 to 1.0.131
- [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases)
- [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.130...v1.0.131)
Updates `github.com/projectdiscovery/dsl` from 0.8.3 to 0.8.4
- [Release notes](https://github.com/projectdiscovery/dsl/releases)
- [Commits](https://github.com/projectdiscovery/dsl/compare/v0.8.3...v0.8.4)
Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.53 to 0.2.54
- [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases)
- [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.53...v0.2.54)
Updates `github.com/projectdiscovery/cdncheck` from 1.2.8 to 1.2.9
- [Release notes](https://github.com/projectdiscovery/cdncheck/releases)
- [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.8...v1.2.9)
---
updated-dependencies:
- dependency-name: github.com/projectdiscovery/retryablehttp-go
dependency-version: 1.0.131
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: modules
- dependency-name: github.com/projectdiscovery/dsl
depe…
4 participants
Proposed changes
Checklist
Summary by CodeRabbit
New Features
Performance
Bug Fixes
Tests