Add option to control number of concurrent templates loaded on startup#6373
Add option to control number of concurrent templates loaded on startup#6373Mzack9999 merged 4 commits intoprojectdiscovery:devfrom
Conversation
WalkthroughAdded a CLI flag --template-loading-concurrency (-tlc) and Options.TemplateLoadingConcurrency (default 50). The template loader now uses that configured value (falling back to default when non-positive) instead of a previously hardcoded concurrency of 50. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant User
participant CLI
participant Options
participant Store
participant Loader
User->>CLI: run with --template-loading-concurrency=N
CLI->>Options: set TemplateLoadingConcurrency = N
CLI->>Store: initialize with ExecutorOptions(Options)
Store->>Loader: LoadTemplatesWithTags(concurrency = Options.TemplateLoadingConcurrency or Default)
Loader->>Loader: spawn up to configured concurrent template loaders
Loader-->>Store: return load results
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (6)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🔭 Outside diff range comments (1)
pkg/types/types.go (1)
468-678: Options.Copy() omits TemplateLoadingConcurrency (risk of silent reset to zero)Copies created via Options.Copy() will drop this value, potentially causing the loader to see 0 and panic on syncutil.New. Include the new field in the copy.
Apply this diff inside the optCopy literal:
@@ - PayloadConcurrency: options.PayloadConcurrency, - ProbeConcurrency: options.ProbeConcurrency, + PayloadConcurrency: options.PayloadConcurrency, + ProbeConcurrency: options.ProbeConcurrency, + TemplateLoadingConcurrency: options.TemplateLoadingConcurrency,
🧹 Nitpick comments (4)
pkg/types/types.go (2)
773-790: Unify the default across CLI and library via a shared constantDefaultOptions sets TemplateLoadingConcurrency: 50 while the CLI also hardcodes 50. To avoid drift, use a shared constant here and in the CLI flag.
Apply this diff:
return &Options{ - RateLimit: 150, - RateLimitDuration: time.Second, - BulkSize: 25, - TemplateThreads: 25, - HeadlessBulkSize: 10, - PayloadConcurrency: 25, - HeadlessTemplateThreads: 10, - ProbeConcurrency: 50, - TemplateLoadingConcurrency: 50, - Timeout: 5, - Retries: 1, - MaxHostError: 30, - ResponseReadSize: 10 * unitutils.Mega, - ResponseSaveSize: unitutils.Mega, + RateLimit: 150, + RateLimitDuration: time.Second, + BulkSize: 25, + TemplateThreads: 25, + HeadlessBulkSize: 10, + PayloadConcurrency: 25, + HeadlessTemplateThreads: 10, + ProbeConcurrency: 50, + TemplateLoadingConcurrency: DefaultTemplateLoadingConcurrency, + Timeout: 5, + Retries: 1, + MaxHostError: 30, + ResponseReadSize: 10 * unitutils.Mega, + ResponseSaveSize: unitutils.Mega, }
1-899: Optional: Add a minimal validator/normalizer for OptionsIf there’s an existing normalization step for Options, ensure TemplateLoadingConcurrency is clamped to >= 1 there as well, complementing the loader-side guard.
pkg/catalog/loader/loader.go (1)
510-523: Optional: degrade panic to warning + serial fallbackCurrent behavior panics on wg creation and on missing dialers. Consider logging an error and falling back to serial processing (concurrency=1) rather than exiting the process, to improve robustness during edge misconfigurations.
cmd/nuclei/main.go (1)
400-412: Optional: validate non-negative in CLIIf goflags supports int range validation, enforce a minimum of 1 at parse time. Otherwise, loader-side clamping (suggested) will handle it.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
cmd/nuclei/main.go(1 hunks)pkg/catalog/loader/loader.go(1 hunks)pkg/types/types.go(2 hunks)
🧰 Additional context used
🧠 Learnings (3)
📓 Common learnings
Learnt from: hdm
PR: projectdiscovery/nuclei#6322
File: pkg/templates/compile.go:79-81
Timestamp: 2025-07-16T21:27:14.937Z
Learning: To make the template caching mechanism in pkg/templates/compile.go production-ready, DSLs need to be updated to use runtime options instead of cached variables, rather than restoring the Compile() calls on each request.
📚 Learning: 2025-07-16T21:27:14.937Z
Learnt from: hdm
PR: projectdiscovery/nuclei#6322
File: pkg/templates/compile.go:79-81
Timestamp: 2025-07-16T21:27:14.937Z
Learning: To make the template caching mechanism in pkg/templates/compile.go production-ready, DSLs need to be updated to use runtime options instead of cached variables, rather than restoring the Compile() calls on each request.
Applied to files:
cmd/nuclei/main.gopkg/types/types.gopkg/catalog/loader/loader.go
📚 Learning: 2025-07-16T21:28:08.073Z
Learnt from: hdm
PR: projectdiscovery/nuclei#6322
File: pkg/templates/compile.go:59-78
Timestamp: 2025-07-16T21:28:08.073Z
Learning: The shallow copy behavior (`tplCopy := *value`) in pkg/templates/compile.go is intentional design for the template caching mechanism. The partial-copy approach is part of the performance optimization strategy, not a bug requiring deep copying.
Applied to files:
pkg/types/types.go
|
@CodeRabbit create an issue for this PR, tag it of type maintenance |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: Length of output: 44 🏁 Script executed: Length of output: 2403 🏁 Script executed: Length of output: 504 @Mzack9999: I've created issue #6518 for this PR and tagged it with "Type: Maintenance". The issue has been assigned to you and includes a link back to this PR. Issue: #6518 |
3 participants
Proposed changes
This PR adds a new command-line option
--template-loading-concurrency(short:-tlc) to control the number of concurrent template loading operations during startup. This enhancement provides users with fine-grained control over resource utilization during the template loading phase, allowing them to optimize performance based on their system capabilities and requirements.Changes made:
--template-loading-concurrencywith a default value of 50 concurrent operationspkg/catalog/loader/loader.goto use the configurable concurrency limit instead of the hardcoded value of 50Optionsstruct inpkg/types/types.goto include the newTemplateLoadingConcurrencyfieldDefaultOptions()function with a default value of 50This change maintains backward compatibility while providing additional control over Nuclei's startup performance characteristics.
Checklist
Summary by CodeRabbit
New Features
Documentation