feat: loading templates performance improvements

[Ice3man543]

Proposed changes

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Summary by CodeRabbit

• Refactor

  • Improved validation performance and efficiency by introducing shared validator instances, reducing repeated validator creation across multiple components.
  • Updated template parsing logic to optimize memory usage and streamline processing for local YAML and JSON files.
  • Simplified matcher validation by switching to a reflection-based approach, eliminating unnecessary data marshaling.

• Chores

  • Added new internal variables to support shared validator instances.

[coderabbitai]

Caution

Review failed

The pull request is closed.

Walkthrough

A singleton pattern for validator instances is introduced across multiple packages to prevent repeated instantiation and struct cache rebuilding. Validator initialization is centralized in new global variables for both options and templates. Additionally, template parsing logic is refactored to optimize memory usage and preprocessing, and matcher validation is rewritten to use reflection instead of YAML marshaling.

Changes

Cohort / File(s)	Change Summary
Validator Singleton Initialization internal/runner/options.go, pkg/templates/validator_singleton.go	Introduced global singleton validator variables (validateOptions, tplValidator) to ensure a single validator instance is used throughout respective packages.
Template Validation Refactor pkg/templates/templates.go	Updated all template validation logic to use the shared tplValidator instance, removing direct instantiation and redundant validator imports.
Template Parsing Optimization pkg/templates/parser.go	Refactored template parsing to conditionally read and preprocess data only for local YAML files, reduced memory usage by no longer caching raw byte data, and adjusted unmarshaling logic for different formats.
Matcher Validation Refactor pkg/operators/matchers/validate.go	Replaced YAML marshaling/unmarshaling with a reflection-based approach for extracting non-zero struct fields for validation, simplifying and optimizing the validation logic.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant TemplateParser
    participant TemplateValidator (tplValidator)
    participant MatcherValidator
    participant OptionsValidator (validateOptions)

    User->>TemplateParser: Parse template file (YAML/JSON)
    TemplateParser->>TemplateValidator: Validate template struct (uses tplValidator)
    TemplateValidator-->>TemplateParser: Validation result

    User->>MatcherValidator: Validate Matcher struct
    MatcherValidator->>MatcherValidator: Use reflection to extract fields
    MatcherValidator-->>User: Validation result

    User->>OptionsValidator: Validate options
    OptionsValidator-->>User: Validation result

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~15 minutes

Possibly related PRs

• feat: loading templates performance improvements #6364: Introduces a global singleton validator instance in internal/runner/options.go and modifies ValidateOptions to use this instance, directly mirroring the changes made here.

Suggested reviewers

• dwisiswant0

Poem

In fields of code where rabbits hop,
Singleton validators now never stop.
Reflection replaces YAML’s old dance,
Parsers read smartly, given the chance.
With memory saved and logic refined,
This bunny’s delighted with changes combined!
🐇✨

Note

⚡️ Unit Test Generation is now available in beta!

Learn more here, or try it out under "Finishing Touches" below.

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 3d7f995 and 1b6ae44.

📒 Files selected for processing (2)

• internal/runner/options.go (2 hunks)
• pkg/templates/parser.go (3 hunks)

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch loading-performance-improvements-v2

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai generate unit tests to generate unit tests for this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[dwisiswant0]

@coderabbitai list out the references that call the method/func updated by this PR.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

I'll help you find all the references to the methods and functions updated by this PR. Let me search through the codebase to identify where these updated components are called.

---

🏁 Script executed:

#!/bin/bash
echo "=== Searching for ValidateOptions function calls ==="
rg "ValidateOptions" -A 2 -B 2

echo -e "\n=== Searching for Matcher.Validate() method calls ==="
rg "\.Validate\(\)" -A 2 -B 2 --type go

echo -e "\n=== Searching for Template marshal/unmarshal method usage ==="
rg "(MarshalYAML|UnmarshalYAML|MarshalJSON|UnmarshalJSON)" -A 2 -B 2 --type go

echo -e "\n=== Searching for template parsing related calls ==="
rg "ParseTemplate|parseTemplate" -A 2 -B 2 --type go

echo -e "\n=== Searching for validator usage patterns ==="
rg "validator\.New\(\)|validate\.New\(\)" -A 1 -B 1 --type go

Length of output: 43006

---

@dwisiswant0 Here are all the references that call the methods/functions updated by this PR:

1. ValidateOptions Function References

Updated in: internal/runner/options.go (now uses singleton validateOptions validator)

Called from:

• lib/sdk_private.go:35 - runner.ValidateOptions(e.opts) (SDK initialization)
• internal/runner/options.go:126 - ValidateOptions(options) (program startup validation)

2. Matcher.Validate() Method References

Updated in: pkg/operators/matchers/validate.go (now uses reflection instead of YAML marshaling)

Called from:

• pkg/operators/matchers/compile.go:24 - matcher.Validate() (during matcher compilation)
• pkg/operators/matchers/validate_test.go - Multiple test calls:
  • Line 13: m.Validate()
  • Line 16: err = m.Validate()
  • Line 20: err = m.Validate()
  • Line 23: err = m.Validate()
  • Line 26: err = m.Validate()

3. Template Marshal/Unmarshal Methods References

Updated in: pkg/templates/templates.go (now uses singleton tplValidator)

These methods (MarshalYAML, UnmarshalYAML, MarshalJSON, UnmarshalJSON) are called automatically by Go's serialization system during:

• Template caching operations
• Template serialization/deserialization
• YAML/JSON encoding/decoding of template structures

4. Template Parsing Function References

Updated in: pkg/templates/parser.go (optimized parsing logic)

ParseTemplate/parseTemplate called from:

• lib/sdk.go:89 - templates.ParseTemplateFromReader() (SDK template parsing)
• pkg/templates/compile.go:43 - ParseTemplateFromReader() (template compilation)
• pkg/templates/compile.go:90 - parseTemplate() (internal parsing)
• pkg/templates/compile.go:104 - parseTemplate() (signature verification)
• pkg/templates/compile.go:124 - parseTemplate() (reparse after preprocessing)
• pkg/templates/template_sign.go:61 - ParseTemplateFromReader() (template signing)
• pkg/templates/parser.go:58 - ParseTemplate() (template loading)
• pkg/templates/parser.go:87 - ParseTemplate() (workflow loading)
• internal/runner/templates.go:26 - ParseTemplate() (verbose logging)
• cmd/tmc/main.go:157 - parseTemplate() (TMC command)
• cmd/tmc/main.go:174 - ParseTemplateFromReader() (TMC template parsing)

5. Validator Singleton Usage

New singletons created:

• pkg/templates/validator_singleton.go - tplValidator (used by Template methods)
• internal/runner/options.go - validateOptions (used by ValidateOptions function)

These performance improvements reduce the overhead of creating new validator instances for each validation operation, which should result in faster template loading and validation across the entire Nuclei codebase.

🐰✨

[dwisiswant0]

Noice! Got perf optimized up to 45.83% on mem, and around 1.84–2.05x faster speed.

diff:

• dev
  • https://flamegraph.com/share/6b6515cf-6f82-11f0-9ac4-4e554e3cebd6
  • https://flamegraph.com/share/6a57d60e-6f82-11f0-bb22-8ac84e7fe810
• this
  • https://flamegraph.com/share/33cc0557-6f8e-11f0-9ac4-4e554e3cebd6
  • https://flamegraph.com/share/326c53dc-6f8e-11f0-9ac4-4e554e3cebd6

[ehsandeep]

merge conflict ^