feat(fuzz): eval variables

[dwisiswant0]

Proposed changes

Closes #6333

Proof

Template:

# dast-issue-6333.yaml
id: dast-issue-6333

info:
  name: DAST Issue projectdiscovery/nuclei#6333
  author: dwisiswant0
  severity: info
  tags: dast,test

variables:
  foo_var: "foo_var_value"

http:
  - payloads:
      foo_payload:
        - "foo_payload_value"

    fuzzing:
      - parts: [query, body, header]
        type: replace
        mode: single
        keys: ["{{foo_var}}"]
        fuzz: ["{{foo_payload}}"]

      - parts: [query, body, header]
        type: replace
        mode: single
        keys-regex: ["^{{foo_var}}"]
        fuzz: ["{{foo_payload}}"]

    stop-at-first-match: true
    matchers-condition: and
    matchers:
      - type: status
        status:
          - 200
          - 204
          - 301
          - 302
          - 303
          - 307
          - 308

Result:

$ make build
$ ./bin/nuclei -u "http://scanme.sh/?foo_var_value=bar" -dast -t dast-issue-6333.yaml -silent
[dast-issue-6333] [http] [info] http://scanme.sh/?foo_var_value=foo_payload_value [query:foo_var_value] [GET]
[dast-issue-6333] [http] [info] http://scanme.sh/?foo_var_value=foo_payload_value [query:foo_var_value] [GET]

Tests:

$ go test -v -timeout 30s -run ^TestEvaluateVariables$ github.com/projectdiscovery/nuclei/v3/pkg/fuzz
=== RUN   TestEvaluateVariables
=== RUN   TestEvaluateVariables/keys
=== RUN   TestEvaluateVariables/keys-regex
=== RUN   TestEvaluateVariables/values-regex
=== RUN   TestEvaluateVariables/complex-variables
=== RUN   TestEvaluateVariables/invalid-variables
=== RUN   TestEvaluateVariables/evaluateVars-function
--- PASS: TestEvaluateVariables (0.00s)
    --- PASS: TestEvaluateVariables/keys (0.00s)
    --- PASS: TestEvaluateVariables/keys-regex (0.00s)
    --- PASS: TestEvaluateVariables/values-regex (0.00s)
    --- PASS: TestEvaluateVariables/complex-variables (0.00s)
    --- PASS: TestEvaluateVariables/invalid-variables (0.00s)
    --- PASS: TestEvaluateVariables/evaluateVars-function (0.00s)
PASS
ok  	github.com/projectdiscovery/nuclei/v3/pkg/fuzz	0.070s

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Summary by CodeRabbit

• New Features

  • Added support for evaluating and resolving variables within rule keys and regex patterns, enabling dynamic substitution of template variables, constants, and runtime variables before rule compilation.

• Bug Fixes

  • Improved error handling by preventing compilation of rules with unresolved or invalid variable expressions.

• Tests

  • Introduced comprehensive tests to verify correct variable evaluation and matching in rule keys and regex patterns.

• Style

  • Refined log message formatting for clearer and more concise output during fuzzing rule execution.

[coderabbitai]

Walkthrough

A new method for variable evaluation within rule strings is introduced and integrated into the rule compilation process, ensuring variables in keys and regex patterns are resolved before use. Comprehensive tests verify correct variable handling. Minor logging and error message formatting changes are also applied to the HTTP request fuzzing logic.

Changes

Cohort / File(s)	Change Summary
Variable Evaluation in Rules pkg/fuzz/execute.go	Adds evaluateVars method to Rule for resolving variables in strings; updates Compile to use this for keys, keys-regex, values-regex, and replace-regex fields; minor error formatting fix; adds marker import.
Variable Evaluation Tests pkg/fuzz/fuzz_test.go	Adds TestEvaluateVariables function to thoroughly test variable evaluation and matching in rules, covering various scenarios and error handling.
Logging and Error Formatting pkg/protocols/http/request_fuzz.go	Refines log message formatting for fuzzing rule execution, removes redundant phrases, and simplifies error handling for non-applicable rules.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant Rule
    participant ExecutorOptions
    participant PayloadGenerator

    User->>Rule: Compile(generator, options)
    Rule->>Rule: evaluateVars(Keys, KeysRegex, ValuesRegex, ReplaceRegex)
    Rule->>ExecutorOptions: Access variables/constants/options
    Rule->>Rule: Substitute variables in strings
    Rule-->>User: Compiled rule or error

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~15 minutes

Assessment against linked issues

Objective	Addressed	Explanation
Evaluate variables in keys and keys-regex during fuzzing rule compilation and matching (#6333)	✅
Return errors or handle unresolved variables in keys/regex (#6333)	✅
Update logging to clarify rule applicability errors (#6333)	✅

Assessment against linked issues: Out-of-scope changes

No out-of-scope changes found.

Poem

In the warren where variables hide,
The rules now search both far and wide.
With regex keys and values too,
Each template now gets its due.
Tests abound, logs refined—
Fuzzy bunnies, peace of mind!
🐇✨

Note

⚡️ Unit Test Generation is now available in beta!

Learn more here, or try it out under "Finishing Touches" below.

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 91adfeb and db5ade2.

📒 Files selected for processing (3)

• pkg/fuzz/execute.go (5 hunks)
• pkg/fuzz/fuzz_test.go (2 hunks)
• pkg/protocols/http/request_fuzz.go (4 hunks)

🧰 Additional context used

🧠 Learnings (3)

pkg/protocols/http/request_fuzz.go (1)

Learnt from: dwisiswant0
PR: #6290
File: pkg/fuzz/component/path_test.go:43-44
Timestamp: 2025-06-30T16:33:26.746Z
Learning: The user dwisiswant0 declined adding documentation for internal API changes in pkg/fuzz/component/path when the change was from nested URL field access (rebuilt.URL.Path, rebuilt.URL.String()) to direct field access (rebuilt.Path, rebuilt.String()), indicating they don't consider additional documentation necessary for such internal API modifications.

pkg/fuzz/fuzz_test.go (2)

Learnt from: hdm
PR: #6322
File: pkg/protocols/common/protocolstate/file.go:38-55
Timestamp: 2025-07-16T21:31:43.283Z
Learning: In pkg/protocols/common/protocolstate/file.go, the inconsistent behavior between SetLfaAllowed, GetLfaAllowed, and IsLfaAllowed functions is intentional as an interim step before implementing context-specific LFA logic. The current mixed approach of global and per-instance flags is a temporary transitional state, not a design flaw.

Learnt from: hdm
PR: #6322
File: pkg/templates/compile.go:79-81
Timestamp: 2025-07-16T21:27:14.937Z
Learning: To make the template caching mechanism in pkg/templates/compile.go production-ready, DSLs need to be updated to use runtime options instead of cached variables, rather than restoring the Compile() calls on each request.

pkg/fuzz/execute.go (1)

Learnt from: hdm
PR: #6322
File: pkg/templates/compile.go:79-81
Timestamp: 2025-07-16T21:27:14.937Z
Learning: To make the template caching mechanism in pkg/templates/compile.go production-ready, DSLs need to be updated to use runtime options instead of cached variables, rather than restoring the Compile() calls on each request.

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)

• GitHub Check: Tests (macOS-latest)
• GitHub Check: Tests (ubuntu-latest)
• GitHub Check: Tests (windows-latest)

🔇 Additional comments (19)

pkg/protocols/http/request_fuzz.go (5)

79-79: LGTM: Logging message formatting improvement.

The removal of extra space before the colon improves consistency in log message formatting.

---

106-106: LGTM: Consistent log message formatting.

The formatting change aligns with the other log messages in the file for consistency.

---

112-112: LGTM: Log message formatting consistency.

The change maintains uniform formatting across all log messages.

---

161-161: LGTM: Consistent logging format.

The formatting change maintains consistency with other log messages in the codebase.

---

171-173: LGTM: Simplified error handling with consistent formatting.

The change from fuzz.ErrRuleNotApplicable.Msgf to fmt.Errorf simplifies the error creation while maintaining the same semantic meaning. The message content remains identical, ensuring backward compatibility for error handling logic that checks error message content.

pkg/fuzz/fuzz_test.go (7)

6-12: LGTM: Proper imports for comprehensive testing.

The added imports provide all necessary dependencies for testing variable evaluation functionality, including goflags, protocols, variables, types, and utils packages.

---

46-84: LGTM: Comprehensive test for keys variable evaluation.

This test thoroughly validates that template variables in keys are properly evaluated during rule compilation. The test setup includes proper mock objects for template variables, constants, and runtime variables, and correctly verifies both positive (evaluated variable matches) and negative (unevaluated variable doesn't match) scenarios.

---

86-111: LGTM: Keys-regex variable evaluation test.

The test correctly validates that variables within regex patterns are evaluated before regex compilation. This addresses the core issue described in the PR objectives where variables in keys-regex were not being replaced.

---

113-138: LGTM: Values-regex variable evaluation test.

This test ensures that variables in values regex patterns are properly evaluated, completing the coverage for all regex-based matching scenarios.

---

140-181: LGTM: Complex variables test with comprehensive coverage.

Excellent test that validates the integration of all three variable types (template variables, constants, and runtime variables) in a single scenario. This ensures the variable precedence and merging logic works correctly.

---

183-204: LGTM: Proper error handling test for invalid variables.

The test correctly handles the scenario where variables cannot be resolved, providing appropriate error checking. The conditional error handling accounts for different implementation behaviors while ensuring correctness.

---

206-260: LGTM: Direct testing of evaluateVars function.

This test provides excellent coverage of the evaluateVars method directly, testing various scenarios including simple variables, constants, runtime variables, mixed content, and error conditions. The test structure is thorough and follows best practices.

pkg/fuzz/execute.go (7)

17-17: LGTM: Import addition for expression parsing.

The import of the marker package provides the necessary delimiters for expression parsing in the new evaluateVars method.

---

27-27: LGTM: Error message formatting consistency.

The removal of extra space before the colon aligns with the formatting changes in the request_fuzz.go file.

---

193-218: LGTM: Well-implemented variable evaluation method.

The evaluateVars method is correctly implemented with the following strengths:

1. Proper variable merging: Combines template variables, constants, and runtime options in the correct precedence order
2. Expression handling: Uses the existing expression evaluation infrastructure with proper delimiters
3. Error handling: Checks for unresolved variables before evaluation and returns appropriate errors
4. Null safety: Guards against nil options with early return

The implementation follows established patterns in the codebase and addresses the core requirement from issue #6333.

---

373-381: LGTM: Variable evaluation integration for keys.

The integration correctly applies variable evaluation to keys before storing them in the keysMap. The error handling ensures that compilation fails if variables cannot be resolved, preventing runtime issues.

---

384-396: LGTM: Variable evaluation for values regex.

Proper integration of variable evaluation for values regex patterns. The evaluation occurs before regex compilation, ensuring that variables are resolved when the regex is used for matching.

---

398-411: LGTM: Variable evaluation for keys regex.

This integration addresses the specific issue mentioned in the PR objectives where variables in keys-regex were not being replaced. The implementation ensures variables are evaluated before regex compilation.

---

422-433: LGTM: Variable evaluation for replace regex.

Consistent application of variable evaluation to replace regex patterns, completing the coverage for all regex-based rule components.

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch dwisiswant0/feat/fuzz/eval-variables

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai generate unit tests to generate unit tests for this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.