feat: fixed max-host-error blocking + progress mismatch + misc#6193
feat: fixed max-host-error blocking + progress mismatch + misc#6193
Conversation
WalkthroughThis set of changes introduces enhancements to request tracking, result counting, error handling, and progress reporting across multiple components. Timing and summary logging are improved in the enumeration runner. Output writers now track and expose the number of results written. The HTTP request generator gains accurate methods for total and remaining requests, with corresponding changes in request counting logic. Flow execution now tracks executed requests and reconciles progress for unexecuted ones. Host error caching and context key extraction are refined. Minor improvements are made to variable scoping, error handling, and progress counter updates. Changes
Sequence Diagram(s)sequenceDiagram
participant Runner
participant ProgressBar
participant Enumeration
participant OutputWriter
Runner->>ProgressBar: Start()
Runner->>Enumeration: RunEnumeration()
Enumeration->>OutputWriter: Write(result)
OutputWriter->>OutputWriter: Increment resultCount()
Enumeration-->>Runner: Done
Runner->>ProgressBar: Stop()
Runner->>OutputWriter: ResultCount()
Runner->>Runner: Log summary with duration and result count
sequenceDiagram
participant FlowExecutor
participant Protocol
participant Request
participant Progress
FlowExecutor->>Protocol: Iterate requests
Protocol->>Request: Execute (with id)
Request-->>FlowExecutor: Execution complete
FlowExecutor->>FlowExecutor: Mark request as executed
FlowExecutor->>FlowExecutor: After all, reconcileProgress()
FlowExecutor->>Progress: Mark unexecuted requests as finished
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (3)
⏰ Context from checks skipped due to timeout of 90000ms (3)
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (1)
pkg/tmplexec/flow/flow_executor.go (1)
263-273: Progress reconciliation logic is well-implementedThe method correctly identifies unexecuted requests and updates the progress tracker to maintain accurate statistics. Consider enhancing the comment to better explain the purpose of this reconciliation.
- // never executed → pretend it finished so that stats match + // For requests that weren't executed, update the progress counter + // to reflect completion. This ensures progress statistics remain + // accurate by accounting for requests that were skipped or not run.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (14)
internal/runner/runner.go(2 hunks)pkg/core/execute_options.go(1 hunks)pkg/output/multi_writer.go(1 hunks)pkg/output/output.go(4 hunks)pkg/output/output_stats.go(1 hunks)pkg/progress/progress.go(1 hunks)pkg/protocols/common/hosterrorscache/hosterrorscache.go(2 hunks)pkg/protocols/http/build_request.go(0 hunks)pkg/protocols/http/http.go(1 hunks)pkg/protocols/http/request.go(3 hunks)pkg/protocols/http/request_generator.go(1 hunks)pkg/testutils/testutils.go(1 hunks)pkg/tmplexec/flow/flow_executor.go(4 hunks)pkg/tmplexec/flow/flow_internal.go(2 hunks)
💤 Files with no reviewable changes (1)
- pkg/protocols/http/build_request.go
🧰 Additional context used
🧬 Code Graph Analysis (4)
pkg/tmplexec/flow/flow_executor.go (1)
pkg/progress/progress.go (1)
Progress(17-35)
pkg/protocols/http/request_generator.go (2)
pkg/fuzz/dataformat/raw.go (1)
Raw(3-3)pkg/protocols/http/request_annotations.go (1)
Once(46-46)
pkg/protocols/common/hosterrorscache/hosterrorscache.go (1)
pkg/protocols/common/contextargs/metainput.go (1)
MetaInput(18-30)
pkg/protocols/http/request.go (1)
pkg/progress/progress.go (1)
Progress(17-35)
🪛 GitHub Check: Lint
pkg/tmplexec/flow/flow_internal.go
[failure] 79-79:
Error return value of f.executed.Set is not checked (errcheck)
🔇 Additional comments (20)
pkg/progress/progress.go (1)
122-124: Improved request counting behaviorThe implementation of
SetRequestsnow directly increments the counter by the provided count value, which simplifies the logic and improves consistency with other progress tracking components in the codebase.pkg/output/output_stats.go (1)
52-54: Implementation of ResultCount interface methodThis correctly implements the Writer interface by adding the ResultCount method to StatsOutputWriter. Returning 0 is appropriate since this writer doesn't store actual results but only tracks statistics.
pkg/testutils/testutils.go (1)
136-138: Mock implementation of ResultCountThis properly implements the ResultCount method for testing purposes, maintaining consistency with the interface changes in the output package.
pkg/output/multi_writer.go (1)
69-77: Implement result counting for MultiWriterThe implementation correctly aggregates result counts from all writers, returning the first positive count found. This supports the request tracking system being implemented across the codebase.
pkg/tmplexec/flow/flow_internal.go (1)
91-96: Well-structured helper function for request identificationThe
requestKeyfunction properly handles request identification by combining protocol name and request ID. This supports the new functionality for tracking executed requests.internal/runner/runner.go (2)
705-705: Good practice: Capture start time for performance trackingCapturing the start time right before execution allows for accurate time tracking of the scan duration.
729-738: Improved scan completion reportingThe code now:
- Explicitly stops the progress bar before printing the final message
- Calculates and displays the scan duration in minutes
- Shows the number of matches found when results exist
This provides better feedback to users about the scan results.
pkg/protocols/http/http.go (1)
519-521: Simplified request counting logicThe code now uses the generator's
Total()method directly to get the count of requests, instead of complex conditional logic. This makes the code more maintainable and ensures consistent request counting across the codebase.pkg/output/output.go (4)
57-58: Interface extension looks good!Adding the
ResultCount()method to theWriterinterface is a clean way to expose result counting functionality to all implementors.
85-85: Good use of atomic for thread safetyUsing
atomic.Int32is the right choice for thread-safe counter operations in a concurrent environment.
294-296: Method implementation looks correctThe implementation properly uses atomic loading to retrieve the counter value in a thread-safe manner.
347-347: Atomic increment in the right placeThe counter is incremented after successful writing operations, ensuring accurate counting of actual written results.
pkg/tmplexec/flow/flow_executor.go (3)
55-55: Good addition for tracking executed requestsUsing a
SyncLockMapwith an empty struct value is a memory-efficient way to track executed requests in a concurrent environment.
103-103: Proper initialization in constructorInitializing the executed map in the constructor ensures it's never nil when accessed later.
249-249: Progress reconciliation at the right placeCalling
reconcileProgress()after program execution but before error handling ensures accurate progress reporting regardless of execution outcome.pkg/protocols/common/hosterrorscache/hosterrorscache.go (2)
92-94: Performance optimization for host:port formatEarly return for strings containing a colon avoids unnecessary URL parsing for values that likely represent host:port formats, reducing processing overhead.
242-254: Improved address priority logicThe refactored code now correctly prioritizes the error-specific address before falling back to the context's meta input address. This ensures that errors are tracked with the most accurate address information, leading to more precise error handling, especially for templates with errors.
pkg/protocols/http/request_generator.go (2)
139-173: Well-implemented Remaining() methodThe method accurately calculates remaining requests by accounting for the current position, request markers, and remaining payload iterations. The logic correctly handles special cases like
@oncemarked requests.
175-201: Total() method handles edge cases properlyThe implementation correctly calculates the total number of requests, properly accounting for requests with the
@oncemarker that should only be executed once regardless of payload iterations.pkg/protocols/http/request.go (1)
546-549: Better error wrapping using errkit instead of errorutil.Replaced the previous error wrapping method with
errkit.FromError, which provides improved error handling capabilities, better structure, and more detailed context about the error. The formatted error message now includes the specific URL that caused the issue.
Ice3man543
changed the title
3 participants
Proposed changes
Reference - #5633 (Added in Nuclei v3.3.3)
Reference - #5668 (Added in Nuclei v3.3.4)
Changes Summary
Checklist
Summary by CodeRabbit
New Features
Bug Fixes
Refactor