per-host http client pool - #7301
Merged
Merged
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughPer-host HTTP client pooling and connection-tracking were added: client Get now accepts a host, transports collect new vs reused connection counts, pool eviction windows shortened, keep-alive disabling by scan-strategy removed, client acquisition moved to request execution, shutdown logs and benchmark suite added. Changes
Sequence Diagram(s)sequenceDiagram
participant R as Request
participant P as HTTPClientPool
participant C as retryablehttp.Client
participant T as Transport
participant S as TargetServer
participant A as Analyzer
R->>P: Get(options, config, host)
note right of P: select/create per-host client\nwrap transport with connTrackingTransport
P-->>C: return client
R->>C: Do(req)
C->>T: RoundTrip(req)
T->>S: TCP/TLS connect & send
S-->>T: response
T->>T: httptrace.GotConn -> increment New/Reused
T-->>C: response
C-->>R: response
R->>A: analyze(response, client=C)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~65 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (2)
pkg/protocols/common/automaticscan/automaticscan.go (1)
97-97: Disable cookie storage on the shared Wappalyzer client.This client is reused across every target, and
httpclientpool.Get(..., &Configuration{}, "")will give it a default jar.getTagsUsingWappalyzeronly does a stateless fingerprinting GET, so those cookies just retain cross-target state/memory for no benefit.DisableCookie: truelooks like the safer default here.Proposed change
- httpclient, err := httpclientpool.Get(opts.ExecuterOpts.Options, &httpclientpool.Configuration{}, "") + httpclient, err := httpclientpool.Get(opts.ExecuterOpts.Options, &httpclientpool.Configuration{ + DisableCookie: true, + }, "")🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/protocols/common/automaticscan/automaticscan.go` at line 97, The shared Wappalyzer HTTP client is created via httpclientpool.Get in getTagsUsingWappalyzer and currently receives a default jar; update the Configuration passed to httpclientpool.Get (the second argument) to set DisableCookie: true so the returned httpclient does not store cookies across targets (use opts.ExecuterOpts.Options and the existing Configuration struct but set DisableCookie to true) to ensure stateless fingerprinting and avoid cross-target cookie retention.pkg/protocols/http/httpclientpool/clientpool_benchmark_test.go (1)
101-118: Add at least one test that goes throughGet().
tracedClientbuilds standalonehttp.Clients, so this suite never exercisespkg/protocols/http/httpclientpool.Get, the host-keyed cache path, or the explicit-jar cache bypass added in this PR. As written, these are useful microbenchmarks, but they do not lock down the actual regression surface.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/protocols/http/httpclientpool/clientpool_benchmark_test.go` around lines 101 - 118, Add a unit test that actually calls the httpclientpool Get method so the host-keyed cache and explicit-jar bypass are exercised: create a pool (using the same factory/hooks your code exposes), use tracedClient (the connTrackingRoundTripper with newConns/reusedConns counters) as the underlying client factory, call pool.Get(host) twice and assert the second call exercises the cache (no newConns, increased reusedConns or same client pointer), then call pool.Get(host, explicitJar=true) (or the equivalent API) and assert it bypasses the cache (newConns increments or returns a different client). Reference tracedClient, connTrackingRoundTripper, newConns, reusedConns and Get in your test so it validates both the host-keyed cache path and the explicit-jar cache bypass.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@internal/runner/runner.go`:
- Around line 430-434: GetConnectionStats() returns package-global cumulative
counters, so per-scan logs in runner.go show aggregated totals across in-process
runs; update the httpclientpool usage to produce per-execution stats by either
calling a reset or using a scoped API: add/ call
httpclientpool.ResetConnectionStats() at the start (or end) of a run inside the
runner (around where GetConnectionStats() is invoked) or change to an API like
httpclientpool.GetConnectionStatsForExecution(executionId) /
httpclientpool.ScopedStats(executionId) and pass the run's ExecutionId so the
logged totals (from GetConnectionStats / new scoped method) reflect only the
current scan. Ensure you reference and modify the call site in runner.go where
GetConnectionStats() is used and wire in ExecutionId from the current Runner
context.
In `@pkg/protocols/http/httpclientpool/clientpool_benchmark_test.go`:
- Around line 215-218: The test currently enforces a hard performance threshold
by asserting require.Greater(t, speedup, 1.5) on the local speedup variable;
remove that flaky assertion and instead record the observed speedup (e.g.,
t.Logf or similar) so the test no longer fails on noisy CI—locate the speedup
variable and the require.Greater call in clientpool_benchmark_test.go and
replace the failing assertion with a non-fatal log of the speedup, relying on
the existing deterministic connection-count assertions to validate behavior.
In `@pkg/protocols/http/httpclientpool/clientpool.go`:
- Around line 220-244: wrappedGet currently keys the pool by
configuration.Hash() plus host but configuration.Hash() doesn't include
Connection.DisableKeepAlive, so clients with different DisableKeepAlive values
can collide; update the pool key construction in clientpool.go (the code around
wrappedGet / the local hash variable) to append the effective DisableKeepAlive
flag (e.g., strconv.FormatBool(disableKeepAlives) or "keepalive=off"/"on") to
the hash before calling dialers.HTTPClientPool.Get, ensuring the same unique
symbols (configuration.Hash(), host, disableKeepAlives) are used when creating
and retrieving the client so keep-alive behavior is correctly honored.
- Around line 51-67: The connTrackingTransport wrapper only implements
RoundTrip, so http.Client.CloseIdleConnections() doesn't reach the underlying
transport; add a CloseIdleConnections method on connTrackingTransport that
forwards the call to the wrapped transport when it supports it (assert t.base to
an interface with CloseIdleConnections and call it), ensuring no-op if the base
doesn't implement that method; reference connTrackingTransport, RoundTrip,
CloseIdleConnections and the base http.RoundTripper in your change.
In `@pkg/protocols/http/request.go`:
- Around line 1010-1015: The analyzer is being given a new client via
request.getHTTPClientForHost(hostname), which can drop per-request overrides
(cookie jar, WithCustomTimeout) applied when the request was executed; instead
pass the same HTTP client instance that executed the request (the local
httpclient/httpClient variable created earlier when cloning connConfig and
resolving the client) into analyzer.Analyze so follow-up requests use the same
session/timeout; replace the getHTTPClientForHost(hostname) call with the
executing client variable when invoking analyzer.Analyze.
- Around line 847-850: The pool key currently passed to httpclientpool.Get uses
the Host header override (hostname / generatedRequest.request.Host), which
collapses distinct connection targets; change the key to use the actual
connection target generatedRequest.request.URL.Host (or concatenate URL.Host +
generatedRequest.request.Host if you need both host-target and Host-header
isolation) when calling httpclientpool.Get (the call with
request.options.Options, connConfig, hostname) so each distinct target gets its
own pool entry and transports are not incorrectly reused across vhost/IP scans.
---
Nitpick comments:
In `@pkg/protocols/common/automaticscan/automaticscan.go`:
- Line 97: The shared Wappalyzer HTTP client is created via httpclientpool.Get
in getTagsUsingWappalyzer and currently receives a default jar; update the
Configuration passed to httpclientpool.Get (the second argument) to set
DisableCookie: true so the returned httpclient does not store cookies across
targets (use opts.ExecuterOpts.Options and the existing Configuration struct but
set DisableCookie to true) to ensure stateless fingerprinting and avoid
cross-target cookie retention.
In `@pkg/protocols/http/httpclientpool/clientpool_benchmark_test.go`:
- Around line 101-118: Add a unit test that actually calls the httpclientpool
Get method so the host-keyed cache and explicit-jar bypass are exercised: create
a pool (using the same factory/hooks your code exposes), use tracedClient (the
connTrackingRoundTripper with newConns/reusedConns counters) as the underlying
client factory, call pool.Get(host) twice and assert the second call exercises
the cache (no newConns, increased reusedConns or same client pointer), then call
pool.Get(host, explicitJar=true) (or the equivalent API) and assert it bypasses
the cache (newConns increments or returns a different client). Reference
tracedClient, connTrackingRoundTripper, newConns, reusedConns and Get in your
test so it validates both the host-keyed cache path and the explicit-jar cache
bypass.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 753306ea-3133-400d-8336-97eb7846a7b9
📒 Files selected for processing (11)
internal/runner/runner.golib/sdk_private.gopkg/protocols/common/automaticscan/automaticscan.gopkg/protocols/common/protocolstate/state.gopkg/protocols/http/build_request.gopkg/protocols/http/http.gopkg/protocols/http/httpclientpool/clientpool.gopkg/protocols/http/httpclientpool/clientpool_benchmark_test.gopkg/protocols/http/request.gopkg/protocols/utils/http/requtils.gopkg/tmplexec/exec.go
Neo - PR Security ReviewNo security issues found Highlights
Comment |
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
pkg/protocols/common/protocolstate/state.go (1)
283-289: LGTM! Proper cleanup sequence for idle connections.The order of operations is correct—closing idle connections and clearing the pool before closing the fastdialer ensures transports don't attempt to use an already-closed dialer.
One minor defensive improvement: consider adding a nil check for
clientto guard against unexpected nil entries in the pool.,
🛡️ Optional: Add defensive nil check
_ = dialersInstance.HTTPClientPool.Iterate(func(_ string, client *retryablehttp.Client) error { + if client != nil && client.HTTPClient != nil { client.HTTPClient.CloseIdleConnections() + } return nil })🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/protocols/common/protocolstate/state.go` around lines 283 - 289, Add a defensive nil check inside the Iterate callback to skip any nil entries before calling CloseIdleConnections: in the block using dialersInstance.HTTPClientPool.Iterate(func(_ string, client *retryablehttp.Client) error { ... }), verify client != nil and also client.HTTPClient != nil before calling client.HTTPClient.CloseIdleConnections(), then return nil as before; keep the subsequent dialersInstance.HTTPClientPool.Clear() unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@pkg/protocols/common/protocolstate/state.go`:
- Around line 283-289: Add a defensive nil check inside the Iterate callback to
skip any nil entries before calling CloseIdleConnections: in the block using
dialersInstance.HTTPClientPool.Iterate(func(_ string, client
*retryablehttp.Client) error { ... }), verify client != nil and also
client.HTTPClient != nil before calling
client.HTTPClient.CloseIdleConnections(), then return nil as before; keep the
subsequent dialersInstance.HTTPClientPool.Clear() unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 39974cdc-8191-45db-988f-7fd5cd2d5e67
📒 Files selected for processing (1)
pkg/protocols/common/protocolstate/state.go
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
pkg/protocols/http/httpclientpool/clientpool_benchmark_test.go (1)
216-219:⚠️ Potential issue | 🟡 MinorAvoid hard performance thresholds in unit tests.
The
require.Greater(t, speedup, 1.5)assertion is hardware- and load-dependent, making it prone to flakiness on noisy CI runners. The deterministic connection-count assertions (lines 205-214) already validate the behavior correctly.🩹 Proposed fix
// speedup sanity check (at least 1.5x on localhost) speedup := float64(old.Duration) / float64(new.Duration) - require.Greater(t, speedup, 1.5, - "expected at least 1.5x speedup with connection reuse") + t.Logf("observed speedup with connection reuse: %.2fx", speedup)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/protocols/http/httpclientpool/clientpool_benchmark_test.go` around lines 216 - 219, The test currently computes speedup := float64(old.Duration) / float64(new.Duration) and asserts require.Greater(t, speedup, 1.5), which is a flaky, environment-dependent performance threshold; remove the speedup calculation and the require.Greater assertion (the speedup variable, old.Duration/new.Duration usage, and the require.Greater call) and rely on the existing deterministic connection-count assertions in this test to validate behavior instead.
🧹 Nitpick comments (1)
pkg/protocols/http/httpclientpool/clientpool_benchmark_test.go (1)
81-100: Consider addingCloseIdleConnections()for consistency with production wrapper.The test's
connTrackingRoundTripperdoesn't implementCloseIdleConnections(), unlike the productionconnTrackingTransportinclientpool.go. While this doesn't break current tests (since they don't call it), adding the method would maintain consistency and prevent issues if tests are extended.🔧 Proposed addition
func (rt *connTrackingRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) { trace := &httptrace.ClientTrace{ GotConn: func(info httptrace.GotConnInfo) { if info.Reused { rt.reused.Add(1) } else { rt.newConns.Add(1) } }, } req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace)) return rt.base.RoundTrip(req) } + +func (rt *connTrackingRoundTripper) CloseIdleConnections() { + type closeIdler interface{ CloseIdleConnections() } + if ci, ok := rt.base.(closeIdler); ok { + ci.CloseIdleConnections() + } +}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@pkg/protocols/http/httpclientpool/clientpool_benchmark_test.go` around lines 81 - 100, The test's connTrackingRoundTripper is missing a CloseIdleConnections method (unlike production connTrackingTransport in clientpool.go); add a CloseIdleConnections receiver on connTrackingRoundTripper that delegates to the underlying base transport when available (use a type assertion to an interface with CloseIdleConnections or assert *http.Transport) so idle connections are closed consistently with production behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@pkg/protocols/http/httpclientpool/clientpool_benchmark_test.go`:
- Around line 136-186: The scan runners (runTemplateSpray, runHostSpray,
runConcurrentHostSpray) currently ignore the error returned by doRequest which
can hide failures and corrupt connection counts; update each loop to capture the
error (err := doRequest(...)) and fail fast on non-nil errors by aborting (e.g.,
panic or log.Fatalf) with a descriptive message that includes the target URL and
the error; apply this change in the bodies of runTemplateSpray, runHostSpray,
and the goroutine inside runConcurrentHostSpray (use the same failure mechanism
so the benchmark exits immediately when doRequest fails).
---
Duplicate comments:
In `@pkg/protocols/http/httpclientpool/clientpool_benchmark_test.go`:
- Around line 216-219: The test currently computes speedup :=
float64(old.Duration) / float64(new.Duration) and asserts require.Greater(t,
speedup, 1.5), which is a flaky, environment-dependent performance threshold;
remove the speedup calculation and the require.Greater assertion (the speedup
variable, old.Duration/new.Duration usage, and the require.Greater call) and
rely on the existing deterministic connection-count assertions in this test to
validate behavior instead.
---
Nitpick comments:
In `@pkg/protocols/http/httpclientpool/clientpool_benchmark_test.go`:
- Around line 81-100: The test's connTrackingRoundTripper is missing a
CloseIdleConnections method (unlike production connTrackingTransport in
clientpool.go); add a CloseIdleConnections receiver on connTrackingRoundTripper
that delegates to the underlying base transport when available (use a type
assertion to an interface with CloseIdleConnections or assert *http.Transport)
so idle connections are closed consistently with production behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 9d561d91-8fd3-475f-82f2-656d84f3accc
📒 Files selected for processing (1)
pkg/protocols/http/httpclientpool/clientpool_benchmark_test.go
Resolve conflict in clientpool.go: keep ConnectionStats and connTrackingTransport from this branch, adopt dev's removal of Init()/forceMaxRedirects and use ShouldFollowHTTPRedirects().
Mzack9999
marked this pull request as draft
March 25, 2026 19:22
Member
Author
|
Added an end-to-end benchmark (
Apple M1, HTTPS is where it really shows: skipping the TLS handshake on every request drops wall time ~7x and memory ~15x. Under concurrency the win compounds, since hosts no longer fight over a single transport's idle pool. The existing `TestConnectionCount_*` already locks down that with keep-alive on we open exactly one connection per host regardless of request count, so this is real reuse, not measurement noise. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Mzack9999
marked this pull request as ready for review
June 10, 2026 20:23
4 tasks
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Close #5453
Checklist
Summary by CodeRabbit
Performance
Bug Fixes
Chores
Tests