feat: implement per-host rate limiting and statistics#1929

Merged

thomas-zahner merged 43 commits intomasterfrom

per-host-rate-limiting

Dec 21, 2025

Commits on Dec 21, 2025

fix: add support for name attributes in HTML fragment extraction

mre
authored and
thomas-zahner
committed
feat: implement per-host rate limiting and statistics

mre
and
thomas-zahner
committed
fix: skip rate limiting tracking for file:// URLs

mre
authored and
thomas-zahner
committed
feat: improve rate limiting logging and output formatting

mre
authored and
thomas-zahner
committed
Fix lints

mre
authored and
thomas-zahner
committed
Fix cookie jar sharing in per-host rate limiting

mre
authored and
thomas-zahner
committed
Fix missing User-Agent header in per-host clients

mre
authored and
thomas-zahner
committed
Bring back global headers (e.g. for user-agent)

mre
authored and
thomas-zahner
committed
Fix redirect handling in per-host clients

mre
authored and
thomas-zahner
committed
Pass missing args: max_redirects, timeout, allow_insecure

mre
authored and
thomas-zahner
committed
Refactor host stats formatters to remove unused parameters and improve example formatting in README

mre
authored and
thomas-zahner
committed
remove confusing comment

mre
authored and
thomas-zahner
committed
Create `display_per_host_statistics` in separate file

mre
authored and
thomas-zahner
committed
Remove redundant check for `self.hosts`

mre
authored and
thomas-zahner
committed
Import `std::collections::HashMap`

mre
authored and
thomas-zahner
committed
Use closures instead of if

mre
authored and
thomas-zahner
committed
Rename flags:

mre
authored and
thomas-zahner
committed
Fix `help` formatting

mre
authored and
thomas-zahner
committed
Reduce code duplication
thomas-zahner
committed
Update documentation to reference hosts option
thomas-zahner
committed
clippy --fix
thomas-zahner
committed
Return HostPool instead of Client & code cleanup
thomas-zahner
committed
Move inner `Arc`s to the outside
thomas-zahner
committed
Fix deadlock
thomas-zahner
committed
Update config options
thomas-zahner
committed
Simplify host pool
thomas-zahner
committed
Build host-specific reqwest clients again
thomas-zahner
committed
Remove max_concurrency and global_semaphore
thomas-zahner
committed
Update docs & reduce complexity
thomas-zahner
committed
Extract output functions
thomas-zahner
committed
Replace Window with Vec
thomas-zahner
committed
Update RateLimitError
thomas-zahner
committed
Create RequestInterval
thomas-zahner
committed
Remove RateLimitError
thomas-zahner
committed
Test and improve rate limit header handling
thomas-zahner
committed
Apply @mre's suggestions
thomas-zahner
committed
Apply suggestions from code review

thomas-zahner
and
mre
committed
Fix tests
thomas-zahner
committed
Minor improvements
thomas-zahner
committed
Remove reqwest_client from WebsiteChecker
thomas-zahner
committed
Reference rate-limits crate as per @mre's suggestion
thomas-zahner
committed
Update option names and the default interval value
thomas-zahner
committed
Allow 0 to disable per-host rate limiting
thomas-zahner
committed