-
Notifications
You must be signed in to change notification settings - Fork 499
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tools/goreplay-middleware: Add goreplay middleware #4496
base: master
Are you sure you want to change the base?
Conversation
I'm concerned about using |
Both things can be done via CLI flags to goreplay command:
No, prod doesn't care about mirroring and it doesn't affect it at all: How Traffic Mirroring works. |
tools/goreplay-middleware/main.go
Outdated
if !req.ResponseEquals() { | ||
// TODO improve the message to at least print the requested path | ||
// TODO in the future publish the results to S3 for easier processing | ||
os.Stderr.WriteString("MISMATCH " + req.SerializeBase64() + "\n") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can these replay comparison results potentially intermix with normal horizon app log output to console, making it harder to read overall? if so, maybe it's worth optional configurable target(defaulted to console err) for goreplay results output early on?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This messages land in completely different log: goreplay log. It's separate from Horizon log. Ideally the diffs are sent to external server but for testing purposes I wanted to start with something simple.
looks good, would it be possible to show a small diagram of how the replay process works for context? such as the flow from prod(source), target(mirror server), goreplay.log , and the jenkins |
@sreuland I added a short comment explaining how middleware works. @stellar/horizon-committers I think this is ready for review because it helped a lot during Horizon v2.20.0 testing. PTAL. |
PR Checklist
PR Structure
otherwise).
services/friendbot
, orall
ordoc
if the changes are broad or impact manypackages.
Thoroughness
.md
files, etc... affected by this change). Take a look in the
docs
folder for a given service,like this one.
Release planning
needed with deprecations, added features, breaking changes, and DB schema changes.
semver, or if it's mainly a patch change. The PR is targeted at the next
release branch if it's not a patch change.
What
Adds middleware for
goreplay
which checks if the mirrored response matches the original response.Close #2840.
Why
goreplay
middleware gives access to request and responses of original and mirrored targets. This allows us to replicatehorizon-cmp
functionality but on a larger scale (ex.horizon-cmp
sends requests to public load balancers so normal rate limiting applies).Known limitations
Currently logs mismatched response bodied to
stderr
. In the future, we can send files to S3 and build some diff checker infrastructure on top of it.