-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
producer write broken pipe #1565
Comments
👍 |
update:
|
So the client does already run a periodic metadata refresh based on backgroundMetadata goroutine in client.go which ticks on However, that is sent to a random broker each time so won't necessarily keep alive your idle connections to all of the brokers in the cluster |
thanks for the quick reply 👍 so i have seen this configuration and set it to 5min (kafka broker remains on 10min for idle connection) LOGS
{"level":"info","version":"dev","time":"2020-04-13T14:15:59+03:00","message":"Scout service starting"}
{"level":"info","time":"2020-04-13T14:15:59+03:00","message":"starting to init server config"}
{"level":"debug","caller":"/Users/talbenshabtay/Desktop/workspace/dev/go/pkg/mod/github.com/rs/[email protected]/log.go:383","time":"2020-04-13T14:15:59+03:00","message":"[Initializing new client]"}
{"level":"debug","caller":"/Users/talbenshabtay/Desktop/workspace/dev/go/pkg/mod/github.com/rs/[email protected]/log.go:383","time":"2020-04-13T14:15:59+03:00","message":"[ClientID is the default of 'sarama', you should consider setting it to something application-specific.]"}
{"level":"debug","caller":"/Users/talbenshabtay/Desktop/workspace/dev/go/pkg/mod/github.com/rs/[email protected]/log.go:383","time":"2020-04-13T14:15:59+03:00","message":"[ClientID is the default of 'sarama', you should consider setting it to something application-specific.]"}
{"level":"debug","caller":"/Users/talbenshabtay/Desktop/workspace/dev/go/pkg/mod/github.com/rs/[email protected]/log.go:391","time":"2020-04-13T14:15:59+03:00","message":"client/metadata fetching metadata for all topics from broker [localhost:9092]\n"}
{"level":"debug","caller":"/Users/talbenshabtay/Desktop/workspace/dev/go/pkg/mod/github.com/rs/[email protected]/log.go:391","time":"2020-04-13T14:15:59+03:00","message":"Connected to broker at [localhost:9092] (unregistered)\n"}
{"level":"debug","caller":"/Users/talbenshabtay/Desktop/workspace/dev/go/pkg/mod/github.com/rs/[email protected]/log.go:391","time":"2020-04-13T14:15:59+03:00","message":"client/brokers registered new broker #[1001 %!d(string=localhost:9092)] at %!s(MISSING)"}
{"level":"debug","caller":"/Users/talbenshabtay/Desktop/workspace/dev/go/pkg/mod/github.com/rs/[email protected]/log.go:383","time":"2020-04-13T14:15:59+03:00","message":"[Successfully initialized new client]"}
{"level":"debug","caller":"/Users/talbenshabtay/Desktop/workspace/dev/go/pkg/mod/github.com/rs/[email protected]/log.go:383","time":"2020-04-13T14:15:59+03:00","message":"[ClientID is the default of 'sarama', you should consider setting it to something application-specific.]"}
{"level":"debug","caller":"/Users/talbenshabtay/Desktop/workspace/dev/go/pkg/mod/github.com/rs/[email protected]/log.go:391","time":"2020-04-13T14:15:59+03:00","message":"producer/broker/[1001] starting up\n"}
{"level":"debug","caller":"/Users/talbenshabtay/Desktop/workspace/dev/go/pkg/mod/github.com/rs/[email protected]/log.go:391","time":"2020-04-13T14:15:59+03:00","message":"producer/broker/[1001 %!d(string=serverconfigurations) 0] state change to [open] on %!s(MISSING)/%!d(MISSING)\n"}
{"level":"debug","caller":"/Users/talbenshabtay/Desktop/workspace/dev/go/pkg/mod/github.com/rs/[email protected]/log.go:391","time":"2020-04-13T14:15:59+03:00","message":"Connected to broker at [localhost:9092 %!s(int32=1001)] (registered as #%!d(MISSING))\n"}
@dnwe updated with logs |
@talbspx interesting. If it's just local single broker instance can you change the idle broker config to 2m, set Sarama to 60s refresh and enable debug logs on the broker side? I'll also try and reproduce in the week if I can |
@dnwe yeah np :) |
Interested, we've seen lots of these lately. |
- Fix sarama issue: IBM/sarama#1565 Signed-off-by: Pierangelo Di Pilato <[email protected]>
- Fix sarama issue: IBM/sarama#1565 Signed-off-by: Pierangelo Di Pilato <[email protected]>
* Create a new ClusterAdmin at each loop - Fix sarama issue: IBM/sarama#1565 Signed-off-by: Pierangelo Di Pilato <[email protected]> * Remove error return value from SetBootstrapServer Signed-off-by: Pierangelo Di Pilato <[email protected]>
I am also interested in this issue. I am experiencing it with: This is quite a mission critical application, so I do need to fix this. |
Interested in this issue. I'm experiencing it with: Sarama version v1.27.2 |
Interested in this issue. I'm experiencing it with: Sarama version v1.28.0 |
Experiencing the same issue with the recent sarama version upgrade v1.29.1 and go-1.15.6. Any help in figuring out the cause is appreciated. The producer client is running on a k8s container with 1 core and 2 GiB memory. Does it have to do anything on the producer client end since the only change done was the sarama version upgrade. Thanks in advance :) |
Yes this is a bug. Update: hmm after looking into this some more, the brokerProducer should account for the dropped connection and re-establish it fairly transparently anyway. Needs more investigation |
any update? |
Hi I met the same issue and could you share how can i send a metadata refresh as you said? Thanks a lot! |
any update |
we are facing the same issue, any update? |
Interested in this issue. I'm experiencing it with: Sarama version v1.37.2 |
Any update on this? Sarama version v1.37.2 |
Any update on this? Sarama version v1.36.0 |
A possible workaround is sending heart beat message from client side which
just lookup topic metadata. This workaround works in our product
environment.
Ling Jin ***@***.***> 于2023年2月10日周五 17:59写道:
… Any update on this?
Sarama version v1.36.0
Go version 1.19
Kafka 3.x
—
Reply to this email directly, view it on GitHub
<#1565 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AEDWME4FGEOEUN3JMJY7QVDWWYGRTANCNFSM4J7DXWRQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
I am still experiencing this issue with the latest sarama |
With @david-bergman it sounds like you are still seeing issues — I wonder if you were inadvertently using sasl v0 in v1.38.1 and missing out on the re-auth. Since #2572 the default was (put back) to sasl v1 and we also have a lot better protocol support. Can you re-test with the latest release? |
Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. |
Versions
V1.24.1
Configuration
What configuration values are you using for Sarama and Kafka?
Logs
logs: CLICK ME
time="2019-12-25T09:15:51Z" level=info msg="producer/broker/3 state change to [closing] because write tcp xxxxx:59382->xxxx:9092: write: broken pipe\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/broker/2 state change to [closing] because write tcp xxxx:33336->xxxx:9092: write: broken pipe\n"
time="2019-12-25T09:15:51Z" level=info msg="Closed connection to broker xxxx:9092\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/leader/metric.transfer/0 state change to [retrying-1]\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/leader/metric.transfer/0 abandoning broker 3\n"
time="2019-12-25T09:15:51Z" level=info msg="Closed connection to broker xxxx:9092\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/broker/3 input chan closed\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/broker/3 shut down\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/leader/msg.republish/1 state change to [retrying-1]\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/leader/msg.republish/1 abandoning broker 2\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/leader/metric.hit/0 state change to [retrying-1]\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/leader/metric.hit/0 abandoning broker 2\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/broker/2 input chan closed\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/broker/2 shut down\n"
time="2019-12-25T09:15:51Z" level=info msg="client/metadata fetching metadata for [metric.transfer] from broker xxxx:9092\n"
time="2019-12-25T09:15:51Z" level=info msg="client/metadata fetching metadata for [msg.republish] from broker xxxx:9092\n"
time="2019-12-25T09:15:51Z" level=info msg="client/metadata fetching metadata for [metric.hit] from broker xxxx:9092\n"
time="2019-12-25T09:15:51Z" level=info msg="client/brokers replaced registered broker #1 with xxxx:9092"
time="2019-12-25T09:15:51Z" level=info msg="ClientID is the default of 'sarama', you should consider setting it to something application-specific."
time="2019-12-25T09:15:51Z" level=info msg="producer/broker/3 starting up\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/broker/3 state change to [open] on metric.transfer/0\n"
time="2019-12-25T09:15:51Z" level=info msg="client/brokers replaced registered broker #1 with xxxx:9092"
time="2019-12-25T09:15:51Z" level=info msg="ClientID is the default of 'sarama', you should consider setting it to something application-specific."
time="2019-12-25T09:15:51Z" level=info msg="producer/leader/metric.transfer/0 selected broker 3\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/leader/metric.transfer/0 state change to [flushing-1]\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/leader/metric.transfer/0 state change to [normal]\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/broker/2 starting up\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/broker/2 state change to [open] on metric.ruleengine.hit/0\n"
time="2019-12-25T09:15:51Z" level=info msg="client/brokers replaced registered broker #1 with xxxx:9092"
time="2019-12-25T09:15:51Z" level=info msg="producer/leader/metric.hit/0 selected broker 2\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/leader/msg.republish/1 selected broker 2\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/broker/2 state change to [open] on msg.republish/1\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/leader/msg.republish/1 state change to [flushing-1]\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/leader/msg.republish/1 state change to [normal]\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/leader/metric.hit/0 state change to [flushing-1]\n"
time="2019-12-25T09:15:51Z" level=info msg="producer/leader/metric.hit/0 state change to [normal]\n"
time="2019-12-25T09:15:52Z" level=info msg="Connected to broker at xxxx:9092 (registered as #2)\n"
time="2019-12-25T09:15:53Z" level=info msg="Connected to broker at xxxx:9092 (registered as #3)\n"
Problem Description
After 10 minutes, the producer scoket write broken pipe.
The text was updated successfully, but these errors were encountered: