Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Message max size #13

Open
tondacrha opened this issue Dec 11, 2015 · 4 comments
Open

Message max size #13

tondacrha opened this issue Dec 11, 2015 · 4 comments

Comments

@tondacrha
Copy link

Hi. I wanted to make sure I understand the "Design Overview" correctly. Is it true I cannot use Bruce for a use case where messages of size over ~426KB needs to be send to Kafka?

If I want to overcome this limitation I need to cut the big messages into smaller once prior to the to_bruce call, is that correct?

@dspeterson
Copy link
Contributor

Yes that's correct, although in the near future I intend to add support for stream socket input (UNIX domain stream sockets and local TCP). This will allow larger messages, and make Bruce available to clients that cannot easily write to a UNIX domain datagram socket. I'd say chances are pretty good that the feature will be available in the next couple of months, although I can't make any promises.

@dspeterson
Copy link
Contributor

One thing to keep in mind is that Kafka imposes a limit on the size of a single message (message.max.bytes as documented here: https://kafka.apache.org/08/configuration.html ). Although this limit is configurable, increasing it beyond a certain point may have performance implications. If you want to send messages much larger than 1 Mb, you may want to check the Kafka mailing lists to see if increasing message.max.bytes may have unintended effects.

@tondacrha
Copy link
Author

Thanks for reply. Looking forward to the stream sockets. It's not that I want to constantly send large messages to Kafka yet I need to plan for relatively small amount of big messages alongside the majority of small messages.

@dspeterson
Copy link
Contributor

The stream socket input feature is now available in Dory, Bruce's successor. See https://github.com/dspeterson/dory for details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants