-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
*: multi-topic nsq_tail; automated docker builds; etc #957
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for proposing these changes 😍 ! Just a few questions and comments.
@@ -7,6 +7,7 @@ env: | |||
- GOARCH=amd64 | |||
- GOARCH=386 | |||
sudo: false | |||
go_import_path: github.com/nsqio/nsq |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you explain why this is needed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mreiferson without this line - fork build fails, you can see example run here:
https://travis-ci.org/soar/nsq/jobs/294934597
package github.com/soar/nsq/apps/nsq_pubsub
imports github.com/nsqio/nsq/internal/app: use of internal package not allowed
And here is Travis documentation for this line:
https://docs.travis-ci.com/user/languages/go/#Go-Import-Path
With this line any fork will work.
@@ -1,10 +1,28 @@ | |||
FROM golang:latest AS build |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How large is the resulting container image with this approach?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The special part is the separate FROM alpine:3.6
below - this first image is only used for building, then the result is copied from this image to the separate final image. This technique is newly possible in docker moby 17.05.0-ce
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, curious about the results.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mreiferson This is multistage-build feature from Docker 17.05+. First image is used for build, second - for distribution.
Dockerfile
Outdated
&& chmod +x /bin/dep \ | ||
&& /bin/dep ensure \ | ||
&& ./test.sh \ | ||
&& ./coverage.sh --coveralls \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we need to run code coverage as part of this build?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mreiferson Why not? In my projects I always have testing/coveraging as part of build process. For example if one test fails - build will be interrupted and broken image will not be uploaded to Docker repository. I think this is good practice. Yes, it has some impact on build time, but it is not too drastical.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Runnings tests here can be useful because they can fail the build. But in this case the coverage results are just thrown out.
apps/nsq_tail/nsq_tail.go
Outdated
if err != nil { | ||
log.Fatalf("ERROR: failed to write to os.Stdout - %s", err) | ||
} | ||
_, err = os.Stdout.Write(m.Body) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A side effect of this change is that the output can no longer be (easily) piped into subsequent processes, a common operation when using nsq_tail
.
I suppose one option is to just multiplex the output without prefixing the topic name?
dist.sh
Outdated
@@ -30,7 +30,7 @@ echo "... running tests" | |||
|
|||
for os in linux darwin freebsd windows; do | |||
echo "... building v$version for $os/$arch" | |||
BUILD=$(mktemp -d -t nsq) | |||
BUILD=$(mktemp -d -t nsq-XXXXX) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this still work on OSX?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We've hit this before, not sure what happened 😝
thread starts at #718 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool, let's update this to reflect @ploxiln's suggestion in #718 (comment) ?
This would be better as two separate pull requests IMHO |
Agreed, unless we can quickly come to a conclusion around |
apps/nsq_tail/nsq_tail.go
Outdated
} | ||
|
||
for { | ||
for _, consumer := range consumers { | ||
select { | ||
case <-consumer.StopChan: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not really useful: it will only wait for the first consumer to stop.
I don't think any consumers will stop before we tell them to though, they will try to reconnect. So this case could just be removed (and the loop as well). This just needs to wait for the sigChan and then tell all consumers to stop (as you have below). After that, to be most correct, it should then wait for all consumers to have stopped.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ploxiln I've simply rewritten existing logic. Are you sure, that we need to have only waiting for SigChan
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It wasn't 100% correct before, but in the process of rewriting it you made it more nonsensical. The previous bare for loop is different in nature from a loop over all consumers.
Dockerfile
Outdated
|
||
WORKDIR /go/src/github.com/nsqio/nsq | ||
|
||
RUN go get github.com/mattn/goveralls \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
goveralls isn't needed here since ./coverage.sh is not run here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right, yeah...
I found another little thing, but the updates look good, thanks. |
according to nsqio#957 (review)
This looks good to me. I didn't want to be a bother about squashing commits, so I went ahead and did that in #960 I'll still wait a bit for @mreiferson's approval |
probably worth mentioning: I've tested the nsq_tail multi-topic functionality, but not the docker image build, I don't have a new-enough version of docker/moby set up at the moment |
@ploxiln you don't need Docker to check that It builds successfully, here is automated build on Docker Hub: https://hub.docker.com/r/soarname/nsq/builds/ |
Ah, yes, I can pull and inspect that image. It comes out to 57.36 MB uncompressed, and seems to work fine. |
with optional flag "--print-topic" to print topic name before message body
avoid "use of internal package not allowed" error in this case by forcing the import path to be as if it were the upstream repo
now works for any $GOPATH, not just $HOME/gopath
build in a golang-based container, then copy resulting binaries into a minimal alpine-based container
2217b6c
to
4480364
Compare
(updated) |
@ploxiln @mreiferson Thank you! |
I've used NSQ for one of our projects - It is cool system, thanks!
But I had very annoying problem - We have more than 5 queues and for each I should start new container of nsq_tail:
I know that Docker is "lightweight" system... But not lightweight-enough for running too many containers for simple tasks. So my idea was to make
nsq_tail
listen for multiple topics at once and write all messages from them. Now I can do simply:And I see in logs:
Also I've fixed:
Dockerfile
- now builds for Docker ecosystem are fully automated - you can check it here: soarname/nsq.travis.yml
- import path for building forks (without this line - I've got an error "use of internal package not allowed")coverage.sh
- fix path forgoveralls
dist.sh
- fix callingmktemp
withoutXXX
in template