Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: multi-topic nsq_tail; automated docker builds; etc #957

Merged
merged 5 commits into from
Nov 6, 2017

Conversation

soar
Copy link
Contributor

@soar soar commented Oct 30, 2017

I've used NSQ for one of our projects - It is cool system, thanks!

But I had very annoying problem - We have more than 5 queues and for each I should start new container of nsq_tail:

version: '3.2'
services:
  nsqtail:
    image: nsqio/nsq
    command: /nsq_tail --topic=mytopic1 --lookupd-http-address=nsqlookupd:4161
  nsqtail2:
    image: nsqio/nsq
    command: /nsq_tail --topic=mytopic2 --lookupd-http-address=nsqlookupd:4161
  nsqtail3:
    image: nsqio/nsq
    command: /nsq_tail --topic=mytopic3 --lookupd-http-address=nsqlookupd:4161
...

I know that Docker is "lightweight" system... But not lightweight-enough for running too many containers for simple tasks. So my idea was to make nsq_tail listen for multiple topics at once and write all messages from them. Now I can do simply:

version: '3.2'
services:
  nsqtail:
    image: soarname/nsq
    command: /nsq_tail --topic=topic1 --topic=topic2 --topic=topic3 --lookupd-http-address=nsqlookupd:4161

And I see in logs:

nsqtail_1     | topic1 | test-message-1
nsqtail_1     | topic2 | test-message-2
nsqtail_1     | topic3 | test-message-3

Also I've fixed:

  • Dockerfile - now builds for Docker ecosystem are fully automated - you can check it here: soarname/nsq
  • .travis.yml - import path for building forks (without this line - I've got an error "use of internal package not allowed")
  • coverage.sh - fix path for goveralls
  • dist.sh - fix calling mktemp without XXX in template

Copy link
Member

@mreiferson mreiferson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for proposing these changes 😍 ! Just a few questions and comments.

@@ -7,6 +7,7 @@ env:
- GOARCH=amd64
- GOARCH=386
sudo: false
go_import_path: github.com/nsqio/nsq
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain why this is needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mreiferson without this line - fork build fails, you can see example run here:

https://travis-ci.org/soar/nsq/jobs/294934597

package github.com/soar/nsq/apps/nsq_pubsub
    imports github.com/nsqio/nsq/internal/app: use of internal package not allowed

And here is Travis documentation for this line:

https://docs.travis-ci.com/user/languages/go/#Go-Import-Path

With this line any fork will work.

@@ -1,10 +1,28 @@
FROM golang:latest AS build
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How large is the resulting container image with this approach?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The special part is the separate FROM alpine:3.6 below - this first image is only used for building, then the result is copied from this image to the separate final image. This technique is newly possible in docker moby 17.05.0-ce

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, curious about the results.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mreiferson This is multistage-build feature from Docker 17.05+. First image is used for build, second - for distribution.

Dockerfile Outdated
&& chmod +x /bin/dep \
&& /bin/dep ensure \
&& ./test.sh \
&& ./coverage.sh --coveralls \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need to run code coverage as part of this build?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mreiferson Why not? In my projects I always have testing/coveraging as part of build process. For example if one test fails - build will be interrupted and broken image will not be uploaded to Docker repository. I think this is good practice. Yes, it has some impact on build time, but it is not too drastical.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Runnings tests here can be useful because they can fail the build. But in this case the coverage results are just thrown out.

if err != nil {
log.Fatalf("ERROR: failed to write to os.Stdout - %s", err)
}
_, err = os.Stdout.Write(m.Body)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A side effect of this change is that the output can no longer be (easily) piped into subsequent processes, a common operation when using nsq_tail.

I suppose one option is to just multiplex the output without prefixing the topic name?

dist.sh Outdated
@@ -30,7 +30,7 @@ echo "... running tests"

for os in linux darwin freebsd windows; do
echo "... building v$version for $os/$arch"
BUILD=$(mktemp -d -t nsq)
BUILD=$(mktemp -d -t nsq-XXXXX)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this still work on OSX?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've hit this before, not sure what happened 😝
thread starts at #718 (comment)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, let's update this to reflect @ploxiln's suggestion in #718 (comment) ?

@mreiferson mreiferson changed the title Better nsq_tail and automatic builds *: multi-topic nsq_tail; automated docker builds; etc Oct 31, 2017
@ploxiln
Copy link
Member

ploxiln commented Oct 31, 2017

This would be better as two separate pull requests IMHO

@mreiferson
Copy link
Member

This would be better as two separate pull requests IMHO

Agreed, unless we can quickly come to a conclusion around nsq_tail.

}

for {
for _, consumer := range consumers {
select {
case <-consumer.StopChan:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not really useful: it will only wait for the first consumer to stop.

I don't think any consumers will stop before we tell them to though, they will try to reconnect. So this case could just be removed (and the loop as well). This just needs to wait for the sigChan and then tell all consumers to stop (as you have below). After that, to be most correct, it should then wait for all consumers to have stopped.

Copy link
Contributor Author

@soar soar Nov 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ploxiln I've simply rewritten existing logic. Are you sure, that we need to have only waiting for SigChan?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't 100% correct before, but in the process of rewriting it you made it more nonsensical. The previous bare for loop is different in nature from a loop over all consumers.

soar added a commit to soar/nsq that referenced this pull request Nov 3, 2017
soar added a commit to soar/nsq that referenced this pull request Nov 3, 2017
Dockerfile Outdated

WORKDIR /go/src/github.com/nsqio/nsq

RUN go get github.com/mattn/goveralls \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

goveralls isn't needed here since ./coverage.sh is not run here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, yeah...

@ploxiln
Copy link
Member

ploxiln commented Nov 3, 2017

I found another little thing, but the updates look good, thanks.

soar added a commit to soar/nsq that referenced this pull request Nov 5, 2017
@ploxiln
Copy link
Member

ploxiln commented Nov 5, 2017

This looks good to me. I didn't want to be a bother about squashing commits, so I went ahead and did that in #960

I'll still wait a bit for @mreiferson's approval

@ploxiln
Copy link
Member

ploxiln commented Nov 5, 2017

probably worth mentioning: I've tested the nsq_tail multi-topic functionality, but not the docker image build, I don't have a new-enough version of docker/moby set up at the moment

@soar
Copy link
Contributor Author

soar commented Nov 5, 2017

@ploxiln you don't need Docker to check that It builds successfully, here is automated build on Docker Hub: https://hub.docker.com/r/soarname/nsq/builds/

@ploxiln
Copy link
Member

ploxiln commented Nov 5, 2017

Ah, yes, I can pull and inspect that image. It comes out to 57.36 MB uncompressed, and seems to work fine.

soar added 5 commits November 5, 2017 15:17
with optional flag "--print-topic" to print topic name before message body
avoid "use of internal package not allowed" error in this case
by forcing the import path to be as if it were the upstream repo
now works for any $GOPATH, not just $HOME/gopath
build in a golang-based container, then copy
resulting binaries into a minimal alpine-based container
@ploxiln
Copy link
Member

ploxiln commented Nov 5, 2017

(updated)

@mreiferson mreiferson merged commit 9d6dad6 into nsqio:master Nov 6, 2017
@soar
Copy link
Contributor Author

soar commented Nov 6, 2017

@ploxiln @mreiferson Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants