Is Container#attach streaming working for you ?

[zimbatm]

I'm not getting any streaming output until docker passes the threshold of Excon's :chunk_size. I think it's a bug in excon. See excon/excon#350

[zimbatm]

Example usage:

d = Docker::Container.get('f7758e4a811d')
d.attach do |*a|
  p a
end

[wader]

The problem is that exconn (the http library used by docker-api) uses a quite large buffer when reading. To make attach interactive i do this:

@conn_interactive = Docker::Connection.new(docker_url, {:chunk_size => 1})
container = Docker::Container.create(..., @conn_interactive)
container.attach do |_fd, chunk|
  $stdout.write(chunk)
end

Remember to only use @conn_interactive for interactive thing as downloading or uploading big files will be really slow with a 1 byte buffer. I have some more code if you want to do line oriented things even when reading small chunks (like indenting the output etc).

[bfulton]

@phemmer nice!

@tlunter does it make sense to have #attach default to :chunk_size => nil?

[phemmer]

@bfulton ack, sorry, I deleted my comment. I forgot I had modified excon :-(

Removed my patch, trying to solve again.

[bfulton]

Ah. No worries! Let us know how you make out.

[wader]

What does it mean to have nil as max_len argument for read_nonblock? can't find anything in the documentation about it.

@bfulton i guess people assume attach to be "interactive" (i did) so it probably make sense to change it just for attach.

If efficiency matters I think the largest chunk size we can have and still make it interactive is 8, see https://github.com/dotcloud/docker/blob/master/utils/stdcopy.go (StdWriterPrefixLen)

[phemmer]

@wader That comment was a mistake on my part. I forgot I had patched excon elsewhere.

Yeah, there's no way to do this without patching excon. I've gone through that Excon::Socket#read method dozens of times, there is no way to get the desired behavior.

I really don't understand the design either. It looks like it's trying to behave like IO.read as it always blocks until the requested bytes have been read, but I don't understand why it's been complicated with all the read_nonblock/rescue stuff. Simply calling read and rescuing EOFError would achieve the same result with a fraction of the code.

At this point, I see 2 potential ways of proceeding:

1. Excon modifies that read method so that it doesn't block if the :nonblock option is set.
   This option will be a backwards incompatible and a very drastic change from the method's previous behavior.
2. Monkey patch the method to replace it with one that obeys the :nonblock option.
   Monkey patching is just bad, but as option 1 will likely be a significant problem, it may be the only option.

Any other options?

---

EDIT: 3 options:
3. use :chunk_size => 1 and read byte by byte. This is just horribly inefficient, I would prefer monkey patching over this option.

[bfulton]

@phemmer thanks.

A default-interactive #attach is important, so I would be in favor of adding :chunk_size => 1 (or 8 if we can get away with it as @wader suggests) to the default #attach opts, along with a comment referencing excon/excon/issues/350 so we can revisit if that issue is resolved.

@tlunter thoughts?

[tlunter]

I can't see :chunk_size => 8 being a sure thing. I can see how 1 would be inefficient since it's going down to Excon and back to Docker-API Message every byte, but I think that would honestly be the easiest.

It seems like socket.read(length) will read until EOF or until that size, so there really is no other way besides setting length to 1 or Excon's :nonblock is changed.

We can definitely set :chunk_size => 1 as the default to that, or at least specify in the README that if this method should be interactive in your application that this will need to be used. Chunk size can be set on a per request basis and isn't needed on a full connection. Perhaps we open up attach to a :chunk_size parameter and use a sensible larger default to prevent this from being too slow?

I don't want applications that expect larger responses to be bogged down by the :chunk_size => 1.

[wader]

@tlunter To be clear i meant 8 only for attach but i guess it wont make much difference to 1, both are quite inefficient but my guess is that it is not big deal if the log is mean to be used "interactively". If people do use attach to get very large logs in batch, maybe there should be two methods, e.g. attach and attach_interactive to make it clear?

[tlunter]

I can see attach_interactive being the route we should go down.

[phemmer]

ok, so I started digging into excon looking to solve this issue on that side. Conclusion: I'm not certain this is something excon should handle.

Because of excon/excon#350, I originally thought the issue was with that Excon::Socket#read method, but now I'm not sure it is. Yes, I think the method is ugly, and the #readline method even more so, but changing the way the methods behave isn't a good fix for the issue.
The methods are behaving as they were designed to. You're supposed to pass in a :chunk_size and your callback is supposed to get exactly that many bytes back, no more, no less (unless the connection closes).

Now if the server were using chunked transfer encoding, the callback would get called for every chunk received. But the docker API doesn't use chunked transfer encoding, it becomes a raw stream. As such, Excon has no clue how many bytes should be in each "chunk" that gets sent to the callback, so it reads however many bytes it was told to by the :chunk_size parameter.
The only way to solve this in excon would be to either pass the socket back to the caller in which the caller can call #read or #readpartial on the socket directly, or add a :chunk_size_min parameter which reads until it has chunk_size_min bytes and then calls the callback.

So basically, it's not a bug, it's just the way it was designed.
So now I would like to ask, why is docker-api using Excon? It seems like most of the requests being made are pretty basic. Why not extend Docker::Connection to handle the http request, and then it could handle the raw stream as it wants to.

---

I personally dislike the #attach_interactive idea. The docker-api gem puts a focus on the methods behaving like their API call counterparts. This would change that as #attach would behave radically different from the API call it's named after, and instead you'd have #attach_interactive which behaves properly.
While I think solving this so that #attach behaves like it should in all cases, if the behavior must be changed, #attach with no arguments should behave as the container/attach API call is documented, and then if you want to get large chunks, add a parameter to the method call, and not create a separate method.

[bfulton]

@phemmer I agree that #attach should be interactive, and that if we need a buffered version it should be a different method, or different opts. Abandoning excon seems excessive since the Docker API is mostly a normal HTTP API. What do you think about grabbing the low-level socket used in #attach through reflection?

[tombh]

I'm using the @conn_interactive = Docker::Connection.new(docker_url, {:chunk_size => 1}) fix with success. Just wondering if there's been any progress since?

[arlimus]

I've added a MR to Excon to support chunking/streams since I need it for docker-api and another project on top. Works like a charm for me with docker.attach ;)
See: excon/excon#378