LogMessageWaitStrategy for DockerComposeContainer

[s2929346]

Hi,

I am starting some docker containers via a docker-compose.yml file. In code, I use the DockerComposeContainer class as following:

DockerComposeContainer c = new DockerComposeContainer(<path>).withTailChildContainers(true);
c.starting(null);

I would like to use the LogMessageWaitStrategy (https://github.com/testcontainers/testcontainers-java/blob/master/core/src/main/java/org/testcontainers/containers/wait/LogMessageWaitStrategy.java) on this container. I must wait until certain log events occurred before I can start using the containers. The LogMessageWaitStrategy should be perfect for this.

Is this possible? I am having a hard time to figure out how I should do this.

Thanks guys!

[s2929346]

Hi,

Update from my side. I got it working with the following code:

DockerClient client = DockerClientFactory.instance().client();
WaitingConsumer consumer = new WaitingConsumer();

LogUtils.followOutput(client, <id of container>, consumer, OutputFrame.OutputType.STDOUT, OutputFrame.OutputType.STDERR);

try {
    consumer.waitUntil(frame -> frame.getUtf8String().matches(<regex>), 300, TimeUnit.SECONDS, 1);
} catch (TimeoutException e) {
    // todo - handle exception properly
} finally {
    // todo - unregister logger/consumer!
}

The only problem left here is that I should unregister the consumer in the finally clause. If I don't do this, I think I will run out of memory eventually, because WaitingConsumer.frames is basically unbounded. That said, I have no idea how to do this. I could execute DockerClientFactory.instance().client().stop(), but that will probably kill the whole test-containers library.

I could try to initialize my own docker client, instead of using DockerClientFactory.instance().client(). Do you have any other ideas?

Are there plans to integrate the wait strategy-functionality on DockerComposeContainer?

Thanks guys!

[rnorth]

Hi @s2929346
As you've seen I'm afraid we don't have the log wait functionality for docker-compose yet. I can't see it being too difficult to add, but it might still not be something we could add quickly due to other work that's ongoing. Sorry.

What you're doing looks fine as a workaround. As you point out WaitingConsumer.frames being unbounded could cause a problem if your logs are that big, so that is a bug that I'd like to address (as this would affect all users, not just people using docker-compose). We could probably drop log frames as they're consumed, which would be significantly more efficient. Will have a look.
Thanks
Richard

[s2929346]

Hi @rnorth,

Thank you for your thoughts here. I'll continue using my current implementation for now then. An in-house support for this would be great though!

With regards to WaitingConsumer and unbounded buffer: I think you can remove a logging frame once it has been evaluated by the predicate. Since the regex is matched against each frame separately, we don't need the frame anymore after predicate evaluation. And now, looking again to the actual code in WaitingConsumer, this is exactly what you are doing in frames.pollLast(100, TimeUnit.MILLISECONDS). So I think there is no real problem after all. My apologies for the confusion.

While writing this down, it made me wonder if a multi-line regex wait on the complete log so far would ever be useful. This is currently not supported, as each frame is checked individually in WaitingConsumer. This would also make the logic above more complex, as you need to keep the frames longer and need to decide when to remove them anyway.

Thanks!

[s2929346]

Right, I remember what I was thinking earlier: It might be useful to support that a WaitingConsumer (and its underlying docker container callback) could be removed/disposed somehow. I think a WaitingConsumer's job becomes obsolete once the waiting predicate has evaluated to true (of the timeout has reached). Now, after the waitUntil returns or throws TimeoutException, the WaitingConsumer is still receiving frames. And this will lead to out of memory errors eventually. There is no way to stop this.

That's basically what I wanted to do in the finally clause in the code snippet above.

Something like LogUtils.removeFollowOutput(docker client, container id, consumer). But then you would probably need the callback instance and not the consumer instance.

[bsideup]

Fixed by #600 (to be released very soon)

[bsideup]

Also, Duplicates #174. Will close this one.