Problem with text/event-stream when compress=true

[opavlov24]

Do you want to request a feature or report a bug?

Bug

What did you do?

We use concourse (it's a CI/CD tool) under traefik. Concourse UI uses text/stream to get information about builds from its backend. In traefik 1.3.8 everything works fine, but after updating to traefik 1.4.0 - 1.4.5 concourse can't get any event from its backend (infinite waiting...).

What did you expect to see?

Concourse gets events from the backend.

What did you see instead?

Infinite waiting.

Output of traefik version: (What version of Traefik are you using?)

Traefik 1.4.0 - 1.4.5

What is your environment & configuration (arguments, toml, provider, platform, ...)?

The route to concourse was configured statically.

################################################################
# Global configuration
################################################################

logLevel = "DEBUG"

# Traefik logs file
traefikLogsFile = "/var/log/traefik-debug.log"

# Access logs file
accessLogsFile = "/var/log/accessTraefik.log"

# Entrypoints
defaultEntryPoints = ["http", "https"]
[entryPoints]
  [entryPoints.http]
  address = ":80"
	[entryPoints.http.redirect]
	entryPoint = "https"
  [entryPoints.https]
  address = ":443"
  compress = true
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      certFile = ""
      keyFile = ""

# ...

[file]

[backends]
  [backends.concourse]
    [backends.concourse.servers.server1]
    url = "http://some-ip:8080"
# ...

[frontends]
  [frontends.concourse]
  backend = "concourse"
  passHostHeader = true
    [frontends.concourse.routes.host]
    rule = "Host:some-host"

If applicable, please paste the log output in debug mode (--debug switch)

Only these lines are related to concourse in traefik logs.

time="2017-12-15T22:13:12Z" level=debug msg="Round trip: http://some-ip:port, code: 200, duration: 6.000457ms tls:version: 303, tls:resume:false, tls:csuite:c02f, tls:server:some-host"
time="2017-12-15T22:13:14Z" level=debug msg="Round trip: http://some-ip:port, code: 200, duration: 2.51749ms tls:version: 303, tls:resume:false, tls:csuite:c02f, tls:server:some-host"
time="2017-12-15T22:13:14Z" level=debug msg="Round trip: http://some-ip:port, code: 200, duration: 3.597858ms tls:version: 303, tls:resume:false, tls:csuite:c02f, tls:server:some-host"

[ldez]

Could you try with 1.5.0-rc2 ?

[opavlov24]

I tried. The same thing.

[juliens]

Hi @opavlov24 , can you give me more information about your use case because I have tried to install a concourse stack, but I'm not sure where is the real issue (where in the web UI).

Can you try with the last RC release too (1.5.0-rc4) ?

[TheCase]

After you log in to the Concourse UI, open a pipeline and open an event (one that has logs). The log will not display, instead it gives you a link to the login page.

Check your browser console. You'll see an error about the text/stream event.

[TheCase]

To follow up, I tested with 1.5.0-rc4 and there is no longer an issue related to displaying the logs for pipeline events in Concourse.

[juliens]

Good to know.
Thank you

[mandrean]

Hi @TheCase and @juliens. We run traefik v1.5.4 and concourse v3.9.2 but still have this issue.

Any thoughts?

[juliens]

Hi @mandrean,
Could you open a new issue with your configuration, and everything needed in order to reproduce your problem?
Thx

[opavlov24]

Hi everyone!
I have new information about the problem. So, we discovered that traefik1.5.4 works with SSE (we write a test-service with SSE support), but it hasn't worked with Concourse 3.9.2.

[gbgdev]

Hey @TheCase, what version of Concourse you were using when you saw the events displayed successfully on 1.5.0-rc4?

[TheCase]

my working Concourse version is 3.8.0. I'm currently running Traefik 1.5.3

[emcniece]

Reporting in with Traefik 1.5.4-1 and Concourse 3.11.0. Not only are build logs not loading, the entire web UI is timing out. Currently testing to determine if this is a red herring.

[emcniece]

Ok so the web container timeouts were due to a taxed instance, but I am reliably experiencing the lack of build logs. Confirmed that Traefik is the cause - this works fine with an Nginx proxy.

Devtools network panel shows that the events API call is the one that hangs with Traefik. With Nginx this call completes with a 200 and returns content, but with Traefik it hangs and never completes. I have tested and confirmed this via cURL by copying the request in curl form from devtools and executing it in terminal:

curl 'https://ci.site.com/api/v1/builds/1352/events' -H 'Accept: text/event-stream' -H 'Cache-Control: no-cache' -H 'cookie: __utmz=xx.xx.xx.xx.utmcsr=linkedin.com|utmccn=(referral)|utmcmd=referral|utmcct=/feed/; hubspotutk=xx; __utmc=xx; __hssrc=1; _ga=GA1.2.xx.xx; __utma=xx.xx.xx.xx.xx.xx; __hstc=xx.xx.xx.xx.xx.xx; ATC-Authorization="Bearer xxxxxx"' -H 'Referer: https://ci.site.com/teams/main/pipelines/mobile-test/jobs/test-pr/builds/7' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36' --compressed

A quick search reveals #819 which mentioned that perhaps the Content-Type header requires charset=uft-8 at the end, but this didn't change anything for me. I also tried adding -H "transfer-encoding:chunked" from #503 to no avail.

This setup is in Docker, managed by Rancher.

Traefik docker-compose.yml:

version: '2'
volumes:
  $${acme_vol_name}:
    driver: rancher-nfs
services:
  traefik:
    image: rawmind/alpine-traefik:1.5.4-1
    environment:
      TRAEFIK_ACME_CASERVER: https://acme-v01.api.letsencrypt.org/directory
      TRAEFIK_ACME_CHALLENGE: http
      TRAEFIK_ACME_CHALLENGE_HTTP_ENTRYPOINT: http
      TRAEFIK_ACME_EMAIL: [email protected]
      TRAEFIK_ACME_ENABLE: 'true'
      TRAEFIK_ACME_ONHOSTRULE: 'true'
      TRAEFIK_ADMIN_AUTH_METHOD: basic
      TRAEFIK_ADMIN_AUTH_USERS: someuser
      TRAEFIK_ADMIN_ENABLE: 'true'
      TRAEFIK_ADMIN_PORT: '8000'
      TRAEFIK_ADMIN_SSL: 'false'
      TRAEFIK_ADMIN_STATISTICS: '10'
      TRAEFIK_CONSTRAINTS: ''
      TRAEFIK_FILE_ENABLE: 'false'
      TRAEFIK_HTTPS_COMPRESSION: 'true'
      TRAEFIK_HTTPS_ENABLE: only
      TRAEFIK_HTTPS_PORT: '443'
      TRAEFIK_HTTP_COMPRESSION: 'true'
      TRAEFIK_HTTP_PORT: '80'
      TRAEFIK_INSECURE_SKIP: 'false'
      TRAEFIK_RANCHER_ENABLE: 'true'
      TRAEFIK_RANCHER_HEALTHCHECK: 'true'
      TRAEFIK_RANCHER_MODE: metadata
      TRAEFIK_TIMEOUT_DIAL: '30'
      TRAEFIK_TIMEOUT_HEADER: '0'
      TRAEFIK_TIMEOUT_IDLE: '180'
      TRAEFIK_TIMEOUT_READ: '0'
      TRAEFIK_TIMEOUT_WRITE: '0'
      TRAEFIK_USAGE_ENABLE: 'false'
    volumes_from:
    - traefik-acme
    ports:
    - 8000:8000/tcp
    - 80:80/tcp
    - 443:443/tcp
    labels:
      io.rancher.scheduler.affinity:host_label: traefik_lb=true
      io.rancher.sidekicks: traefik-acme
      io.rancher.container.hostname_override: container_name
      io.rancher.scheduler.global: 'true'
  traefik-acme:
    image: rawmind/alpine-volume:0.0.2-1
    environment:
      SERVICE_GID: '10001'
      SERVICE_UID: '10001'
      SERVICE_VOLUME: /opt/traefik/acme
    network_mode: none
    volumes:
    - traefik:/opt/traefik/acme
    labels:
      io.rancher.container.start_once: 'true'
      io.rancher.scheduler.affinity:container_label_soft_ne: io.rancher.stack_service.name=$${stack_name}/$${service_name}
      io.rancher.container.hostname_override: container_name

Concourse docker-compose.yml:

version: '2'
volumes:
  concourse-keys/keys/worker:
    external: true
    driver: rancher-nfs
  concourse-keys:
    external: true
    driver: rancher-nfs
  concourse-keys/keys/web:
    external: true
    driver: rancher-nfs
  postgres_db:
    external: true
    driver: local
services:
  concourse-keygen:
    image: emcniece/concourse-keygen
    volumes:
    - concourse-keys:/concourse-keys
    command:
    - tail
    - -f
    - /dev/null
    labels:
      io.rancher.scheduler.affinity:host_label: concourse=web
  concourse-worker-btrfs:
    privileged: true
    image: eugenmayer/concourse-worker-solid:3.11.0
    hostname: worker-btrfs
    environment:
      CONCOURSE_TSA_HOST: concourse-web:2222
      CONCOURSE_GARDEN_DNS_SERVER: 8.8.8.8
    stdin_open: true
    volumes:
    - concourse-keys/keys/worker:/concourse-keys
    - /var/lib/docker/concourse-state/worker-btrfs:/worker-state
    tty: true
    links:
    - concourse-web:concourse-web
    labels:
      io.rancher.container.pull_image: always
      io.rancher.scheduler.affinity:host_label: graphdriver=btrfs
  concourse-db:
    image: postgres:10.1
    environment:
      PGDATA: /postgres
      POSTGRES_DB: concourse
      POSTGRES_PASSWORD: xxx
      POSTGRES_USER: concourse
    volumes:
    - postgres_db10.1:/postgres
    labels:
      io.rancher.scheduler.affinity:host_label: concourse=web
  concourse-web:
    image: concourse/concourse:3.11.0
    environment:
      CONCOURSE_BASIC_AUTH_PASSWORD: user
      CONCOURSE_BASIC_AUTH_USERNAME: user
      CONCOURSE_EXTERNAL_URL: https://ci.site.com
      CONCOURSE_GITHUB_AUTH_CLIENT_ID: xxx
      CONCOURSE_GITHUB_AUTH_CLIENT_SECRET: xxx
      CONCOURSE_GITHUB_AUTH_TEAM: user/project
      CONCOURSE_POSTGRES_DATA_SOURCE: postgres://concourse:xxx@concourse-db:5432/concourse?sslmode=disable
      CONCOURSE_VAULT_URL: https://vault.site.com
      CONCOURSE_VAULT_CLIENT_TOKEN: xxx
    volumes:
    - concourse-keys/keys/web:/concourse-keys
    links:
    - concourse-db:concourse-db
    ports:
    - 51310:8080/tcp
    command:
    - web
    labels:
      traefik.frontend.rule: Host:ci.site.com
      traefik.enable: 'true'
      rgon.port: '8080'
      rgon.ssl: 'true'
      rgon.redirect: https
      io.rancher.scheduler.affinity:host_label: concourse=web
      traefik.port: '8080'
      rgon.domain: site.com

[juliens]

I tried with concourse 3.9.2.
I follow https://concoursetutorial.com/ and https://concoursetutorial.com/basics/task-hello-world/ to install my concourse and create a build.

Then I launched a Traefik 1.5.4 (https://github.com/containous/traefik/releases/download/v1.5.4/traefik_linux-amd64) with this config:

defaultEntryPoints = ["http"]

[entryPoints]
  [entryPoints.http]
  address = ":8081"

[file]
  [frontends.frontend-test]
    backend = "backend-test"
  [frontends.frontend-test.routes.test_1]
  rule = "Host:127.0.0.1"
  [backends]
    [backends.backend-test]
        [backends.backend-test.servers.website]
            url = "http://127.0.0.1:8080"

And everythings seems to work on http://127.0.0.1:8081/builds/1

[mandrean]

Hi @juliens! We talked at KubCon today. So we still seem to have this issue...

GCP GKE cluster: v1.9.6-gke.1 (both master and 3x slaves in node pool)
helm/tiller: v2.9.0 (both)
traefik: v1.5.4, deployed using stable/traefik v1.29.1 helm chart with Let's Encrypt enabled (dns-01/gcloud)
concourse: tried both v3.10.0 and v3.13.0, deployed using stable/concourse v1.3.2 helm chart with traefik ingress & tls-acme enabled
GCP GCE node:

 Kernel Version:		4.13.0-1011-gcp
 OS Image:			Ubuntu 16.04.4 LTS
 Operating System:		linux
 Architecture:			amd64
 Container Runtime Version:	docker://17.3.2
 Kubelet Version:		v1.9.6-gke.1
 Kube-Proxy Version:		v1.9.6-gke.1

[mandrean]

So, in the end we decided to abandon Traefik in favor of nginx-ingress + cert-manager + external-dns.

With this setup Concourse UI worked immediately without any issues, and it also gives us some additional capabilities we didn't have before.

But I really like Traefik; maybe some time in the future I will revisit and see if the text/event-stream issue still exists.

[Overbryd]

Traefik 1.7.3

Same problem with compress = true.
When I set compress = false the issue disappears.

Basically this is a regression on #560

ping @ldez @juliens you can contact me on Slack Overbryd I am working Mon - Fri and I would be very happy to get rid of this issue. Therefore I am available if you have any questions or want to peek into my setup here.

[Overbryd]

Here is a reproducible case on traefik:1.7.3: https://gist.github.com/Overbryd/c07aae9f68f487b0bf699c3ec83998b9
Steps to reproduce:

1. Download docker-compose.yml
2. Download config.toml into conf/config.toml
3. docker-compose up
4. curl --compress localhost:8080/events stalls

Compare against curl localhost:8080/events, which responds as it should with a event stream (note that compression is off for that request).

[Ullaakut]

Hi @Overbryd !

Thank you very much for the stack and steps to reproduce the issue.

I've tried reproducing it using exactly the steps that you provided, and I have been unsuccessful. Both curl commands seem to get a response at the same speed.

I think that I followed the steps correctly, here is my folder's structure:

Let me know if I missed something or if there was a mistake in the gist files and I'll try again :)

[Overbryd]

@Ullaakut all correct. Mind the trailing slash on your request.
It should be curl --compressed localhost:8080/events/.

[camelpunch]

Ran into this with Concourse 4.2.1 and Traefik 1.7.4. Took hours to track down.

Steps to reproduce locally:

1. Set a /etc/hosts entry for 127.0.0.1 my.url

2. Run concourse with:

sudo $(which concourse_linux_amd64) quickstart \
    --worker-work-dir=/tmp/concourse-workers \
    --postgres-user=concourse \
    --postgres-password=somepassword \
    --main-team-local-user concourse \
    --add-local-user concourse:somepassword \
    --external-url http://my.url \
    --worker-garden-dns-server 8.8.8.8

3. Run traefik with --configfile of:

defaultEntryPoints = ["http"]

[entryPoints]

[entryPoints.http]
address = ":80"
compress = true

[file]

[backends]

[backends.concourse]
[backends.concourse.servers]
[backends.concourse.servers.server0]
url = "http://localhost:8080"

[frontends]

[frontends.concourse]
entrypoints = ["http"]
backend = "concourse"
passHostHeader = true

[frontends.concourse.routes]
[frontends.concourse.routes.route0]
rule = "Host:my.url"

4. Set up your fly with fly -t ci login --team-name main --concourse-url http://my.url. Follow instructions and log in with credentials above e.g. concourse/somepassword

5. Execute a build with fly -t ci execute -c <(echo '{"platform": linux, "run": {"path": "ls"}}')

6. Do the same with/without compress = true. With compression, the build output is not streamed.

[Ullaakut]

@camelpunch Thank you for sending us the steps to reproduce the issue, but we already knew how to reproduce it thanks to @Overbryd !

The issue is not related to concourse at all in fact, it simply happens with the combination of compressed entrypoints and HTTP streams.

We know what the issue is, but we have not yet decided how to handle it, we'll let you know when we do know.

In the meantime, the only workaround is to either disable compression or avoid using streams, as far as I know.

[demamoh]

Any update on this? The problem still continues for me on concourse helm chart: concourse-3.0.1 | traefik-1.56.1.

[amir20]

I just ran into this issue. For what its worth, I think the size of the payload makes a difference. When I have less than 100 bytes, it shows no data. But when the payload grows bigger then the data shows up.

[jPrest]

I just ran into this issue. For what its worth, I think the size of the payload makes a difference. When I have less than 100 bytes, it shows no data. But when the payload grows bigger then the data shows up.

I can see the same behaviour on my setup.

[BenjaminBeck]

Similar here:
I had a youtrack behind traefik 1.7.
It made a request via ssl with "accept: text/event-stream" that stuck on pending state.,.
Setting "compress = false" solved the issue.

Some more info:
It actually was enough to remove "compress = true" to fix the issue, the false value was not needed.
I don´t remember where that true came from, maybe c&p from some tutorial.
It behaves now as expected: Looks like everithing except the stream is beeing compressed.

[davinkevin]

Same problem for me, through traefik integrated inside k3s as ingress controller 😓