Docker Swarm error: Service cannot be explicitly attached to the ingress network "ingress"

[schiggi]

When using the default yml for Docker Swarm, I am getting the following error during start:

requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.41/services/9gffebl2qqovhlyf8nuit3kkb/update?version=9790
[...]
docker.errors.APIError: 400 Client Error for http+docker://localhost/v1.41/services/9gffebl2qqovhlyf8nuit3kkb/update?version=9790: Bad Request ("rpc error: code = InvalidArgument desc = Service cannot be explicitly attached to the ingress network "ingress"")
[...]
  File "/scripts/processor/__init__.py", line 176, in inspect_network
    service.update(networks = network_list)
  File "/scripts/processor/__init__.py", line 64, in refresh
    self.inspect_network()
    self.refresh()
  File "/scripts/processor/__init__.py", line 43, in __init__
  File "/scripts/main.py", line 7, in start
    processor_obj = ProcessorInterface.factory(os.getenv("EASYHAPROXY_DISCOVER"))
  File "/scripts/processor/__init__.py", line 52, in factory
    return Swarm()
  File "/scripts/processor/__init__.py", line 155, in __init__
    super().__init__()

My yml looks like this:

version: "3"

services:
  haproxy:
    image: byjg/easy-haproxy
    environment:
      EASYHAPROXY_DISCOVER: swarm
      EASYHAPROXY_SSL_MODE: "loose"
      HAPROXY_CUSTOMERRORS: "true"
      HAPROXY_USERNAME: admin
      HAPROXY_PASSWORD: password
      HAPROXY_STATS_PORT: 1936
    ports:
      - "80:80/tcp"
      - "443:443/tcp"
      - "1936:1936/tcp"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      replicas: 1
    networks:
      - proxy_public
networks:
  proxy_public:
    external: true

[byjg]

Hello,

The network you create probably is not for Docker Swarm. I checked the documentation and notice the procedure to create the network was wrong. I fixed here

The correct way to create a network for Docker Swarm is:

docker network create -d overlay easyhaproxy

Let me know if it worked.

[schiggi]

Thanks for getting backt to me! But the network is created correctly with the overlay option.

Moreover, the error message indicates that the problem is not the created network but the default ingress network.
Bad Request ("rpc error: code = InvalidArgument desc = Service cannot be explicitly attached to the ingress network "ingress"")

Not sure how to debug from here.

[byjg]

I tried several ways to reproduce this error and the way I could reproduce it was attaching easyhaproxy to the default ingress network.

e.g.:

version: "3"

services:
  haproxy:
    image: byjg/easy-haproxy
    environment:
      EASYHAPROXY_DISCOVER: swarm
      EASYHAPROXY_SSL_MODE: "loose"
      HAPROXY_CUSTOMERRORS: "true"
      HAPROXY_USERNAME: admin
      HAPROXY_PASSWORD: password
      HAPROXY_STATS_PORT: 1936
    ports:
      - "80:80/tcp"
      - "443:443/tcp"
      - "1936:1936/tcp"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      replicas: 1
    networks:
      - ingress
networks:
  ingress:
    external: true

And when I try to run:

docker stack deploy --compose-file easyhaproxy_bug2.yml easyhaproxy
Creating service easyhaproxy_haproxy
failed to create service easyhaproxy_haproxy: Error response from daemon: rpc error: code = InvalidArgument desc = Service cannot be explicitly attached to the ingress network "ingress"

For some reason the proxy_public is being handled as a default swarm network.

My suggestion is remove completely the network from easyhaproxy:

version: "3"

services:
  haproxy:
    image: byjg/easy-haproxy
    environment:
      EASYHAPROXY_DISCOVER: swarm
      EASYHAPROXY_SSL_MODE: "loose"
      HAPROXY_CUSTOMERRORS: "true"
      HAPROXY_USERNAME: admin
      HAPROXY_PASSWORD: password
      HAPROXY_STATS_PORT: 1936
    ports:
      - "80:80/tcp"
      - "443:443/tcp"
      - "1936:1936/tcp"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      replicas: 1

It should allow swarm create the proper network for this stack. It is important to note that is NOT REQUIRED you create a different network for EasyHaProxy. It is able to handle automatically.

[schiggi]

Thanks for the advice. Tried it without a network and same error popped up.
In my understanding, every service that publishes ports gets automatically the ingress network attached. So somehow the script tries to reattach the default ingress network.

I tried a different approach and stopped other stacks before starting easyhaproxy. These stacks had no labels. And it worked. haproxy came up properly*. After some more testing, it seems that I need to start haproxy first before other stacks (but services seem to be fine). And if I want to update haproxy settings, I need to stop other stacks, restart haproxy stack and then start the other ones.

Another problems seems to be that I am getting the same error about ingress network, when trying to add some haproxy labels to stacks. Needless to say that proxing doesn't work.

Any ideas where to investigate further?

*Its default network attached itself to another docker service running (Portainer Agent). Not sure if that is good or bad.

[byjg]

I might found the issue. In all tests I did, only EasyHAProxy had the ports exposed. If you also exposing the ports of the other services It might be causing the issue. Since EasyHAProxy is the ingress, you don't need expose the ports of services. However, digging here, I changed the way to add the network and it can fix the issue in this particular case scenario. I opened the PR #37 to track this issue.

Could you test with the version byjg/easy-haproxy:4.3.1-rc1?

[schiggi]

Great, thanks, tried it. It seems that it is starting now. However, in my configuration below it seems to have removed the odoo-test network from services.

Only ingress and lb_default (lb being the stack name for haproxy) are attached to web service container.

version: '3.3'
services:
  db:
    image: postgres:13
    command:
     - postgres
     - -c
     - max_connections=200
    environment:
      PGDATA: /var/lib/postgresql/data/pgdata
      POSTGRES_DB: test
      POSTGRES_PASSWORD_FILE: /run/secrets/odoo_erp_test_db_password
    ports:
     - 5432
    networks:
     - odoo-test
    volumes:
     - odoo-test-db-data:/var/lib/postgresql/data/pgdata
    secrets:
     - odoo_erp_test_db_password
  web:
    image: odoo/odoo:v14
    deploy:
      replicas: 1
      labels:
        easyhaproxy.frontend.host: test.domain.de
        easyhaproxy.frontend.port: 443
        easyhaproxy.frontend.sslcert: /run/secrets/erp-test-cloudflare-certs-base64
        easyhaproxy.frontend.ssl: "true"
        easyhaproxy.frontend.localport: 8069
    command:
     - --max-cron-threads=1
    environment:
      PASSWORD_FILE: /run/secrets/odoo_erp_test_db_password
      USER: odoo
    ports:
     - 8069
     - 8071
     - 8072
    volumes:
     - odoo-test-web-data:/var/lib/odoo
    networks:
     - odoo-test
    secrets:
     - odoo_erp_test_db_password
     - erp-test-cloudflare-certs-base64
networks:
  odoo-test:
    driver: overlay
volumes:
    odoo-test-db-data:
    odoo-test-web-data:
secrets:
  odoo_erp_test_db_password:
    external: true
  erp-test-cloudflare-certs-base64:
    external: true

On a side note, it seems lb_default is attaching itself to every service it can find. Is there a way to constrain that behavior?
The problem in my config is that I have a service (not a stack) running portainer agent for Portainer. When re-deploying or stopping haproxy, I am getting an error that the default network cannot be removed as it is used by portainer-agent. When manually dropping the network, it stops working and I need a rollback. After that the network is removed and I can finally stop the stack and the network is deleted.

[byjg]

Now I could reproduce your issue on my side and I created a new tag byjg/easy-haproxy:4.3.1-rc2 with a possible solution.

What I addressed here:

• Attach the EasyHAProxy network to the service, keeping the existing networks;
• Do not change services that don't have the easyhaproxy label.
• Do not try to change the ingress network

Regarding your issue with the network, the only way EasyHAProxy can communicate with the other services is by putting them in the same network. One consideration with that is if we deploy the EasyHAProxy and assign a network of other services to EasyHAProxy or allow the stack to create a network, we won't be able to remove the EasyHAProxy stack because the network is attached to other services. One way to avoid this is creating the network outside the stack.

docker network create -d overlay easyhaproxy

And on your stack:

services:
  haproxy:
    image: byjg/easy-haproxy:4.3.1-rc2
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ...
    networks:
      - easyhaproxy

networks:
  easyhaproxy:
    external: true

[byjg]

Just pinging to see if your issue was solved.

[schiggi]

Thanks! Sorry for not getting back to you. I have moved on with Traefik for our test env. But I will check your RC and get back to you once I have time.