use '-' as default separator and support of compatibility mode #294

glours · 2022-07-29T13:07:34Z

Change the default separator used between project name and resources to -. This is especially needed for container names to avoid issue with hostname support.

Also add detection of the COMPOSE_COMPATIBILITY env variable to support the compatibility usage of the _ separator

Fix docker/compose#9618

Signed-off-by: Guillaume Lours <[email protected]>

milas

LGTM

cli/options_test.go

Co-authored-by: Milas Bowman <[email protected]> Signed-off-by: Guillaume Lours <[email protected]>

ulyssessouza · 2022-07-29T15:23:26Z

cli/options.go

+		if utils.StringToBool(options.Environment["COMPOSE_COMPATIBILITY"]) {
+			o.Separator = loader.CompatibilitySeparator
+		}


Please note that this only fixes docker/compose#9618 when passing an environment variable. In the case of Docker Compose getting a --compatibility from the command line I'm not sure that will work (didn't test though).

Given this: https://github.com/docker/compose/blob/v2/cmd/compose/compose.go#L162-L169

Looks like compose just evaluates the command line option after this code is already evaluated.
One solution would be checking the command line option before and setting the EnvVar before calling ProjectFromOptions in the Docker Compose repo for this PR to work properly.

logopk · 2022-07-30T20:10:42Z

Stupid breaking change - duplicates all my networks and returns conflicting ip ranges.
"failed to create network step-ca_t-default: Error response from daemon: Pool overlaps with other one on this address space"
Quite a hassle to get this fixed!

reverted to 2.7.0 also because all referred "external" network names changed!

Shouldn't that be announced as major change with necessary migration or the explicit use of "COMPOSE_COMPATIBILITY"?

Thanks.

Peter

MrNaif2018 · 2022-07-31T11:16:28Z

Compose v2 automatically changed container names for me when just using docker compose up command. When upgrading to 2.8.0 it did create new volumes and network, but containers used old ones, so it requires a full restart, also volumes data isn't migrated automatically but probably should be

glours · 2022-07-31T13:01:04Z

If you want to keep the previous behaviour, use the --compatibility flag

docker compose --compatibility up -d
[+] Running 5/5
 ⠿ Network react-express-mongodb_react-express  Created                                                                                                                                                    0.1s
 ⠿ Network react-express-mongodb_express-mongo  Created                                                                                                                                                    0.0s
 ⠿ Container mongo                              Started                                                                                                                                                    4.8s
 ⠿ Container backend                            Started                                                                                                                                                    5.1s
 ⠿ Container frontend                           Started

logopk · 2022-07-31T13:24:34Z

???

i need to ediit ALL of my scripts?

MrNaif2018 · 2022-07-31T13:37:19Z

I think the issue here is that --compatibility flag is for compose v1 -> v2 migration, but breaking change (which should have probably been in v2.0) was introduced in a minor release 2.8.0
We'll try to adapt it and move docker volumes ourselves, docker compose v2 is stable to be used in production, right?

ulyssessouza · 2022-07-31T13:40:34Z

You can also use an environment variable for that:
export COMPOSE_COMPATIBILITY=true

Like this you don't have to use --compatibility on every call on your scripts

glours · 2022-07-31T14:46:08Z

@MrNaif2018 yes you can use it production, End-Of-Life of Compose V1 was announced months ago

logopk · 2022-08-01T11:47:15Z

I still do think this is absurd. All running environments need to be rebuilt - even with recreation of all volumes.
Plus all connections to external networks/volumes need to be rewritten!

I do understand that the environment variable fixes this. But what other side effects has this, as I'm on v2 for a long time and I don't need compatibility with v1!

Me don't like this. It's a minor version release! Please reconsider and make this an opt in.

(Even better: if there are existing networks/volumes, leave them alone!)

BTW: I'm not against the new separator. - is much better than _. But I don't want to change existing running environments. Quite a few in my case.

glours · 2022-08-01T14:09:27Z

@logopk I will rollback the changes and do releases Compose and compose-go

…esource-names" This reverts commit 56e6c33, reversing changes made to 0ab97a2.

…esource-names" This reverts commit 56e6c33, reversing changes made to 0ab97a2. Signed-off-by: Guillaume Lours <[email protected]>

glours · 2022-08-01T14:35:36Z

@MrNaif2018 FYI we're rolling back the changes, do be to fast in your volumes migration!

glours · 2022-08-01T15:00:09Z

A new release of compose-go available without the breaking changes, a version of Docker Compose should follow soon
cc @logopk @MrNaif2018 @AkihiroSuda

edit: Compose v2.9.0 is available

MrNaif2018 · 2022-08-01T16:30:08Z

@glours Oh I see. I didn't start implementing the migration script luckily. What should my next steps be? I mean I am migrating all deployments from compose v1 to v2 and finished it, but in the meantime when I didn't yet release the changes 2.8.0 release occurred. Ideally in our scripts I would install the latest version of compose v2: https://github.com/bitcartcc/bitcart-docker/blob/dc7de21a1826ede14318701a668956935c64a5d8/helpers.sh#L155-L174

What should my further actions be? Should I prepare that those changes will return back at a later point in compose v2, or they won't? Or they will return but with auto-migration from both compose 1.x and 2.x to the new version like it's done for container re-creation? I'm fine either way, ideally I would migrate all users to v2+ and then no weird bug reports would occur (:

glours · 2022-08-01T16:38:59Z

@MrNaif2018 just use v2.9.0 of Compose which was just released.
The value of those changes regarding the impact on existing Compose stacks running is to low, so we won't reintroduce them.

logopk · 2022-08-01T19:48:53Z

Thank you for the quick response. Much appreciated!

jowax · 2022-08-11T00:11:46Z

~~@glours Seems like the separator between the project and service name when generating the image name changed from _ to - from compose v2.7.0 to v2.9.0. Is this correct?~~

Nevermind I just found your response in docker/compose#9700 (comment), sorry for the disturbance.

Compose file:

version: '3.8'

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile

Version 2.9.0

~/projects/foo ❯ ./docker-compose-2.9.0 --version                                                                                                                                                                                                
Docker Compose version v2.9.0
~/projects/foo ❮ ./docker-compose-2.9.0 --project-name foo_devcontainer -f docker-compose.yml build                                                                                                                                              
[+] Building 0.2s (5/5) FINISHED                                                                                                                                                                                                                                                          
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                 0.0s
 => => transferring dockerfile: 31B                                                                                                                                                                                                                                                  0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                                                                                                                      0.2s
 => CACHED [1/1] FROM docker.io/library/ubuntu:20.04@sha256:af5efa9c28de78b754777af9b4d850112cad01899a5d37d2617bb94dc63a49aa                                                                                                                                                         0.0s
 => exporting to image                                                                                                                                                                                                                                                               0.0s
 => => exporting layers                                                                                                                                                                                                                                                              0.0s
 => => writing image sha256:b6139e4d50625d70c36fc0232ae66d887f4a58115d9579a989f4490f0a65cf28                                                                                                                                                                                         0.0s
 => => naming to docker.io/library/foo_devcontainer-app                                                                                                                                                                                                                              0.0s

Version 2.8.0

~/projects/foo ❯ ./docker-compose-2.8.0 --version                                                                                                                                                                                               
Docker Compose version v2.8.0
~/projects/foo ❮ ./docker-compose-2.8.0 --project-name foo_devcontainer -f docker-compose.yml build                                                                                                                                              
[+] Building 0.5s (5/5) FINISHED                                                                                                                                                                                                                                                          
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                 0.0s
 => => transferring dockerfile: 31B                                                                                                                                                                                                                                                  0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                                                                                                                      0.4s
 => CACHED [1/1] FROM docker.io/library/ubuntu:20.04@sha256:af5efa9c28de78b754777af9b4d850112cad01899a5d37d2617bb94dc63a49aa                                                                                                                                                         0.0s
 => exporting to image                                                                                                                                                                                                                                                               0.0s
 => => exporting layers                                                                                                                                                                                                                                                              0.0s
 => => writing image sha256:b6139e4d50625d70c36fc0232ae66d887f4a58115d9579a989f4490f0a65cf28                                                                                                                                                                                         0.0s
 => => naming to docker.io/library/foo_devcontainer-app                                                                                                                                                                                                                              0.0s

Version 2.7.0

~/projects/foo ❯ ./docker-compose-2.7.0 --version                                                                                                                                                                                                
Docker Compose version v2.7.0
~/projects/foo ❯ ./docker-compose-2.7.0 --project-name foo_devcontainer -f docker-compose.yml build                                                                                                                                              
[+] Building 0.6s (5/5) FINISHED                                                                                                                                                                                                                                                          
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                 0.0s
 => => transferring dockerfile: 31B                                                                                                                                                                                                                                                  0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                                                                                                                      0.5s
 => CACHED [1/1] FROM docker.io/library/ubuntu:20.04@sha256:af5efa9c28de78b754777af9b4d850112cad01899a5d37d2617bb94dc63a49aa                                                                                                                                                         0.0s
 => exporting to image                                                                                                                                                                                                                                                               0.0s
 => => exporting layers                                                                                                                                                                                                                                                              0.0s
 => => writing image sha256:b6139e4d50625d70c36fc0232ae66d887f4a58115d9579a989f4490f0a65cf28                                                                                                                                                                                         0.0s
 => => naming to docker.io/library/foo_devcontainer_app                                                                                                                                                                                                                              0.0s

jamshid · 2023-03-05T22:18:43Z

@glours wrote:

The value of those changes regarding the impact on existing Compose stacks running is to low, so we won't reintroduce them.

Eh ok but the network name can end up in a fully qualified dns name:

# nslookup 172.23.0.15
15.0.23.172.in-addr.arpa	name = myproject-myservice-2.myproject_default.

and at least python is very strict about underscores, leading to idna.core.InvalidCodepoint: Codepoint U+005F errors e.g.:

          File "/tmp/build/results/usr/local/lib/python3.9/dist-packages/twisted/python/urlpath.py", line 265, in __str__
          File "/tmp/build/results/usr/local/lib/python3.9/dist-packages/hyperlink/_url.py", line 1668, in to_uri
          File "/tmp/build/results/usr/local/lib/python3.9/dist-packages/idna/core.py", line 360, in encode
          File "/tmp/build/results/usr/local/lib/python3.9/dist-packages/idna/core.py", line 258, in alabel
          File "/tmp/build/results/usr/local/lib/python3.9/dist-packages/idna/core.py", line 297, in ulabel
          File "/tmp/build/results/usr/local/lib/python3.9/dist-packages/idna/core.py", line 250, in check_label
        idna.core.InvalidCodepoint: Codepoint U+005F at position 10 of 'myproject_default' not allowed

Workaround: specify the network name in docker-compose.yml:

networks:
  default:
    name: myproject-default
    driver: bridge
    attachable: true

AlexMikhalev · 2023-03-25T12:56:27Z

You don't realise how much work such changes generate for projects. Mine is a medium-complexity project https://github.com/applied-knowledge-systems/the-pattern/ where deployment is now broken in multiple places. I spent time two years ago making all deployments docker and docker-compose based. Going forward, I will ditch docker-compose and only use docker for builds. Try to reach a point where benchmarks work: https://reference-architecture.ai/docs/bert-qa-benchmarking/ and track the effort required to update all docker-compose files.
Looking forward to your PRs.

use '-' as default separator and support of compatibility mode

aa04417

Signed-off-by: Guillaume Lours <[email protected]>

glours requested a review from ndeloof as a code owner July 29, 2022 13:07

glours requested review from laurazard, ulyssessouza, milas and nicksieger July 29, 2022 13:08

glours self-assigned this Jul 29, 2022

milas mentioned this pull request Jul 29, 2022

as --no-build is set, set service image to default image name docker/compose#9179

Merged

milas approved these changes Jul 29, 2022

View reviewed changes

cli/options_test.go Outdated Show resolved Hide resolved

use t.Setenv instead of os.SetEnv + defer os.Unsetenv

aa0ca5e

Co-authored-by: Milas Bowman <[email protected]> Signed-off-by: Guillaume Lours <[email protected]>

glours force-pushed the use-dash-in-resource-names branch from 2f23f04 to aa0ca5e Compare July 29, 2022 13:46

milas approved these changes Jul 29, 2022

View reviewed changes

ulyssessouza requested changes Jul 29, 2022

View reviewed changes

ulyssessouza approved these changes Jul 29, 2022

View reviewed changes

glours merged commit 56e6c33 into compose-spec:master Jul 29, 2022

AkihiroSuda mentioned this pull request Jul 30, 2022

Bump github.com/compose-spec/compose-go from 1.2.9 to 1.3.0 containerd/nerdctl#1276

Merged

glours added a commit to glours/compose-go that referenced this pull request Aug 1, 2022

Revert "Merge pull request compose-spec#294 from glours/use-dash-in-r…

ed7dcc9

…esource-names" This reverts commit 56e6c33, reversing changes made to 0ab97a2.

glours mentioned this pull request Aug 1, 2022

update to compose-go v1.4.0 as previous version introduced breaking changes docker/compose#9700

Merged

aminvakil mentioned this pull request Aug 2, 2022

Use docker-compose 2.7.0 instead of 2.2.3 in CI getsentry/self-hosted#1591

Merged

jkeech mentioned this pull request Aug 5, 2022

VScode devcontainer build script calls image by wrong name microsoft/vscode-dev-containers#1567

Closed

dvintel mentioned this pull request Aug 9, 2022

Change to apk install of docker-compose intel/Edge-Software-Provisioner#38

Merged

bigtallcampbell mentioned this pull request Aug 10, 2022

Remote Containers Extension incompatible with Docker-Compose v2.9 microsoft/vscode-remote-release#7047

Closed

laurazard mentioned this pull request Nov 8, 2022

Compose using underscore when generating volume and network names docker/compose#9618

Closed

denys-chura mentioned this pull request Oct 3, 2024

Update docker-compose.yml to use hyphen in all resources names instead of underscore django-stars/backend-skeleton#82

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use '-' as default separator and support of compatibility mode #294

use '-' as default separator and support of compatibility mode #294

glours commented Jul 29, 2022 •

edited

Loading

milas left a comment

ulyssessouza Jul 29, 2022 •

edited

Loading

logopk commented Jul 30, 2022 •

edited

Loading

MrNaif2018 commented Jul 31, 2022

glours commented Jul 31, 2022 •

edited

Loading

logopk commented Jul 31, 2022

MrNaif2018 commented Jul 31, 2022

ulyssessouza commented Jul 31, 2022

glours commented Jul 31, 2022

logopk commented Aug 1, 2022 •

edited

Loading

glours commented Aug 1, 2022

glours commented Aug 1, 2022

glours commented Aug 1, 2022 •

edited

Loading

MrNaif2018 commented Aug 1, 2022

glours commented Aug 1, 2022

logopk commented Aug 1, 2022

jowax commented Aug 11, 2022 •

edited

Loading

jamshid commented Mar 5, 2023

AlexMikhalev commented Mar 25, 2023

use '-' as default separator and support of compatibility mode #294

use '-' as default separator and support of compatibility mode #294

Conversation

glours commented Jul 29, 2022 • edited Loading

milas left a comment

Choose a reason for hiding this comment

ulyssessouza Jul 29, 2022 • edited Loading

Choose a reason for hiding this comment

logopk commented Jul 30, 2022 • edited Loading

MrNaif2018 commented Jul 31, 2022

glours commented Jul 31, 2022 • edited Loading

logopk commented Jul 31, 2022

MrNaif2018 commented Jul 31, 2022

ulyssessouza commented Jul 31, 2022

glours commented Jul 31, 2022

logopk commented Aug 1, 2022 • edited Loading

glours commented Aug 1, 2022

glours commented Aug 1, 2022

glours commented Aug 1, 2022 • edited Loading

MrNaif2018 commented Aug 1, 2022

glours commented Aug 1, 2022

logopk commented Aug 1, 2022

jowax commented Aug 11, 2022 • edited Loading

jamshid commented Mar 5, 2023

AlexMikhalev commented Mar 25, 2023

glours commented Jul 29, 2022 •

edited

Loading

ulyssessouza Jul 29, 2022 •

edited

Loading

logopk commented Jul 30, 2022 •

edited

Loading

glours commented Jul 31, 2022 •

edited

Loading

logopk commented Aug 1, 2022 •

edited

Loading

glours commented Aug 1, 2022 •

edited

Loading

jowax commented Aug 11, 2022 •

edited

Loading