Split the binary into two: client and daemon

[dnephin]

Fixes #20362

./client is a new directory for the docker binary (aka the client)
./docker is the old directory which builds the docker-daemon binary (aka the daemon)

docker daemon is supported from the client by exec-ing the docker-daemon binary with the same args.

Notes:

• I think the PostProcess() of CommonFlags could use some cleanup, but I'd like to wait for another branch since it's not required for this change. For now I've used a closure.
• docker daemon will exec docker-daemon using exec.LookPath() to find it. It does not assume they are in the same directory.
• When exec'ed all args are passed to docker-daemon. This means that a call like docker --graph=... daemon --bip ... will actually work. The logic to prevent that would be non-trivial, so I feel like it's better to just allow this. There's very little reason to prevent it now, since the flags on docker-daemon are a combination of both the common flags and daemon flags, they need to be unique anyway.
• dynbinary builds both the client and the daemon, and the deb and rpm packages include both binaries.

Questions:

• Is docker-daemon the right name for the daemon binary? (another suggestion is dockerd which would make things easier for the windows build).

[lowenna]

@mikedougherty @dnephin @mebersol @tiborvass @icecrime This will have an impact to at least the Windows to Windows Jenkins CI script, and I think likely the Windows to Linux Jenkins CI script. Also the deployment scripts on Windows. I have to ask - what is the value of doing this? Is there an issue having a single binary? Or a specific goal being addressed by this change?

[mikedougherty]

@jhowardmsft overall I understand it to be splitting two programs that do vastly different things. I like the idea of it. Of course there will be a transition period :) but this looks worth it to me.

(this is an argument for running the CI scripts from the revision the code is from but that raises potential security issues)

[thaJeztah]

Also see #20362 for the roadmap issue

[lowenna]

OK, I a little looking to see what it would take to convert the CI scripts to run on Windows TP4 in this world, and continue to run before this gets merged. First issue is the name of the daemon binary. The way Windows CI works is it starts the daemon under test with the name docker-gitcommitid.exe. When the script goes through its clear-down for reliability, it looks for any processes matching docker-* and kills them. That way it doesn't kill the control daemon (just docker.exe). So two ways to fix this - either change the script to not use - as seperator, or call the daemon binary something other than docker-daemon.exe. It's generally unusual for native exe's on Windows to include a - in their name. (Not sure about Linux norms.) Could the binary be called dockerd.exe instead?

[dnephin]

Could the binary be called dockerd.exe instead?

I think the name is still very flexible, so this is possible. I'd like to hear other opinions on naming before making any changes, to try and reduce the number of times I rename it.

When the script goes through its clear-down for reliability, it looks for any processes matching docker-* and kills them.

That sounds ... aggressive. Why doesn't it just track the pids of the daemons that get started, and just kill the specific processes that it needs to kill?

[lowenna]

@dnephin - Yes, aggressive, but what it takes to keep windowsTP4 reliable with the current set of issues. (Should add in the context of this, it won't necessarily know PIDs what was running previously, hence the arbitrary nuking of processes and much else)

[lowenna]

@icecrime Who's the 'test this' comment addressed to?

[icecrime]

@jhowardmsft "test this please" is the magic sentence to trigger the rebuild of the "documentation" job. Very much not confusing, right? 😆

[lowenna]

Oh, THAT's how you retrigger documentation! 👿

[tiborvass]

As far as design goes, this makes sense to me so +1.
For the name of the daemon, docker-daemon is better since we still want docker daemon to work by shelling out to docker-daemon. It feels more natural that way. It also opens up possible future git-like cli design.

[lowenna]

@tiborvass While it might 'feel more natural that way' to Linux users, it's pretty alien to Windows users...

[duglin]

I prefer dockerd since I'm used to sshd, ftpd, etc....

[lowenna]

...and containerd soon 😈

[thaJeztah]

Perhaps the daemon should just be called docker-engine 😇

[lowenna]

@thaJeztah It's the - I have a problem with 😇

[tiborvass]

Ping @dnephin let me know if you want us to carry it if you're not available.

[dnephin]

I don't really have a good way to test out these windows only changes right now, but I pushed what I think is a fix.

[lowenna]

It got past unit tests on Windows with that update. LGTM if full CI passes

[tiborvass]

the failed test on windowsTP5 is unrelated to this PR

19:47:48 ----------------------------------------------------------------------
19:47:48 FAIL: docker_cli_restart_test.go:222: DockerSuite.TestRestartContainerwithRestartPolicy
19:47:48 
19:47:48 docker_cli_restart_test.go:229:
19:47:48     c.Assert(err, checker.IsNil)
19:47:48 ... value *errors.errorString = &errors.errorString{s:"condition \"\"false true\" == \"false false\"\" not true in time"} ("condition \"\"false true\" == \"false false\"\" not true in time")
19:47:48 
19:48:10 
19:48:10 ----------------------------------------------------------------------

Possibly related to #22042

[lowenna]

Yes, agreed. Merging.

[nathanleclaire]

@nathanleclaire hey, yo. Heard you like to know when format stuff is changing and its happening again. U R welcome ;)

Thanks !