[Windows] Networking doesn't work #559

lippertmarkus · 2021-11-23T17:06:51Z

When using nerdctl run to start a Windows container, networking doesn't work as there is no ethernet adapter attached:

.\nerdctl.exe pull mcr.microsoft.com/windows/nanoserver:1809
.\nerdctl.exe run --rm mcr.microsoft.com/windows/nanoserver:1809 cmd /c ipconfig

# Windows IP Configuration

Same behaviour like with ctr:

./ctr.exe i pull mcr.microsoft.com/windows/nanoserver:1809
./ctr.exe run -rm  mcr.microsoft.com/windows/nanoserver:1809 test cmd /c ipconfig

# Windows IP Configuration

With crictl it works however:

.\crictl.exe pull mcr.microsoft.com/windows/nanoserver:1809
$POD_ID=(./crictl runp .\pod.json)
$CONTAINER_ID=(./crictl create $POD_ID .\container.json .\pod.json)
./crictl start $CONTAINER_ID
.\crictl.exe exec $CONTAINER_ID cmd /c ipconfig

# Windows IP Configuration
#
# Ethernet adapter vEthernet (3cceb90e9342cc0a23db648c9c4da002a329d285ba3733e8feb666a374e9f842_nat):
#
#   Connection-specific DNS Suffix  . : 
#   Link-local IPv6 Address . . . . . : fe80::1d7f:3f4c:efb5:457b%15
#   IPv4 Address. . . . . . . . . . . : 172.30.197.170
#   Subnet Mask . . . . . . . . . . . : 255.255.240.0
#   Default Gateway . . . . . . . . . : 172.30.192.1

with pod.json:

{
  "metadata": {
    "name": "mycont-sandbox",
    "namespace": "default",
    "attempt": 1,
    "uid": "hdishd83djaidwnduwk28basb"
  },
  "log_directory": "/tmp",
  "linux": {
  }
}

and container.json:

{
  "metadata": {
      "name": "mycont"
  },
  "image":{
      "image": "mcr.microsoft.com/windows/nanoserver:1809"
  },
  "command": [
      "cmd",
      "/c",
      "ping -t 127.0.0.1"
  ],
  "log_path":"mycont.0.log",
  "linux": {
  }
}

Without having any knowledge about the internals, I would assume it's because crictl uses a pod sandbox where networking is set up for and ctr/nerdctl doesn't?

cc @jsturtevant

The text was updated successfully, but these errors were encountered:

jsturtevant · 2021-11-30T16:36:56Z

Looks like CTR has a flag for CNI that needs to be enabled:

Could you try that and see if CTR is working with that flag?

For nerdctl, it look like the network is set up via a OCI hook. Something doesn't seem to be hooked up correctly for Windows. The hook is here:

nerdctl/cmd/nerdctl/internal.go

Lines 24 to 34 in fae1696

    
           var internalCommand = &cobra.Command{ 
        
           	Use:           "internal", 
        
           	Short:         "DO NOT EXECUTE MANUALLY", 
        
           	Hidden:        true, 
        
           	SilenceUsage:  true, 
        
           	SilenceErrors: true, 
        
           } 
        
           internalCommand.AddCommand( 
        
           	newInternalOCIHookCommandCommand(), 
        
           )

and

Here is where the network created with that hook:

nerdctl/pkg/ocihook/ocihook.go

Line 175 in 288848d

o.cni, err = gocni.New(cniOpts...)

jsturtevant · 2021-11-30T16:37:36Z

I will see if I can get this wired up properly.

lippertmarkus · 2021-12-01T08:37:38Z

The cni flag is part of the platformRunFlags and only available on Linux:
https://github.com/containerd/containerd/blob/af1a0908d06b4f7db4a11bfc67ecc2f34baeb04b/cmd/ctr/commands/run/run_unix.go#L82-L85

but not on Windows:
https://github.com/containerd/containerd/blob/af1a0908d06b4f7db4a11bfc67ecc2f34baeb04b/cmd/ctr/commands/run/run_windows.go#L33

lippertmarkus · 2021-12-01T09:46:13Z

I tried adding the cni flag to the platformRunFlags for Windows as well in containerd/containerd#6304 to try it out. It seems that it tries to do the right thing but fails in the end, see containerd/containerd#6305

jsturtevant · 2021-12-01T16:30:06Z

thanks for trying it out and logging the issue on containerd 💯

For nerdctl implementation, it looks like the current use of OCI hooks will not work as they are POSIX only: https://github.com/opencontainers/runtime-spec/blob/v1.0.2/config.md#posix-platform-hooks. A quick look through the hcshim code showed they seem to be implemented for LCOW.

I am guessing we will need to create the network during the run flow (similar to cri flow works) and then pass that information to the OCI spec as defined here: https://github.com/opencontainers/runtime-spec/blob/v1.0.2/config-windows.md#network

@AkihiroSuda do you have any input before I take look at implementing this?

AkihiroSuda · 2021-12-01T16:31:54Z

Why can't Windows support OCI hooks?

AkihiroSuda · 2021-12-01T16:33:52Z

I am guessing we will need to create the network during the run flow (similar to cri flow works) and then pass that information to the OCI spec as defined here:

Does this work with the restart manager (--restart=always)?
On Linux it doesn't work, so it's implemented in the hooks.

jsturtevant · 2021-12-01T16:46:45Z

I am not sure why it doesn't have support. Just inferring from the OCI docs and hcsshim code. spoke briefly with @dcantah and it sounds like it should be technically possible but would need implementation.

Does this work with the restart manager (--restart=always)?
On Linux it doesn't work, so it's implemented in the hooks.

I will have to experiment to figure this out.

lippertmarkus · 2021-12-19T15:21:22Z

Thanks to @kevpar I made networking work via ctr::

./ctr.exe run --cni -rm -t mcr.microsoft.com/windows/nanoserver:1809 test cmd /c ipconfig

Windows IP Configuration

Ethernet adapter vEthernet (default-test_nat):

  Connection-specific DNS Suffix  . :
  Link-local IPv6 Address . . . . . : fe80::a527:27d2:6f6a:6d53%23
  IPv4 Address. . . . . . . . . . . : 172.27.19.133
  Subnet Mask . . . . . . . . . . . : 255.255.240.0
  Default Gateway . . . . . . . . . : 172.27.16.1

Maybe that implementation in containerd/containerd#6304 helps you for nerdctl

jsturtevant · 2022-01-05T16:29:30Z

Maybe that implementation in containerd/containerd#6304 helps you for nerdctl

That is one way to do it but would be different form the Linux implementation here. I also think it might have issues as pointed out above with the flag --restart=always.

I've looked into the OCI hooks and there doesn't seem to be any reason it can't be supported in hcsshim it just hasn't been implemented for Windows. I plan to get a prototype of that working to gauge effort and feasibility and will update here.

kkbruce · 2022-07-15T07:46:46Z

In the beginning, I installed it through install-containerd-runtime.ps1, because the specified SYMLINKD directory was missing, so the --cni of ctr could not be used.

After solving the problems of ctr and --cni, I then tested nerdctl and found that there are similar problems. Provide my environment settings, and whether it can provide a useful solution.

if run nerdctl network ls, command auto-generate nerdctl-nat.conflist file and see two nat network:

PS C:\> nerdctl network ls
NETWORK ID    NAME    FILE
              nat     C:\Program Files\containerd\cni\conf\0-containerd-nat.conf
0             nat     C:\Program Files\containerd\cni\conf\nerdctl-nat.conflist
              host
              none

nerdctl-nat.conflist content:

{
  "cniVersion": "1.0.0",
  "name": "nat",
  "nerdctlID": 0,
  "nerdctlLabels": {},
  "plugins": [
    {
      "type": "nat",
      "master": "Ethernet",
      "ipam": {
        "routes": [
          {
            "gateway": "10.4.0.1"
          }
        ],
        "subnet": "10.4.0.0/24",
        "type": ""
      }
    }
  ]
}

PS C:\> nerdctl -v
nerdctl version 0.21.0

PS C:\> nerdctl info
Client:
 Namespace:     default
 Debug Mode:    false

Server:
 Server Version: v1.6.6
 Storage Driver: windows
 Logging Driver: json-file
 Cgroup Driver:
 Cgroup Version:
 Plugins:
  Log: fluentd journald json-file
  Storage: windows-lcow windows
 Security Options:
 Kernel Version:
 Operating System:
 OSType: windows
 Architecture:
 CPUs: 0
 Total Memory: 0B
 Name: labtestcontaine
 ID: 2755fc58-0723-4ec1-91b9-c652d11765de

Running with --net nat:

PS C:\> nerdctl run --rm --net nat mcr.microsoft.com/windows/nanoserver:ltsc2022 ipconfig

Windows IP Configuration

The run result of ctr with --cni:

PS C:\> ctr run --cni --rm mcr.microsoft.com/windows/nanoserver:ltsc2022 test ipconfig

Windows IP Configuration

Ethernet adapter vEthernet (default-test_nat):

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::fdd7:f487:e1fc:be8f%17
   IPv4 Address. . . . . . . . . . . : 172.20.86.86
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . : 172.20.80.1

lippertmarkus · 2022-07-15T08:14:21Z

@kkbruce Networking support for Windows is not implemented yet in nerdctl. Currently networking only works with crictl and ctr command line tools.

kkbruce · 2022-07-15T08:59:25Z

@lippertmarkus Thanks for your info. Let's take a look at crictl first, and we can only look forward to its implementation.

Refactor the loading and application of the networking-related arguments for the `nerdctl run` command in order to facilitate Windows support. Fixes/alleviates containerd#559 and containerd#1680. Signed-off-by: Nashwan Azhari <[email protected]>

leandroscardua · 2023-02-03T11:53:43Z

He There,

Thank you for the hard work on nerdctl :D
Secondly, Do we have any update on it? to support network on windows container?

jsturtevant · 2023-02-06T18:55:22Z

looks like it is in progress with #1953

leandroscardua · 2023-02-07T11:22:49Z

Thank for the update @jsturtevant

leandroscardua · 2023-03-22T17:07:20Z

#1953

I hope that merge here, may unblock this issue here :D

jsturtevant · 2023-03-22T17:08:26Z

It should, though note it only works with NAT networks right now.

leandroscardua · 2023-03-22T17:12:20Z

It will be an amazing start :D

alkreddy · 2023-07-19T00:44:21Z

I can confirm that version 1.4.0 of nerdctl release does not exhibit this issue. The solution may have been released in an earlier version but I verified it with the latest at the time of this post

AkihiroSuda added the platform/Windows/Non-WSL2 Microsoft Windows (non-WSL2) label Nov 23, 2021

kkbruce mentioned this issue Jul 15, 2022

Windows Server run with --cni got [ctr: no network config found in /etc/cni/net.d: cni plugin not initialized] error microsoft/windows-container-networking#70

Closed

aznashwan mentioned this issue Jan 30, 2023

Refactor run command networking options for Windows support. #1953

Merged

AkihiroSuda closed this as completed Jul 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Windows] Networking doesn't work #559

[Windows] Networking doesn't work #559

lippertmarkus commented Nov 23, 2021

jsturtevant commented Nov 30, 2021

jsturtevant commented Nov 30, 2021

lippertmarkus commented Dec 1, 2021 •

edited

Loading

lippertmarkus commented Dec 1, 2021 •

edited

Loading

jsturtevant commented Dec 1, 2021

AkihiroSuda commented Dec 1, 2021

AkihiroSuda commented Dec 1, 2021

jsturtevant commented Dec 1, 2021

lippertmarkus commented Dec 19, 2021

jsturtevant commented Jan 5, 2022

kkbruce commented Jul 15, 2022

lippertmarkus commented Jul 15, 2022 •

edited

Loading

kkbruce commented Jul 15, 2022

leandroscardua commented Feb 3, 2023

jsturtevant commented Feb 6, 2023

leandroscardua commented Feb 7, 2023

leandroscardua commented Mar 22, 2023

jsturtevant commented Mar 22, 2023

leandroscardua commented Mar 22, 2023

alkreddy commented Jul 19, 2023

[Windows] Networking doesn't work #559

[Windows] Networking doesn't work #559

Comments

lippertmarkus commented Nov 23, 2021

jsturtevant commented Nov 30, 2021

jsturtevant commented Nov 30, 2021

lippertmarkus commented Dec 1, 2021 • edited Loading

lippertmarkus commented Dec 1, 2021 • edited Loading

jsturtevant commented Dec 1, 2021

AkihiroSuda commented Dec 1, 2021

AkihiroSuda commented Dec 1, 2021

jsturtevant commented Dec 1, 2021

lippertmarkus commented Dec 19, 2021

jsturtevant commented Jan 5, 2022

kkbruce commented Jul 15, 2022

lippertmarkus commented Jul 15, 2022 • edited Loading

kkbruce commented Jul 15, 2022

leandroscardua commented Feb 3, 2023

jsturtevant commented Feb 6, 2023

leandroscardua commented Feb 7, 2023

leandroscardua commented Mar 22, 2023

jsturtevant commented Mar 22, 2023

leandroscardua commented Mar 22, 2023

alkreddy commented Jul 19, 2023

lippertmarkus commented Dec 1, 2021 •

edited

Loading

lippertmarkus commented Dec 1, 2021 •

edited

Loading

lippertmarkus commented Jul 15, 2022 •

edited

Loading