k8s port forwarding not working properly

[GabyCT]

While running with the following configuration

[Meta]
  Version = "1.0.23"

[Runtime]
  Debug = true
  Trace = false
  DisableGuestSeccomp = true
  DisableNewNetNs = false
  Path = "/usr/local/bin/kata-runtime"
  [Runtime.Version]
    Semver = "1.8.0-alpha2"
    Commit = "829ac720cdc58f18154a396ccffb158150103156"
    OCI = "1.0.1-dev"
  [Runtime.Config]
    Path = "/usr/share/defaults/kata-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 4.0.0 (v4.0.0-dirty)\nCopyright (c) 2003-2019                                                                                                              Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-system-x86_64"
  BlockDeviceDriver = "virtio-scsi"
  EntropySource = "/dev/urandom"
  Msize9p = 8192
  MemorySlots = 10
  Debug = true
  UseVSock = false
  SharedFS = "virtio-9p"

[Image]
  Path = "/usr/share/kata-containers/kata-containers-clearlinux-30110-osbuilder-                                                                                                             0c48630-agent-bad7d1b.img"

[Kernel]
  Path = "/usr/share/kata-containers/vmlinuz-4.19.52-43"
  Parameters = "init=/usr/lib/systemd/systemd systemd.unit=kata-containers.targe                                                                                                             t systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket age                                                                                                             nt.log=debug agent.log=debug"

[Initrd]
  Path = ""

[Proxy]
  Type = "kataProxy"
  Version = "kata-proxy version 1.8.0-alpha2-4c6ae8af81e25f7f7fea6bc00a7c95d15fb                                                                                                             7054b"
  Path = "/usr/libexec/kata-containers/kata-proxy"
  Debug = true

[Shim]
  Type = "kataShim"
  Version = "kata-shim version 1.8.0-alpha2-03cf26962fde62394b699cd4a43325565c62                                                                                                             86cc"
  Path = "/usr/libexec/kata-containers/kata-shim"
  Debug = true

[Agent]
  Type = "kata"
  Debug = true
  Trace = false
  TraceMode = ""
  TraceType = ""

[Host]
  Kernel = "4.19.13-200.fc28.x86_64"
  Architecture = "amd64"
  VMContainerCapable = true
  SupportVSocks = true
  [Host.Distro]
    Name = "Fedora"
    Version = "28"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz"

[Netmon]
  Version = "kata-netmon version 1.8.0-alpha2"
  Path = "/usr/libexec/kata-containers/kata-netmon"
  Debug = true
  Enable = false

It seems that is not possible to perform a port forwarding on kubernetes (see https://github.com/kata-containers/tests/blob/master/integration/kubernetes/k8s-port-forward.bats) as when we try to perform a ping on the k8s pod we got the following error

   INFO: k8s configured to use runtimeclass
   deployment.apps/redis-master created
   deployment.extensions/redis-master condition met
   service/redis-master exposed
   pod/redis-master-5879fbd6d9-jd7pj condition met
   NAME                      DESIRED   CURRENT   READY   AGE
   redis-master-5879fbd6d9   1         1         1       6s
   Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
   service/redis-master configured
   redis-master   ClusterIP   10.99.240.212   <none>        6379/TCP   1s
   Could not connect to Redis at 127.0.0.1:7000: Connection refused
   Could not connect to Redis at 127.0.0.1:7000: Connection refused
   Could not connect to Redis at 127.0.0.1:7000: Connection refused
   Could not connect to Redis at 127.0.0.1:7000: Connection refused
   Could not connect to Redis at 127.0.0.1:7000: Connection refused
   Could not connect to Redis at 127.0.0.1:7000: Connection refused
   Could not connect to Redis at 127.0.0.1:7000: Connection refused
   Could not connect to Redis at 127.0.0.1:7000: Connection refused
   Could not connect to Redis at 127.0.0.1:7000: Connection refused
   Could not connect to Redis at 127.0.0.1:7000: Connection refused
   deployment.apps "redis-master" deleted
   service "redis-master" deleted

Here are the journals

Jun 27 15:21:44 gabyfedora kubelet[95166]: I0627 15:21:44.822951   95166 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-jv6zj" (UniqueName: "kubernetes.io/secret/45c91d9c-98ef-11e9-86df-000d3a7b83af-default-token-jv6zj") pod "redis-master-5879fbd6d9-4bwwk" (UID: "45c91d9c-98ef-11e9-86df-000d3a7b83af")
Jun 27 15:43:15 gabyfedora kubelet[105131]: I0627 15:43:15.723931  105131 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-p6skj" (UniqueName: "kubernetes.io/secret/4739c21f-98f2-11e9-a128-000d3a7b83af-default-token-p6skj") pod "redis-master-5879fbd6d9-vx7qg" (UID: "4739c21f-98f2-11e9-a128-000d3a7b83af")
Jun 27 18:52:18 gabyfedora kubelet[113289]: I0627 18:52:18.795462  113289 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-7mtdg" (UniqueName: "kubernetes.io/secret/b03018f4-990c-11e9-9dbc-000d3a7b83af-default-token-7mtdg") pod "redis-master-5879fbd6d9-jd7pj" (UID: "b03018f4-990c-11e9-9dbc-000d3a7b83af")

[krsna1729]

@sboeuf @egernst

[grahamwhaley]

An aside - I am/was wondering if we needed to or if there was a specific reason we use the redis image for this test? Primarily I was wondering if we could optimise the test to use a smaller image, and thus micro-optimise our CI a touch :-) I had a quick check under docker, and the redis image is not actually that big (95MB) - but, hey, if you are in the test fixing stuff, something to consider whilst there. I wonder if we can use the gcr.io/google-samples/hello-app like we moved to in the expose IP test for instance?

[amshinde]

@GabyCT I had looked at port-forwarding long back, and this did not work back then as the CRI layer assumed some namespaces. This may need to be fixed at the CRI layer.
Has this test ever worked?

[GabyCT]

@amshinde no it did not

[amshinde]

@GabyCT Thanks. We need to address this at the CRI layer, shimv2 could be a good candidate. I'll look into what is required to fix this.

[krsna1729]

@amshinde @GabyCT apply limitation label once pointer to CRI code is found and linked here?

[ianlewis]

This is a duplicate of https://github.com/kata-containers/runtime/issues/697