Add just recipe to configure multicluster with a flat network

[mateiidavid]

Our CI infrastructure does not currently support running two k3d clusters in a flat network for testing purposes. The process of bridging two k3d clusters is quite simple (assuming one node for each cluster). For each node, a network (IP) route has to be added to route to the other node's cluster CIDR.

This change adds a just recipe to configure two running clusters to participate in a flat network. Routes are added by executing ip route add in each respective server's docker container. Additionally, to support this, target clusters now run with a different pod CIDR (to ensure there isn't any overlap between the two).

---

Note: also cleaned up the mc-test comment since it wasn't showing properly in just --list, see before and after below.

Before

    mc-target-k3d-delete
    mc-test                       # loaded.
    mc-test-build
    mc-test-load
    mc-test-run                   # Run the multicluster tests without any setup
    md-lint

After

    mc-flat-network-init          # Allow two k3d server nodes to participate in a flat network
    mc-target-k3d-delete
    mc-test                       # Run the multicluster tests with cluster setup
    mc-test-build
    mc-test-load
    mc-test-run                   # Run the multicluster tests without any setup

Note 2: still has to be wired in CI. At the moment the recipe is not invoked in our pipeline.

To test, run:

$ just mc-test-load
$ just mc-flat-network-init

And then apply a pod in each cluster (preferably a client/server, I used curl & podinfo). And then try curling the IP address directly. If it's successful, traffic can be routed:

$ kgp --context=k3d-l5d-test-target -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP          NODE                           NOMINATED NODE   READINESS GATES
podinfo-74fbfc4f5-wzcpt   1/1     Running   0          36s   10.23.0.7   k3d-l5d-test-target-server-0   <none>           <none>

$ kgp
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-5d94fd7bcc-8jlps   2/2     Running   0          73m

$ k exec nginx-deployment-5d94fd7bcc-8jlps -it -c curl -- sh
\~ $ curl 10.23.0.7:9898
{
  "hostname": "podinfo-74fbfc4f5-wzcpt",
  "version": "5.0.1",
  "revision": "",
  "color": "#34577c",
  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
  "message": "greetings from podinfo v5.0.1",
  "goos": "linux",
  "goarch": "amd64",
  "runtime": "go1.15.2",
  "num_goroutine": "6",
  "num_cpu": "16"
}

[alpeb]

It doesn't work for me 😢
It appears ip route add did its thing:

$ docker exec -it k3d-l5d-test-server-0 sh -c 'ip route'
default via 172.25.0.1 dev eth0
10.23.0.0/24 via 172.25.0.3 dev eth0
10.42.0.0/24 dev cni0 proto kernel scope link src 10.42.0.1
172.25.0.0/16 dev eth0 proto kernel scope link src 172.25.0.2

But I still can't reach 10.23.0.7:9898 in my tests. Any ideas?

[alpeb]

After spending more time that I'm willing to admit, I figured it was indeed a Nixos issue... it turns out the nixos firewall is just incompatible with this setup. It's of no use trying to allow traffic for all the ports the k3s manual lists; it works only until I set networking.firewall.enable = false (even if this is just for local traffic). CC @hawkw

[adleong]

works for me