Zombienet smoke is flaky or what?

[bkontur]

Closes: #682

Continuation of discussion: #606 (comment)

Run with (zombienet=debug) logs:
https://github.com/pandres95/runtimes/actions/runs/14503153751/job/40687256001?pr=24

cc: @pepoviola

[bkchr]

AFAIK this was fixed.

[bkontur]

AFAIK this was fixed.

@bkchr Actually, I didn't change the description or title from the original investigation - and yes, the original issue was fixed.

However, in this PR I continued with additional changes, including:

• some minor fixes and cleanup for the configuration,
• running the zombienet smoke test with a matrix setup using different polkadot/polkadot-parachain versions, based on discussions here: Zombienet smoke test fails #682,
• switching from the Docker provider to the Native provider, downloading binaries directly from the polkadot-sdk. I believe downloading Docker images is more resource-intensive than fetching the binaries, and we were frequently encountering errors like:
  Error: docker: Cannot connect to the Docker daemon at unix:///run/user/1001/docker.sock. Is the docker daemon running?
  So this change might help mitigate these types of issues.
  • and using matrix setup with Docker provider was pretty flaky because of error above

Do you think it still makes sense to continue with these changes? If not, we can also consider closing this issue: #682.

If you think it's worth continuing, do you have any recommendations or suggestions?
The general idea was to improve test coverage for runtimes across previous and mixed validator/collator versions (Does this even make sense to test?).

---

Docker downloads:

REPOSITORY                  TAG       IMAGE ID       CREATED        SIZE
parity/polkadot             v1.18.0   86ba3f40a0cf   3 weeks ago    228MB
parity/polkadot-parachain   v1.18.0   e40b23e2e9c3   3 weeks ago    507MB
alpine                      latest    aded1e1a5b37   2 months ago   7.83MB


REPOSITORY                  TAG       IMAGE ID       CREATED        SIZE
parity/polkadot-parachain   v1.18.0   e40b23e2e9c3   3 weeks ago    507MB
parity/polkadot             v1.17.4   0249602cc5cd   4 weeks ago    225MB
alpine                      latest    aded1e1a5b37   2 months ago   7.83MB

REPOSITORY                  TAG       IMAGE ID       CREATED        SIZE
parity/polkadot             v1.17.4   0249602cc5cd   4 weeks ago    225MB
parity/polkadot-parachain   v1.17.4   932e52f336bd   4 weeks ago    498MB
alpine                      latest    aded1e1a5b37   2 months ago   7.83MB

Binary downloads:
https://github.com/polkadot-fellows/runtimes/actions/runs/14837429399/job/41651815723#step:9:35

-rwxr-xr-x 1 runner docker 106691704 May  5 13:40 polkadot-polkadot-stable2412-4
-rwxr-xr-x 1 runner docker  14212176 May  5 13:40 polkadot-execute-worker-polkadot-stable2412-4
-rwxr-xr-x 1 runner docker  14973288 May  5 13:40 polkadot-prepare-worker-polkadot-stable2412-4
-rwxr-xr-x 1 runner docker 179270304 May  5 13:40 polkadot-parachain-polkadot-stable2412-4
-rwxr-xr-x 1 runner docker 108436592 May  5 13:40 polkadot-polkadot-stable2503
-rwxr-xr-x 1 runner docker  14787376 May  5 13:40 polkadot-execute-worker-polkadot-stable2503
-rwxr-xr-x 1 runner docker  15571304 May  5 13:40 polkadot-prepare-worker-polkadot-stable2503
-rwxr-xr-x 1 runner docker 185825784 May  5 13:40 polkadot-parachain-polkadot-stable2503