Add scripts to verify BZ1980679 by HuijingHei · Pull Request #1231 · coreos/fedora-coreos-config

HuijingHei · 2021-09-23T03:42:47Z

This PR is to add scripts to verify https://bugzilla.redhat.com/show_bug.cgi?id=1980679:

config.ign to include remote.ign
Verify kernel arg and exists /etc/testfile

Add comment to put an exclusion on qemu:

# This case needs to access remote.ign on github
# qemu-unpriv machines cannot communicate to network
# kola: { "platforms": "! qemu-unpriv", "tags": "needs-internet" }

The remote ignition file will be used to verify BZ1980679, including inject kernel arguments and write something to /etc/testfile

config.ign to include remote.ign verify kernel arg and exists /etc/testfile

HuijingHei · 2021-09-23T03:49:21Z

Test with qemu image, get error:
ignition[698]: GET error: Get "https://raw.githubusercontent.com/coreos/fedora-coreos-config/testing-devel/tests/kola/ignition/remote/remote.ign": dial tcp 0.0.0.0:443: connect: connection refused

$ cosa kola run -p qemu-unpriv -b rhcos --qemu-image /srv/rhcos.qcow2 -E /srv/fedora-coreos-config 'ext.fedora-coreos-config.ignition.remote'
--- FAIL: ext.fedora-coreos-config.ignition.remote (612.25s)
        harness.go:1124: Cluster failed starting machines: machine "8474f2ae-9684-402d-b378-e572ae981bd8" failed to start: ssh journalctl failed: ssh: handshake failed: read tcp 127.0.0.1:45468->127.0.0.1:37099: read: connection reset by peer

Test with azure, result is passed

$ kola run -b rhcos -d -p aws --aws-type t2.small --aws-ami ami-xxx --aws-region us-east-1 -E /srv/fedora-coreos-config 'ext.fedora-coreos-config.ignition.remote'

--- PASS: ext.fedora-coreos-config.ignition.remote (279.74s)

dustymabe · 2021-09-23T03:55:32Z

hmm other qemu tests of ours that need internet work just fine.. this might actually be an issue with our "we need internet, so let's bring up the network" detection logic in the initramfs not working like it should. I can try to dig into this more tomorrow.

The summary is that this should absolutely work (unless I'm extremely confused) and if it doesn't work it's a bug we should fix.

HuijingHei · 2021-09-23T03:59:44Z

hmm other qemu tests of ours that need internet work just fine.. this might actually be an issue with our "we need internet, so let's bring up the network" detection logic in the initramfs not working like it should. I can try to dig into this more tomorrow.

The summary is that this should absolutely work (unless I'm extremely confused) and if it doesn't work it's a bug we should fix.

Thanks! Perhaps my env is not correct, I will try to find

travier

LGTM. Can you squash everything into a single commit?

HuijingHei · 2021-09-23T11:30:01Z

Here are the related console logs when testing on qemu qcow2 image, is this because guest (with ip 10.0.2.15) can not reach github?

[    6.014691] ignition[697]: op(1): [started]  loading QEMU firmware config module
[    6.060606] ignition[697]: op(1): [finished] loading QEMU firmware config module
[    6.062180] ignition[697]: GET https://raw.githubusercontent.com/coreos/fedora-coreos-config/testing-devel/tests/kola/ignition/remote/remote.ign: attempt 1
[    6.064201] ignition[697]: GET error: Get "https://raw.githubusercontent.com/coreos/fedora-coreos-config/testing-devel/tests/kola/ignition/remote/remote.id
[    6.266399] ignition[697]: GET https://raw.githubusercontent.com/coreos/fedora-coreos-config/testing-devel/tests/kola/ignition/remote/remote.ign: attempt 2
[    6.267388] ignition[697]: GET error: Get "https://raw.githubusercontent.com/coreos/fedora-coreos-config/testing-devel/tests/kola/ignition/remote/remote.id
[    6.330254] NetworkManager[726]: <info>  [1632390500.7483] NetworkManager (version 1.30.0-10.el8_4) is starting... (for the first time)
[    6.331144] NetworkManager[726]: <info>  [1632390500.7494] Read config: /etc/NetworkManager/NetworkManager.conf
[    6.332571] NetworkManager[726]: <info>  [1632390500.7508] auth[0x55d89b2a6ea0]: create auth-manager: D-Bus connection not available. Polkit is disabled a.
[    6.334183] NetworkManager[726]: <info>  [1632390500.7524] manager[0x55d89b2d4020]: monitoring kernel firmware directory '/lib/firmware'.
[    6.336028] NetworkManager[726]: <info>  [1632390500.7535] hostname: hostname: hostnamed not used as proxy creation failed with: Could not connect: No sucy
[    6.337047] NetworkManager[726]: <info>  [1632390500.7535] dns-mgr[0x55d89b2c31d0]: init: dns=default,systemd-resolved rc-manager=symlink
[    6.339035] NetworkManager[726]: <info>  [1632390500.7561] Loaded device plugin: NMTeamFactory (/usr/lib64/NetworkManager/1.30.0-10.el8_4/libnm-device-plu)
[    6.340279] IPv6: ADDRCONF(NETDEV_UP): ens4: link is not ready
[    6.341029] NetworkManager[726]: <info>  [1632390500.7561] manager: rfkill: Wi-Fi enabled by radio killswitch; enabled by state file
[    6.341840] NetworkManager[726]: <info>  [1632390500.7561] manager: rfkill: WWAN enabled by radio killswitch; enabled by state file
[    6.344046] NetworkManager[726]: <info>  [1632390500.7561] manager: Networking is enabled by state file
[    6.344682] NetworkManager[726]: <info>  [1632390500.7561] dhcp-init: Using DHCP client 'internal'
[    6.345614] NetworkManager[726]: <warn>  [1632390500.7564] ifcfg-rh: dbus: don't use D-Bus for com.redhat.ifcfgrh1 service
[    6.346504] NetworkManager[726]: <info>  [1632390500.7564] settings: Loaded settings plugin: ifcfg-rh ("/usr/lib64/NetworkManager/1.30.0-10.el8_4/libnm-se)
[    6.349591] NetworkManager[726]: <info>  [1632390500.7564] settings: Loaded settings plugin: keyfile (internal)
[    6.350307] NetworkManager[726]: <info>  [1632390500.7571] device (lo): carrier: link connected
[    6.350892] NetworkManager[726]: <info>  [1632390500.7571] manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1)
[    6.351740] NetworkManager[726]: <info>  [1632390500.7583] manager: (ens4): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2)
[    6.352575] NetworkManager[726]: <info>  [1632390500.7584] device (ens4): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'exte)
[    6.353534] NetworkManager[726]: <info>  [1632390500.7609] device (ens4): carrier: link connected
[    6.354145] NetworkManager[726]: <warn>  [1632390500.7612] sleep-monitor-sd: failed to acquire D-Bus proxy: Could not connect: No such file or directory
[    6.355292] NetworkManager[726]: <info>  [1632390500.7613] device (ens4): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'mana)
[    6.356236] NetworkManager[726]: <info>  [1632390500.7615] policy: auto-activating connection 'Wired Connection' (56a92705-1c4c-4ec5-bc5c-12375dde1d6a)
[    6.357338] NetworkManager[726]: <info>  [1632390500.7617] device (ens4): Activation: starting connection 'Wired Connection' (56a92705-1c4c-4ec5-bc5c-1237)
[    6.358327] NetworkManager[726]: <info>  [1632390500.7617] device (ens4): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
[    6.359547] NetworkManager[726]: <info>  [1632390500.7617] manager: NetworkManager state is now CONNECTING
[    6.360444] NetworkManager[726]: <info>  [1632390500.7618] device (ens4): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
[    6.361351] NetworkManager[726]: <info>  [1632390500.7619] device (ens4): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
[    6.362260] NetworkManager[726]: <info>  [1632390500.7620] dhcp4 (ens4): activation: beginning transaction (timeout in 90 seconds)
[    6.363595] NetworkManager[726]: <info>  [1632390500.7659] dhcp4 (ens4): state changed unknown -> bound, address=10.0.2.15
[    6.364353] NetworkManager[726]: <info>  [1632390500.7661] device (ens4): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
[    6.365256] NetworkManager[726]: <info>  [1632390500.7662] device (ens4): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
[    6.367033] NetworkManager[726]: <info>  [1632390500.7662] device (ens4): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed)
[    6.367946] NetworkManager[726]: <info>  [1632390500.7662] manager: NetworkManager state is now CONNECTED_LOCAL
[    6.368630] NetworkManager[726]: <info>  [1632390500.7665] manager: NetworkManager state is now CONNECTED_SITE
[    6.370045] NetworkManager[726]: <info>  [1632390500.7665] policy: set 'Wired Connection' (ens4) as default for IPv4 routing and DNS
[    6.370826] NetworkManager[726]: <info>  [1632390500.7665] policy: set-hostname: set hostname to 'qemu0' (from DHCPv4)
[    6.373049] NetworkManager[726]: <info>  [1632390500.7666] device (ens4): Activation: successful, device activated.
[    6.373740] NetworkManager[726]: <info>  [1632390500.7666] manager: NetworkManager state is now CONNECTED_GLOBAL
[    6.376044] NetworkManager[726]: <info>  [1632390500.7668] manager: startup complete
[    6.376574] NetworkManager[726]: <info>  [1632390500.7668] quitting now that startup is complete
[    6.378051] NetworkManager[726]: <info>  [1632390500.7669] dhcp4 (ens4): canceled DHCP transaction
[    6.378649] NetworkManager[726]: <info>  [1632390500.7669] dhcp4 (ens4): state changed bound -> done
[    6.379274] NetworkManager[726]: <info>  [1632390500.7669] device (ens4): DHCPv4: trying to acquire a new lease within 180 seconds
[    6.380054] NetworkManager[726]: <info>  [1632390500.7669] manager: NetworkManager state is now CONNECTED_SITE
[    6.380713] NetworkManager[726]: <info>  [1632390500.7670] exiting (success)
[  OK  ] Started dracut initqueue hook.
[  OK  ] Reached target Remote File Systems (Pre).
[  OK  ] Reached target Remote File Systems.
         Starting dracut pre-mount hook...
[    6.387049] systemd[1]: Started dracut initqueue hook.
[    6.387444] systemd[1]: Reached target Remote File Systems (Pre).
[    6.387869] systemd[1]: Reached target Remote File Systems.
[    6.388465] systemd[1]: Starting dracut pre-mount hook...
[  OK  ] Started dracut pre-mount hook.
[    6.416618] systemd[1]: Started dracut pre-mount hook.
[    6.667448] ignition[697]: GET https://raw.githubusercontent.com/coreos/fedora-coreos-config/testing-devel/tests/kola/ignition/remote/remote.ign: attempt 3
...

dustymabe · 2021-09-23T13:25:24Z

The test passes for me locally with FCOS and with the latest 4.9 RHCOS. What version string of RHCOS are you testing with?

Also maybe it has something to do with your environment not being able to get DNS somehow? To test that theory you could substitute the hostname in the url with an ip address.

dustymabe · 2021-09-23T13:26:46Z

tests/kola/ignition/remote/config.ign

+{
+  "ignition": {
+    "version": "3.3.0",
+    "config": {
+      "merge": [
+        {
+          "source": "https://raw.githubusercontent.com/coreos/fedora-coreos-config/testing-devel/tests/kola/ignition/remote/remote.ign",
+          "verification": {
+            "hash": "sha512-1c840823419a2eae431356b58d0c498f4ec84ef3d2b9a4fa42f75749a89fe1f413a848d9082d5dc6c243324b57fa7a76b4ef6dde5d023f9bba549b7755836170"
+          }
+        }
+      ]
+    }
+  }
+}


optional: for some reason I prefer these files to be in butane format (a little more easy to read and hack on if we need to add anything in the future). We could convert this to butane and rename it to config.bu.

tests/kola/ignition/remote/test.sh

HuijingHei · 2021-09-23T14:01:46Z

The test passes for me locally with FCOS and with the latest 4.9 RHCOS. What version string of RHCOS are you testing with?

Also maybe it has something to do with your environment not being able to get DNS somehow? To test that theory you could substitute the hostname in the url with an ip address.

I use the latest 4.9 RHCOS.
Test with https://185.199.110.133/coreos/fedora-coreos-config/testing-devel/tests/kola/ignition/remote/remote.ign, and get error: x509: cannot validate certificate for 185.199.110.133 because it doesn't contain any IP SANs, could you share your url?

dustymabe · 2021-09-23T14:16:39Z

I use the latest 4.9 RHCOS.
Test with https://185.199.110.133/coreos/fedora-coreos-config/testing-devel/tests/kola/ignition/remote/remote.ign, and get error: x509: cannot validate certificate for 185.199.110.133 because it doesn't contain any IP SANs, could you share your url?

yeah sorry. that was a bad idea on my part. Sorry for leading you down that path.

saqibali-2k

looks good 👍

dustymabe · 2021-09-23T14:36:18Z

summary of why it was failing for @HuijingHei on qemu - raw.githubusercontent.com is not accessible in china and thus the local qemu test from where she is was failing for obvious reasons.

HuijingHei · 2021-09-23T14:40:20Z

Thanks @dustymabe to find the root cause https://www.tutorialexample.com/fix-raw-githubusercontent-com-is-not-accessed-in-china-and-download-files/, will remove the related lines

This works with qemu-unpriv Co-authored-by: Dusty Mabe <dusty@dustymabe.com>

Remove exclusion for qemu-unpriv

tests/kola/ignition/remote/test.sh

dustymabe · 2021-09-23T16:59:07Z

Looks good to me but we need to squash everything down into a single commit

Co-authored-by: Dusty Mabe <dusty@dustymabe.com>

HuijingHei · 2021-09-24T02:48:32Z

Sorry I delete my forked repo by mistake. Make this duplicate of #1232

HuijingHei added 4 commits September 22, 2021 23:05

Add tests/kola/ignition/remote/remote.ign

449ccbe

The remote ignition file will be used to verify BZ1980679, including inject kernel arguments and write something to /etc/testfile

Merge branch 'coreos:testing-devel' into testing-devel

f30cc46

Merge branch 'coreos:testing-devel' into testing-devel

d57bded

Add test script for verify BZ1980679

547f701

config.ign to include remote.ign verify kernel arg and exists /etc/testfile

HuijingHei mentioned this pull request Sep 23, 2021

Add scripts to verify BZ1980679 #1230

Closed

travier previously approved these changes Sep 23, 2021

View reviewed changes

dustymabe reviewed Sep 23, 2021

View reviewed changes

tests/kola/ignition/remote/test.sh Outdated Show resolved Hide resolved

saqibali-2k previously approved these changes Sep 23, 2021

View reviewed changes

Remove exclusion for qemu-unpriv

c0554b1

This works with qemu-unpriv Co-authored-by: Dusty Mabe <dusty@dustymabe.com>

HuijingHei dismissed stale reviews from saqibali-2k and travier via c0554b1 September 23, 2021 14:52

HuijingHei added 4 commits September 23, 2021 23:01

Change config.ign to config.bu

2a27426

Remove exclusion for qemu-unpriv

Change config.ign to config.bu

1aa7b8c

Remove exclusion for qemu-unpriv

Update config.bu

a37592c

Add new line for config.bu

da780a2

dustymabe reviewed Sep 23, 2021

View reviewed changes

tests/kola/ignition/remote/test.sh Outdated Show resolved Hide resolved

HuijingHei and others added 2 commits September 24, 2021 09:00

Update tests/kola/ignition/remote/test.sh

1f4c827

Co-authored-by: Dusty Mabe <dusty@dustymabe.com>

Merge branch 'coreos:testing-devel' into testing-devel

9232d1a

HuijingHei closed this Sep 24, 2021

HuijingHei added the duplicate This issue or pull request already exists label Sep 24, 2021

Conversation

HuijingHei commented Sep 23, 2021

Uh oh!

HuijingHei commented Sep 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dustymabe commented Sep 23, 2021

Uh oh!

HuijingHei commented Sep 23, 2021

Uh oh!

travier left a comment

Choose a reason for hiding this comment

Uh oh!

HuijingHei commented Sep 23, 2021

Uh oh!

dustymabe commented Sep 23, 2021

Uh oh!

dustymabe Sep 23, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HuijingHei commented Sep 23, 2021

Uh oh!

dustymabe commented Sep 23, 2021

Uh oh!

saqibali-2k left a comment

Choose a reason for hiding this comment

Uh oh!

dustymabe commented Sep 23, 2021

Uh oh!

HuijingHei commented Sep 23, 2021

Uh oh!

Uh oh!

dustymabe commented Sep 23, 2021

Uh oh!

HuijingHei commented Sep 24, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HuijingHei commented Sep 23, 2021 •

edited

Loading