Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

edge-commit CS9 failure: ostree-remount.service fails to remount /sysroot and /var mountpoints #3942

Open
mcattamoredhat opened this issue Feb 13, 2024 · 1 comment

Comments

@mcattamoredhat
Copy link
Contributor

mcattamoredhat commented Feb 13, 2024

Describe the bug
During edge-commit test case in CS9, our CI has detected that ostree-remount.service fails to remount /sysroot and /var mountpoints.
The output of the playbook task that checks journalctl of ostree-remount does not include the expected output Remounted rw: /sysroot nor Remounted rw: /var

TASK [assert] ******************************************************************
fatal: [192.168.100.50]: FAILED! => {
    "assertion": "'/sysroot' in result_remount_jounalctl.stdout",
    "changed": false,
    "evaluated_to": false,
    "msg": "/sysroot or /var are not remounted by ostree-remount"
}

Inside edge vm:

[admin@vm ~]$ sudo journalctl -u ostree-remount
Feb 12 18:16:57 localhost systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Feb 12 18:16:57 localhost systemd[1]: Finished OSTree Remount OS/ Bind Mounts.
-- Boot 750cc61b1dcf4a7597e93bc937705d6b --
Feb 12 18:17:12 localhost systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Feb 12 18:17:12 localhost systemd[1]: Finished OSTree Remount OS/ Bind Mounts.
-- Boot 102a413dd5714e5a8af5ce5b4e394f78 --
Feb 12 18:19:20 localhost systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Feb 12 18:19:20 localhost systemd[1]: Finished OSTree Remount OS/ Bind Mounts.
-- Boot 12671331afda4e9489744c5e3488f319 --
Feb 12 18:19:35 localhost systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Feb 12 18:19:35 localhost systemd[1]: Finished OSTree Remount OS/ Bind Mounts.
-- Boot 59099ebb918c4e0a8176227730157c34 --
Feb 12 18:19:50 localhost systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Feb 12 18:19:50 localhost systemd[1]: Finished OSTree Remount OS/ Bind Mounts.
-- Boot 519566393b414d16862bfc84599bbb30 --
Feb 12 18:20:03 localhost systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Feb 12 18:20:03 localhost systemd[1]: Finished OSTree Remount OS/ Bind Mounts.
Feb 12 18:37:27 vm systemd[1]: ostree-remount.service: Deactivated successfully.
Feb 12 18:37:27 vm systemd[1]: Stopped OSTree Remount OS/ Bind Mounts.
Feb 12 18:37:27 vm systemd[1]: Stopping OSTree Remount OS/ Bind Mounts...
Feb 12 18:37:27 vm systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
Feb 12 18:37:27 vm systemd[1]: Finished OSTree Remount OS/ Bind Mounts.
[admin@vm ~]$ sudo journalctl -xeu ostree-remount
~
~
Feb 12 18:20:03 localhost systemd[1]: Starting OSTree Remount OS/ Bind Mounts...
░░ Subject: A start job for unit ostree-remount.service has begun execution
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░ 
░░ A start job for unit ostree-remount.service has begun execution.
░░ 
░░ The job identifier is 159.
Feb 12 18:20:03 localhost systemd[1]: Finished OSTree Remount OS/ Bind Mounts.
░░ Subject: A start job for unit ostree-remount.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://access.redhat.com/support
░░ 
░░ A start job for unit ostree-remount.service has finished successfully.
░░ 
░░ The job identifier is 159.

The service seems to be running but it does not work even after manual restart.

Environment

  • OS version (/etc/os-release and /etc/redhat-release):
 + source /usr/libexec/osbuild-composer-test/set-env-variables.sh
++ source /etc/os-release
+++ NAME='CentOS Stream'
+++ VERSION=9
+++ ID=centos
+++ ID_LIKE='rhel fedora'
+++ VERSION_ID=9
+++ PLATFORM_ID=platform:el9
+++ PRETTY_NAME='CentOS Stream 9'
+++ ANSI_COLOR='0;31'
+++ LOGO=fedora-logo-icon
+++ CPE_NAME=cpe:/o:centos:centos:9
+++ HOME_URL=https://centos.org/
+++ BUG_REPORT_URL=https://bugzilla.redhat.com/
+++ REDHAT_SUPPORT_PRODUCT='Red Hat Enterprise Linux 9'
+++ REDHAT_SUPPORT_PRODUCT_VERSION='CentOS Stream'
  • osbuild-composer version (rpm -qi osbuild-composer)
 osbuild-composer                         x86_64  101-1.20240212git643be1c.el9      osbuild-composer   17 k
 osbuild-composer-core                    x86_64  101-1.20240212git643be1c.el9      osbuild-composer   10 M
 osbuild-composer-dnf-json                x86_64  101-1.20240212git643be1c.el9      osbuild-composer   11 k
 osbuild-composer-worker                  x86_64  101-1.20240212git643be1c.el9      osbuild-composer   17 M

To Reproduce
Steps to reproduce the behavior:

Expected behavior
ostree-remount.service remounts successfully /sysroot and /var mountpoints.

Additional context
In the steps to reproduce #3907 been used as an example. Although, this failure has been detected so far in several PRs such as:
#3928
#3939
#3828
#3907
#3886
#3934

@cgwalters
Copy link
Contributor

Any test that's parsing output from a tool that's intended solely for human consumption is going to be fragile.

our CI has detected that ostree-remount.service fails to remount /sysroot and /var mountpoints.

But are the mounts actually not in the expected state? That's what the test should be checking right?

As noted in private chat I think what's going on here is we generally expect the mounts setup in the initramfs nowadays, not in this service.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants