IndexError: list index out of range using virtnbdbackup tool

[damirsucic]

Version used
Provide output of virtnbdbackup -V
/usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.
from cryptography.hazmat.backends import default_backend
2.4

Describe the bug
When I want to create full backup (on Oracle KVM host) with this command:
virtnbdbackup -U qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf -d rocky92 -l full -o /tmp/backupset/rocky92
I get these errors below (Logfiles section) and no backupset in /tmp/backupset/rocky92 directory

Expected behavior
I except to get full backup set in /tmp/backupset/rocky92 directory.

Hypervisor information:

• OS: Oracle Linux 8.9
• HV type: Oracle Linux Virtualization Manager (OLVM)

Logfiles:
[root@okvm2 ~]# virtnbdbackup -U qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf -d rocky92 -l full -o /tmp/backupset/rocky92
/usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.
from cryptography.hazmat.backends import default_backend
[2024-03-28 12:27:37] INFO lib common - printVersion [main]: Version: 2.4 Arguments: /bin/virtnbdbackup -U qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf -d rocky92 -l full -o /tmp/backupset/rocky92
[2024-03-28 12:27:37] INFO root virtnbdbackup - main [main]: Backup level: [full]
IndexError: list index out of range

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/bin/virtnbdbackup", line 481, in
main()
File "/bin/virtnbdbackup", line 312, in main
virtClient = virt.client(args)
File "/usr/local/lib/python3.6/site-packages/virtnbdbackup-2.4-py3.6.egg/libvirtnbdbackup/virt/client.py", line 55, in init
File "/usr/local/lib/python3.6/site-packages/virtnbdbackup-2.4-py3.6.egg/libvirtnbdbackup/virt/client.py", line 136, in _connect
File "/usr/local/lib/python3.6/site-packages/virtnbdbackup-2.4-py3.6.egg/libvirtnbdbackup/virt/client.py", line 82, in _connectAuth
File "/usr/lib64/python3.6/site-packages/libvirt.py", line 145, in openAuth
ret = libvirtmod.virConnectOpenAuth(uri, auth, flags)
SystemError: returned a result with an error set

Workaround:
No workarounds till now.

[abbbi]

hi,

thanks for the report. havent used it with authfile since a long time.
Since i currently have no RHEV/OLVM setup, could you try with the following fix i pushed:

52ffc6f

thanks

[abbbi]

should work with master checkout

[abbbi]

hi again,

i managed to get hands on some RHEV node. You can also use the remote backup functionality with the latest master version, so you dont have to install the backup utility on the hypervisor host itself (which can be a struggle if its the stripped down node installation).

Ive added something to README:

https://github.com/abbbi/virtnbdbackup?tab=readme-ov-file#ovirt-rhev-or-olvm

however, both the local auth via credential file and remote auth against libvirt using qemu+ssh and user/password combination should work now.

[damirsucic]

Hi Abbbi,
thanks for your great work.
I applied both, fix 167 (52ffc6f) and master. First tested 52ffc6f, and after that master. In both cases I got different output than before:
[root@okvm2 virtnbdbackup-2.4]# virtnbdbackup -U qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf -d rocky92 -l full -o /tmp/backupset/rocky92
/usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.
from cryptography.hazmat.backends import default_backend
[2024-03-29 12:00:06] INFO lib common - printVersion [main]: Version: 2.4 Arguments: /bin/virtnbdbackup -U qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf -d rocky92 -l full -o /tmp/backupset/rocky92
[2024-03-29 12:00:06] INFO root virtnbdbackup - main [main]: Backup level: [full]
[2024-03-29 12:00:06] ERROR root virtnbdbackup - main [main]: Can't connect libvirt daemon: [authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)]

But as you can see it still doesn't work. In meanwhile I tried to check my virtual machines in my OLVM manager and I'm not able to access it anymore.
This is testing environment, and a lot of testing goes there (for example one host removed recently), so in this moment I'm not able to say what caused the "OLVM-VirtualMachines-NoDisplay" problem.

If I see correctly you didn't apply this 167 fix to 2.5 version - just to 52ffc6f and master branches, am I right? By change log I would say you will apply 167 in 2.6.

[abbbi]

[2024-03-29 12:00:06] ERROR root virtnbdbackup - main [main]: Can't connect libvirt daemon: [authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)]

it seems the libvirt daemon on OLVM needs some more authentication than just the authfile..
does it work with following command?:

[root@ovm-node1 ~]# virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf
Welcome to virsh, the virtualization interactive terminal.
[..]

all fixes currently in master

[damirsucic]

Yes, I tried this before, and just right now at your suggestion:

[root@okvm2 virtnbdbackup-2.4]# virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf
Welcome to virsh, the virtualisation interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # 

[abbbi]

this is strange..
does it work using python directly?

python3
Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvirt
>>> libvirt.open('qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf')

the current master version should to exactly this.
if thats working for you, please run virtnbdbackup (master checkout) with -v option so i can see whats going on.
Unfortunately i dont have access to an olvm so i cant test ..

[damirsucic]

It doesn't work, my python3 output:

[root@okvm2 virtnbdbackup-master]# python3
Python 3.6.8 (default, Jan 16 2024, 02:08:59) 
[GCC 8.5.0 20210514 (Red Hat 8.5.0-20.0.1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import libvirt
>>> 
>>> 
>>> libvirt.open('qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf')
libvirt: XML-RPC error : authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.6/site-packages/libvirt.py", line 335, in open
    raise libvirtError('virConnectOpen() failed')
libvirt.libvirtError: authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
>>> 

[abbbi]

Ok then thats an issue on the hypervisor at the Moment.. Maybe .. only if this Uri works with Python you could Make backup work

Or try with —User and —Passwort options

[damirsucic]

Abbbi I fixed my OLVM not displaying VMs, but unfortunately I still have the same issue regardless authentication I used.
With user/pass, my output is the same:

[root@okvm2 virtnbdbackup-master]# ./virtnbdbackup --user root --password 'MY_PASS_CHANGED_HERE' -d rocky92 -l full -o /tmp/backupset/rocky92
/usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.
  from cryptography.hazmat.backends import default_backend
[2024-03-29 15:52:47] INFO lib common - printVersion [main]:  Version: 2.6 Arguments: ./virtnbdbackup --user root --password Am(4=ass -d rocky92 -l full -o /tmp/backupset/rocky92
[2024-03-29 15:52:47] INFO root virtnbdbackup - main [main]:  Backup level: [full]
[2024-03-29 15:52:47] ERROR root virtnbdbackup - main [main]:  Can't connect libvirt daemon: [authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)]

[abbbi]

I dont know.. Maybe try Remote Backup.

[damirsucic]

Unfortunately Remote Backup doesn't work either:

[root@kvmmgmt .ssh]# virtnbdbackup -V
2.6
[root@kvmmgmt .ssh]# virtnbdbackup -U qemu+ssh://root@okvm2 --ssh-user root -d rocky92 -o  /tmp/backupset/rocky92
[2024-03-29 21:13:37] INFO lib common - printVersion [main]:  Version: 2.6 Arguments: /bin/virtnbdbackup -U qemu+ssh://root@okvm2 --ssh-user root -d rocky92 -o /tmp/backupset/rocky92
[2024-03-29 21:13:37] INFO root virtnbdbackup - main [main]:  Backup level: [copy]
[2024-03-29 21:13:38] ERROR root virtnbdbackup - main [main]:  Can't connect libvirt daemon: [authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)]
[root@kvmmgmt .ssh]# 

[abbbi]

sorry, i dont know. Maybe reach out to ovirt or OLVM support forums. It seems OLVM libvirt is denying SASL based authentication. Cant tell why it doesnt work with the auth file locally, it does with my (older) RHEV setup. It seems to me OLVM may have has disabled authentication for other componens than virsh. Thats certainly not default behavior on other libvirt based setups.

Might also be an certificate issue:

dmacvicar/terraform-provider-libvirt#810
https://stackoverflow.com/questions/72680442/sasl-error-connecting-to-remote-libvirt-over-ssh-no-worthy-mechs-found

At the moment i dont have the time and energy to setup an complete OLVM myself to track down
whats wrong here.

As long as you cant make the autentication work as with:

#167 (comment)

i dont see a way youre able to use virtnbdbackup here.

[abbbi]

hi again,

the cause for the connection issue

Can't connect libvirt daemon: [authentication failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)]

is the same as reported for the terraform provider:

dmacvicar/terraform-provider-libvirt#810

you need to include the hostname that is used for the libvirt certificate in your connection URI.

I used the following command on a test OLVM system to get the hostname from the certificate:

openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -text
Subject: O=srv.test.local, CN=ovm-node1.test.local

i could then connect libvirt locally using python via:

python3
>>> import libvirt
>>> libvirt.open('qemu://ovm-node1.test.local/system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf')
<libvirt.virConnect object at 0x7ff8b46c9110>

means you need to include the hostname that is set in the certificate in the connection uri passed as -U option (for both remote or local backup) otherwise libvirtd cant match the certificate and denies SASL based authentication methods.

[abbbi]

the cause for the initial issue should be fixed as by now. Closing ticket -> if further discussion required, please open a discussion.