Destructive Update after addition of network device

[alex-ioma]

Describe the bug
If I create a VM with 1 network_device and I later installed other components manually, which in turn creates additional virtual net iface, the addition of a second network device via proxmox apply destroyes the other networks.

To Reproduce
Steps to reproduce the behavior:

1. Create one or more VM with 1 device terraform apply
2. Install additional components such a minikube or a k8s cluster from VM shell.
3. Run terraform plan and see that network modifications would be removed
4. Run terraform refresh and terraform plan and see that networks no longer be removed
5. Modify the VM by adding 1 additional network_device
6. Run terraform plan and see that interfaces will be destroyed

Please also provide a minimal Terraform configuration that reproduces the issue.

resource "proxmox_virtual_environment_vm" "k8s_worker" {
  name      = "test"
  count     = 3 # optional
  node_name = "pve1"
  on_boot   = true

  agent {
    enabled = true
  }

  cpu {
    cores   = 8
    sockets = 1
    type    = "x86-64-v4"
    units   = "100"
  }

  memory {
    dedicated = 16384
    floating  = 16384
  }

  scsi_hardware = "virtio-scsi-single"

  disk {
    datastore_id = "local-lvm"
    file_id      = proxmox_virtual_environment_download_file.image.id
    file_format  = "raw"
    interface    = "scsi0"
    iothread     = true
    discard      = "on"
    ssd          = true
    size         = 50
  }

    network_device {
    bridge  = "vmbr0"
    vlan_id = "1008"
  }

  network_device {
    bridge = "vmbr1"
    mtu    = 9000
  }

  boot_order = ["scsi0"]

}

Expected behavior
The additions of a network device is performed correctly.

Additional context
Possibly the issues lies in the fact that you cannot set if the device is added as eth0 or eth1. As TF doesn't know how to prioritize the net devices, it might remove first the virtual so that it can freely name the devices or might be waiting for a name from qemu agent.

Could the solution be as simple as adding a parameter name inside network_device so that during VM creation / update TF could perform something similar to this?

qm create 101 --name "my-vm" --net0 model=virtio,bridge=vmbr0,name=eth0 --net1 model=e1000,bridge=vmbr1,name=eth1

or change the network block in an array?

[alex-ioma]

Tested on v0.64.0

[bpg]

Hey @alex-ioma 👋🏼

I'm not quite following your steps...

2. Install additional components such a minikube or a k8s cluster from VM shell.
3. Run terraform plan and see that network modifications would be removed.

Not sure what does it do to the VM. Could you provide the output of terraform apply on each step?

[alex-ioma]

Hi @bpg 👋 Apologies.

What I meant was simply the fact that, starting from VMs deployed with the first apply, if you then manually install kubernetes (or any other components that add additional virtual interface to said VMs), a subsequent apply (step 3) would try to remove the changes (i.e. destroy the additional network interfaces, which would be expected).

At this point, if you refresh, a subsequent apply would show the VMs as in sync and no longer divergent (as expected).

However, if you now add another network interface, a terraform apply would destroy interfaces that are instead in-sync, in the attempt to add eth1. See output below:

Terraform used the selected providers to generate the following execution plan. Resource
actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # proxmox_virtual_environment_vm.k8s_worker[0] will be updated in-place
  ~ resource "proxmox_virtual_environment_vm" "k8s_worker" {
        id                      = "111"
      ~ ipv4_addresses          = [
          - [
              - "127.0.0.1",
            ],
          - [
              - "10.1.8.21",
            ],
          - [],
          - [],
          - [
              - "10.42.3.0",
            ],
          - [],
          - [],
          - [],
          - [],
          - [],
          - [],
        ] -> (known after apply)
      ~ ipv6_addresses          = [
          - [
              - "::1",
            ],
          - [
              - "fe80::be24:11ff:xxxx:3c05",
            ],
          - [
              - "fe80::ecee:eeff:feee:eeee",
            ],
          - [
              - "fe80::ecee:eeff:feee:eeee",
            ],
          - [
              - "fe80::xxx:57ff:fed0:82dc",
            ],
          - [
              - "fe80::ecee:eeff:feee:eeee",
            ],
          - [
              - "fe80::ecee:eeff:feee:eeee",
            ],
          - [
              - "fe80::ecee:eeff:feee:eeee",
            ],
          - [
              - "fe80::ecee:eeff:feee:eeee",
            ],
          - [
              - "fe80::ecee:eeff:feee:eeee",
            ],
          - [
              - "fe80::ecee:eeff:feee:eeee",
            ],
        ] -> (known after apply)
        name                    = "k8s-n1"
      ~ network_interface_names = [
          - "lo",
          - "eth0",
          - "cali46xxxxx356d",
          - "calibde260dxxxb",
          - "flannel.1",
          - "cali3xxxxxdca9fc",
          - "cali16dxxxxa0c76",
          - "cali7d8xxxx48e2d",
          - "cali4xxx5b7f536d",
          - "cali102fxxxxxx445",
          - "calif42xxxxxxx1b4",
        ] -> (known after apply)
        tags                    = [
            "k8s",
            "terraform",
            "worker",
        ]
        # (27 unchanged attributes hidden)

      + network_device {
          + bridge     = "vmbr1"
          + enabled    = true
          + firewall   = false
          + model      = "virtio"
          + mtu        = 9000
          + queues     = 0
          + rate_limit = 0
          + vlan_id    = 0
        }

        # (7 unchanged blocks hidden)
    }

[bpg]

Ah, I see. You're right about:

... you cannot set whether the device is added as eth0 or eth1.

This is a key limitation of the current provider's design. It treats network devices as a list rather than a map, so any changes in the list order are likely to result in the removal and re-creation of one or more devices. Unfortunately, this is not easy to fix in the current implementation. This will be addressed in v1.0 (#1231) with major changes to the config format.