[BUG] vmdk file is incomplete when downloaded from SDDC vsphere content library via govc libary.export

[mimcdevitt]

Describe the bug
vsphere vmdk file is incomplete/corrupted when downloaded from SDDC vpshere content library using via the govc library.export command.

To Reproduce
Steps to reproduce the behavior:

use govc to perform library.export of OVA from SDDC Vsphere content library.

$govc library.export CodeBuild/ubuntu-2204

Checking ubuntu-2204.nvram... PREPARED
[19-12-23 02:26:03] Downloading https://vcenter.sddc.vmwarevmc.com/cls/data/6aa4ffc4bd0a3545a125388a0cad8d1d/ubuntu-2204.nvram... OK
Checking ubuntu-2204.vmdk... PREPARED
[19-12-23 02:31:20] Downloading https://vcenter.sddc.vmwarevmc.com/cls/data/6aa4ffc4bd0a3545a125388a0cad8d1d/ubuntu-2204.vmd)... OK
Checking ubuntu-2204.ovf... PREPARED
govc: downloading https://vcenter.sddc.vmwarevmc.com/cls/data/6aa4ffc4bd0a3545a125388a0cad8d1d/ubuntu-2204.ovf: 404 Not Found

After download use qemu-ing to check validity of vmdk file

$qemu-img check ubuntu-2204.vmdk       
qemu-img: Could not open 'ubuntu-2204.vmdk': Invalid footer

Expected behavior
govc library.export function should download all components of a OVA and files should be complete.

Affected version
Tested with these versions
govc version 0.34 and 0.34.1
VMware SDDC Version: 1.22v6

Additional context
Initially we though the problem was related to govc-library export not exporting all files in OVA. When using govc to export a OVA from our content library we are getting a 404 error intermittently when the export runs. However when we look at the data store we can see all three files that make up the OVA. It appears that the issue is related to the vmdk file not downloading entirely. Any files in the OVA that come after the vmdk do not download. The downloaded vmdk is not complete but there is no error from govc to indicate that the vmdk file did not download in its entirely.

As part of our workflow we download the OVF, VMDK and NVRAM files from the SDDC Content Library and then pass them as inputs to a script which performs additional operations and then recreates a new OVA.

Since mid-December or so, we noticed that the OVAs being created from the exported files had shrunk in size compared to previous uploads (~6 GB before vs ~2 GB after). When inspecting the VMDK file that was exported we encountered the following error.

$ qemu-img check ubuntu-2204.vmdk       
qemu-img: Could not open 'ubuntu-2204.vmdk': Invalid footer

This indicates that the VMDK is corrupt and from the size reduction, we hypothesized that it’s possibly not being downloaded fully from the Content Library . To investigate if this was caused due to a vCenter content library session timeout issue, we monitored the govc export sessions using the following command

$ while true; do govc library.session.ls; done

Surprisingly, keeping this session monitoring alive during the VMDK download caused it to download the entire 6GB VMDK instead of cutting short at ~2 GB, and the downloaded VMDK had no errors when inspected. The later usage of this VMDK to build a new OVA also worked fine.

$ qemu-img check ubuntu-2204.vmdk
No errors were found on the image.

[github-actions]

Howdy 🖐   mimcdevitt ! Thank you for your interest in this project. We value your feedback and will respond soon.

If you want to contribute to this project, please make yourself familiar with the CONTRIBUTION guidelines.

[dougm]

Thank you for the detailed report @mimcdevitt
I've not tried to reproduce yet, but it does sound like expired session is the problem. The library.session.ls loop will keep the session alive, we just need to do so within the govc command. We do for library.import here:

govmomi/govc/library/import.go

Line 147 in a130044

cmd.KeepAlive(c)

Following patch adds the same for library.export, I'll take a closer look later and open a PR:

diff --git a/govc/library/export.go b/govc/library/export.go
index 45162144..c5f0fd82 100644
--- a/govc/library/export.go
+++ b/govc/library/export.go
@@ -84,6 +84,7 @@ func (cmd *export) Run(ctx context.Context, f *flag.FlagSet) error {
        if err != nil {
                return err
        }
+       cmd.KeepAlive(c)
 
        var names []string
        m := library.NewManager(c)

[mimcdevitt]

Thank you very much @dougm appreciate your help on this issue.