NSFS | S3 | Versioning: DeleteObjects response does not return Deleted record

[hseipp]

Environment info

• NooBaa Version: 5.15.4
• Platform: CentOS 8, deployed using https://github.com/IBM/StorageScaleVagrant

Actual behavior

DeleteObjects does not return Deleted record in a version-enabled bucket.

    @pytest.mark.fails_on_dbstore
    @pytest.mark.versioning
    def test_versioning_multi_object_delete_with_marker_create():
        client = get_client()
        bucket_name = get_new_bucket(client=client)
    
        check_configure_versioning_retry(bucket_name, "Enabled", "Enabled", client=client)
    
        key = 'key'
    
        # use delete_objects() to create a delete marker
        response = client.delete_objects(Bucket=bucket_name, Delete={'Objects': [{'Key': key}]})
>       assert len(response['Deleted']) == 1
E       KeyError: 'Deleted'

Expected behavior

DeleteObjects, even on non-existing keys should return a Deleted record and create a delete marker in a version-enabled bucket.

Steps to reproduce

Execute Ceph s3-tests test_versioning_multi_object_delete_with_marker_create(). See code snippet above.

More information - Screenshots / Logs / Other output

Log with all log level attached
noobaa_20240913_1.log.gz

[shirady]

Hi,
I will share that this issue was not reproduced in my NC POSIX environment and the tests passed and I could see in the logs the Deleted part:

<Deleted><Key>key</Key><DeleteMarker>true</DeleteMarker><DeleteMarkerVersionId>mtime-d48aueu7imf4-ino-2fvtyq</DeleteMarkerVersionId></Deleted></DeleteResult>

Looking at the attached logs I could observe:

<Error><Key>key</Key><Code>ENOENT</Code><Message>No such file or directory</Message></Error></DeleteResult>

And also:

NamespaceFS._get_version_info version of /ibm/fs1/examples/buckets/s3tests-0bb14099p169n3d3brnim-1/key doesn't exist [Error: No such file or directory] { code: 'ENOENT' }

The flow is: post_bucket_delete -> req.object_sdk.delete_multiple_objects -> namespace_fs function delete_multiple_objects -> _delete_objects_versioned ->
There we can see the printings:

core.sdk.namespace_fs:: NamespaceFS._delete_objects_versioned key [ undefined ]

( [ undefined ] is an array with one undefined item)

-> _delete_latest_version -> latest_ver_info is undefined after _get_version_info.
Since it is GPFS we used _open_files_gpfs - but it fails since the file doesn’t exist.

In PR #8358 It is suggested that we will use _open_files_gpfs only if latest_ver_info is defined.
With this fix, we would need to test this again on a GPFS machine.

[shirady]

This should be fixed after PR #8358 is merged.