Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ACL errors for newly created and pre-existing blobs #1016

Merged
merged 3 commits into from
Jan 8, 2023

Conversation

RachitSharma2001
Copy link
Contributor

@RachitSharma2001 RachitSharma2001 commented Dec 22, 2022

This fixes #944 and fixes #945.

The following two python snippets no longer crash and instead give the correct output. The first one is for a pre-existing blob (thus fixing #944):

os.environ["STORAGE_EMULATOR_HOST"] = "http://0.0.0.0:4443"

client = storage.Client(
    credentials=AnonymousCredentials(),
    project="test-project",
)

# initial bucket/file for docker image
bucket_name = "sample-bucket"
blob_name = "some_file.txt"

# test
bucket = client.get_bucket(bucket_name)
blob = bucket.blob(blob_name)
print(list(blob.acl))
blob.make_public() 
print(list(blob.acl))
blob.make_private()
print(list(blob.acl))

The outputs:
[{'entity': 'projectOwner-test-project', 'role': 'OWNER'}]
[{'entity': 'projectOwner-test-project', 'role': 'OWNER'}, {'entity': 'allUsers', 'role': 'READER'}]
[{'entity': 'projectOwner-test-project', 'role': 'OWNER'}]

The second snippet shows how for a newly created blob, the acl's also update when make_public() and make_private() are called:

os.environ["STORAGE_EMULATOR_HOST"] = "http://0.0.0.0:4443"

client = storage.Client(
    credentials=AnonymousCredentials(),
    project="test-project",
)

# buckets/files to create and test
upload_bucket_name = "test-bucket-with-globally-unique-name"
upload_blob_name = "test-blob-upload.svg"
upload_blob_file_path = "./image.svg"

# initialize
try:
    bucket = client.bucket(upload_bucket_name)
    bucket.storage_class = storage.constants.STANDARD_STORAGE_CLASS
    client.create_bucket(bucket, location="EU", retry=None)
except:
    pass
bucket = client.get_bucket(upload_bucket_name)
blob = bucket.blob(upload_blob_name)
blob.upload_from_filename(upload_blob_file_path, retry=None)

# test
bucket = client.get_bucket(upload_bucket_name)
blob = bucket.blob(upload_blob_name)
print(list(blob.acl))
blob.make_public()
print(list(blob.acl)) 
blob.make_private()
print(list(blob.acl))

The outputs:
[{'entity': 'projectOwner-test-project', 'role': 'OWNER'}]
[{'entity': 'projectOwner-test-project', 'role': 'OWNER'}, {'entity': 'allUsers', 'role': 'READER'}]
[{'entity': 'projectOwner-test-project', 'role': 'OWNER'}]

Explanation of my implementation:

For fixing #944:
Within main.go, when it reads all the existing files within the bucket, it previously never set an ACL for any of these objects. I assumed that for these pre-existing blobs, the ACL for each of them would just be [{'entity': 'projectOwner-test-project', 'role': 'OWNER'}].

For fixing #945:
From looking at how the python API updates the ACL's, I found that rather than sending a POST to the endpoint /b/{bucketName}/o/{objectName:.+}/acl, it instead sends a PATCH request to the endpoint /b/{bucketName}/o/{objectName:.+}.

Thus, I needed to update the patchObject method within fakestorage/object.go to detect if new ACL's are passed in and update the object's ACL if so.

Role string
}

var dataInBody struct {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can we call this payload?

Suggested change
var dataInBody struct {
var payload struct {

Comment on lines 954 to 964
backendObj, err := s.backend.PatchObject(bucketName, objectName, dataInBody.Metadata)
if len(dataInBody.Acl) > 0 {
backendObj.ACL = []storage.ACLRule{}
for _, aclData := range dataInBody.Acl {
newAcl := storage.ACLRule{Entity: storage.ACLEntity(aclData.Entity), Role: storage.ACLRole(aclData.Role)}
backendObj.ACL = append(backendObj.ACL, newAcl)
}
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm this won't work with the backend filesystem, will it? We're modifying the value in memory and not persisting it. We need to update the PatchObject signature to take something like "attrsToUpdate" and apply changes to other attributes, not just metadata (this is necessary for #1024 too).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah good point. I will fix this and push the changes.

Copy link
Contributor Author

@RachitSharma2001 RachitSharma2001 Jan 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have made the requested change. Let me know if what I did was best practice, and if there is any changes needed.

main.go Outdated
"github.com/fsouza/fake-gcs-server/fakestorage"
"github.com/fsouza/fake-gcs-server/internal/checksum"
"github.com/fsouza/fake-gcs-server/internal/config"
"github.com/sirupsen/logrus"
)

var DefaultACL = []storage.ACLRule{
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't really need to be a global, let's inline it (I know it means we'll repeat it in tests, but that's OK).

@RachitSharma2001 RachitSharma2001 force-pushed the fix_944_945 branch 4 times, most recently from 7e056dd to 681a93a Compare January 1, 2023 22:33
Copy link
Owner

@fsouza fsouza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this! Just one nit on generalizing some code, otherwise it's ready to go.

If you prefer, I can take care of it.

internal/backend/fs.go Outdated Show resolved Hide resolved
internal/backend/fs.go Outdated Show resolved Hide resolved
Copy link
Owner

@fsouza fsouza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much for contributing!

@fsouza fsouza merged commit 34afa14 into fsouza:main Jan 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants