Batch requests generating multiple response codes for each request

[BjornPrime]

Batch requests (using Batch._responses to access the response data) are returning multiple responses for each request. 4/request for copy requests, 2/request for deletes. Not sure if other methods are affected.

When raise_exception=False, this results in exceptions not being associated with the right request, or not being registered at all.

Refer to example here.

Basically, the first line of text in the quote block is printing length of requests and then length of responses. The next five lines are printing every response code associated with a particular item (four for each). The final five lines show the results as read by our implementation, notably missing the 404s thrown by the fourth item.

Obviously, we can fix our particular implementation to account for this, but thought y'all might want to know that something seems to be up with the batch responses, especially as this line seems to indicate an expectation that each request only have one response.

[cojenco]

Hi @BjornPrime, I took a look at the linked example. IIUC, the reason you are seeing 4/request for copy requests is because there are multiple operation calls being made in the batch context - get_bucket() get_blob() in addition to copy_blob()

Likewise for the delete example, replace get_bucket() API call with simply instantiating the bucket object via client.bucket(bucket_name)

Consider changing your code block to something along the lines of:

with current_batch:
    for pair in current_pairs:
        src_bucket_name, src_blob_name = parse_gcs_path(pair[0])
        dest_bucket_name, dest_blob_name = parse_gcs_path(pair[1])

        # Instantiate a bucket object instead of making a GET call
        src_bucket = self.client.bucket(src_bucket_name)
        # Instantiate a blob object instead of making a GET call
        src_blob = src_bucket.blob(src_blob_name)
        # Same here
        dest_bucket = self.client.bucket(dest_bucket_name)

        # Only one API call corresponds to one subresponse
        src_bucket.copy_blob(src_blob, dest_bucket, dest_blob_name)

[BjornPrime]

Will that code block replicate the behavior of the current implementation?

[cojenco]

Yes it should; I'm assuming the objective is to copy the blob. It's important that only the copy_blob API call is wrapped in the batch context. The changes are mainly removing the extra API calls.

[BjornPrime]

Okay, I think I was just concerned about making sure the right buckets and blobs were identified but it's not surprising to me if just passing the correctly named objects to the client allows it to figure that part out on its own.

[cojenco]

Right, hope this clarifies your question. I'm closing this issue for now. Feel free to reopen if you have further questions.