Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent integer overflow when reading large blobs #1093

Closed
wants to merge 1 commit into from

Conversation

tbeatty
Copy link

@tbeatty tbeatty commented Jun 30, 2016

The position instance variable of BlobReadChannel overflows when reading blobs with size larger than Integer.MAX_VALUE. This results in a request against the storage api with an invalid Range header.

The api ignores the invalid range header, which results in the the entire file being downloaded. When an array is allocated to hold the contents of the file, the runtime throws an OutOfMemoryError.

The issue can be reproduced as follows:

  1. Create a large random file

    $ dd if=/dev/urandom of=large_file bs=1M count=3072
    
  2. Confirm the file size is greater than Integer.MAX_VALUE

    $ du -h large_file
    3.1G  large_file
    
  3. Create a bucket and upload the file

    $ gsutil mb gs://gcloud-java
    $ gsutil cp large_file gs://gcloud-java
    
  4. Use com.google.cloud.examples.storage.StorageExample to download the file

    $ mvn exec:java -Dexec.mainClass="com.google.cloud.examples.storage.StorageExample" -Dexec.args="my-gcloud-project download gcloud-java large_file large_file_copy"
    

Result:

[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO]   ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building GCloud Java examples 0.2.4
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ gcloud-java-examples ---
[WARNING]
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2271)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
    at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:55)
    at com.google.api.client.util.IOUtils.copy(IOUtils.java:94)
    at com.google.api.client.util.IOUtils.copy(IOUtils.java:63)
    at com.google.api.client.http.HttpResponse.download(HttpResponse.java:421)
    at com.google.cloud.storage.spi.DefaultStorageRpc.read(DefaultStorageRpc.java:474)
    at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:127)
    at com.google.cloud.storage.BlobReadChannel$1.call(BlobReadChannel.java:124)
    at com.google.cloud.RetryHelper.doRetry(RetryHelper.java:179)
    at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:244)
    at com.google.cloud.storage.BlobReadChannel.read(BlobReadChannel.java:124)
    at com.google.cloud.examples.storage.StorageExample$DownloadAction.run(StorageExample.java:387)
    at com.google.cloud.examples.storage.StorageExample$DownloadAction.run(StorageExample.java:365)
    at com.google.cloud.examples.storage.StorageExample$DownloadAction.run(StorageExample.java:361)
    at com.google.cloud.examples.storage.StorageExample.main(StorageExample.java:830)
    ... 6 more
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17:20 min
[INFO] Finished at: 2016-06-30T10:29:02-03:00
[INFO] Final Memory: 13M/2026M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java (default-cli) on project gcloud-java-examples: An exception occured while executing the Java class. null: InvocationTargetException: Java heap space -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

@googlebot
Copy link

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.


  • If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
  • If you signed the CLA as a corporation, please let us know the company's name.

@googlebot googlebot added the cla: no This human has *not* signed the Contributor License Agreement. label Jun 30, 2016
@coveralls
Copy link

Coverage Status

Coverage remained the same at 86.153% when pulling b4e3d29 on tbeatty:master into 1118c0e on GoogleCloudPlatform:master.

@mziccard
Copy link
Contributor

mziccard commented Jul 1, 2016

Hi @tbeatty,

thank you for helping out!
We are aware of this issue and in fact we fixed it in a bigger PR. I plan to merge that PR soon and release a new version of gcloud-java soon. I'll keep you posted.

@mziccard mziccard added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Jul 1, 2016
@mziccard
Copy link
Contributor

mziccard commented Jul 4, 2016

@tbeatty this issue has been fixed and a new version (0.2.5) of gcloud-java released. Thanks again for this PR!

@mziccard mziccard closed this Jul 4, 2016
@tbeatty
Copy link
Author

tbeatty commented Jul 4, 2016

@mziccard I had glanced over that PR but didn't notice #1065. Thanks for the response!

github-actions bot pushed a commit that referenced this pull request Oct 5, 2022
🤖 I have created a release *beep* *boop*
---


### Updating meta-information for bleeding-edge SNAPSHOT release.

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: no This human has *not* signed the Contributor License Agreement. do not merge Indicates a pull request not ready for merge, due to either quality or timing.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants