Document bigquery.arrow-serialization.max-allocation property
#26409
Conversation
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
ebyhr
force-pushed
the
docs-bigquery-arrow-config
branch
from
f8dfb50 to
47ccf1c
Compare
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
ebyhr
changed the title
bigquery.arrow-serialization.max-allocation property
ebyhr
left a comment
ebyhr
left a comment
There was a problem hiding this comment.
Have you already submitted CLA?
Yes, CLA has been submitted from my end today. Waiting for the team to review and approve. |
|
@cla-bot check |
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
|
The cla-bot has been summoned, and re-checked this pull request! |
|
@cla-bot check |
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
|
The cla-bot has been summoned, and re-checked this pull request! |
|
My CLA doc has been rejected as they need physical sign. I will resubmit it today. |
|
This pull request has gone a while without any activity. Ask for help on #core-dev on Trino slack. |
|
@cla-bot check |
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
|
The cla-bot has been summoned, and re-checked this pull request! |
|
@archangel2018 it takes quite long time, could you check if you sent right username and email? |
|
@cla-bot check |
|
The cla-bot has been summoned, and re-checked this pull request! |
4 participants
Description
This pull request updates the documentation for the BigQuery connector to include the
bigquery.arrow-serialization.max-allocationproperty. This property allows users to increase the maximum memory usage for the Apache Arrow buffer allocator, which can help prevent memory leaks and out-of-memory errors.Additional context and related issues
The issue is linked to a bug in the BigQuery connector's Arrow serialization process. Despite ample memory resources on the worker nodes, the Arrow allocator would fail to reserve memory, leading to an
OutOfMemoryException. Disabling Arrow serialization (bigquery.arrow-serialization.enabled=false) resolved the issue but sacrificed performance. This fix allows us to keep the performance benefits of Arrow serialization while preventing memory allocation failures.Fixes #26400
Release notes
(x) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text: