[Bug]: Bigquery Load jobs with WRITE_TRUNCATE disposition may truncate valid records.

[robertwb]

What happened?

TriggerCopyJobs was written to assume all jobs belonging to the same table are processed by the same bundle, but this precondition appears to have been invalidated with future changes to the code. Though this still seems to hold most of the time, it may be violated for large writes, and seems to trigger more on runner v2 than runner v1.

Issue Priority

Priority: 1

Issue Component

Component: io-py-gcp

[kennknowles]

Is this a regression? It sounds severe and worth a cherry pick most likely.

[robertwb]

I don't know at what point this regression was introduced, but it is pretty severe and I think worth a cherry-pick.

…

On Tue, Dec 6, 2022 at 7:24 AM Kenn Knowles ***@***.***> wrote: Is this a regression? It sounds severe and worth a cherry pick most likely. — Reply to this email directly, view it on GitHub <#24535 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADWVAO6XBEP3HZUHCHILYTWL5LDLANCNFSM6AAAAAASU3MJEM> . You are receiving this because you were assigned.Message ID: ***@***.***>

[kennknowles]

I see it is first reported by a user in #23306. I think keeping the earlier bug is worthwhile so I will dupe this one.

[Abacn]

The bug can still occur when the number of files are greater than 10,000 then bigquery load jobs will be conducted in multiple partitions:

beam/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py

Line 70 in 428ec97

_MAXIMUM_SOURCE_URIS = 10 * 1000

Did a test that decrease this number and force the write happens in multiple bigquery load jobs, still see dropping elements

[Abacn]

reopen for cherry pick