fix(bigquery): create read session with client or job projectID

[alvarowolfx]

When reading result sets using the Storage Read API Acceleration enabled, currently the read session is created by default in the table's project. This works for cases where the destination table is not specified and automatically created, which defaults to the project where the the query or job was created. But when reading a table directly or specifying a destination table, it doesn't work in cases where the client doesn't have BQ Storage permissions (just table read permission for example). This is a common use case where some customers have a main billing project and this project has access to other GCP projects with just permission to read data from BigQuery tables.

With this PR, we default to use the defined Query/Job projectID (which defaults to the current bigquery.Client.projectID or when reading the a table directly, we also use default to the bigquery.Client.projectID.

Reported initially on PR #10924

Supersedes #10924

[alvarowolfx]

@juanli16 @fsaintjacques I think this should have been the default behavior in the first place, which solves for your use cases and doesn't require to add overrides. Let me know if makes sense.

[fsaintjacques]

@alvarowolfx I don't mind the fix since it won't break our use case, but beware that this might break (due to IAM) the existing users of this library.

[alvarowolfx]

@fsaintjacques thanks for raising the concern. I think was a mistake in the first place to use the table projectID to create session, as for the basic use case of reading public datasets, would fail with this approach.

Other than that, I think I can see two major groups here:

• For users using through the Query.Read and Job.Read paths, I don't think is going to break anything as the destination table project was already the same as the one set on the Query or Job.
• For users reading a table directly (or setting a destination table on a query), I feel like most users doing that was actually facing the same problem as you're facing, where they didn't have permission to create BigQuery Storage Read sessions. All customers that I worked with have a similar set up that you described, a centralized billing project ( set on the client ) and can read result sets on other projects with just table read permissions.
  • It would break for users that have BQ Storage permission on the table being read but not on the project set on the client (which sounds like an edge case)

Do you see any other scenario that would break ?

In any case, I think we can move with PR #10924 in parallel, to allow users to set the project used to create sessions.

[fsaintjacques]

Nope, I don't foresee any other issues. I wanted to warn you that this was potentially breaking some existing (un)lucky setup.