Skip to content

Remove hack from BigQuery DTS hook #23887

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 4 commits into from
Jun 1, 2022
Merged

Remove hack from BigQuery DTS hook #23887

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
51291b9
remove 'hack' on bigquery dts hook
henriqueribeiro May 24, 2022
8f8e682
fix version of proto-plus for google
henriqueribeiro May 26, 2022
7eecaaa
Explain the proto-plus constraint
uranusjr May 27, 2022
93b29e3
Merge branch 'main' into fix-remove-hack-bigquery-dts
uranusjr May 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions airflow/providers/google/cloud/hooks/bigquery_dts.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,10 +82,7 @@ def _disable_auto_scheduling(config: Union[dict, TransferConfig]) -> TransferCon
schedule_options["disable_auto_scheduling"] = True
else:
new_config["schedule_options"] = {"disable_auto_scheduling": True}
# HACK: TransferConfig.to_dict returns invalid representation
# See: https://github.com/googleapis/python-bigquery-datatransfer/issues/90
if isinstance(new_config.get('user_id'), str):
new_config['user_id'] = int(new_config['user_id'])

return TransferConfig(**new_config)

def get_conn(self) -> DataTransferServiceClient:
Expand Down
3 changes: 3 additions & 0 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,9 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
'pandas-gbq',
pandas_requirement,
'sqlalchemy-bigquery>=1.2.1',
# A transient dependency of google-cloud-bigquery-datatransfer, but we
# further constrain it since older versions are buggy.
'proto-plus>=1.19.6',
]
grpc = [
# Google has very clear rules on what dependencies should be used. All the limits below
Expand Down