Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test failures on presubmit session #337

Open
tswast opened this issue Jan 22, 2024 · 1 comment
Open

test failures on presubmit session #337

tswast opened this issue Jan 22, 2024 · 1 comment
Assignees
@chelsea-lin
Labels
api: bigquery Issues related to the googleapis/python-bigquery-dataframes API.

Comments

@tswast
Copy link
Collaborator

tswast commented Jan 22, 2024

Many tests are failing in the presubmit session.

nox > python -m pip freeze
aiohttp==3.9.1
aiosignal==1.3.1
appdirs==1.4.4
argcomplete==3.2.1
asttokens==2.4.1
asyncmock==0.4.2
atpublic==4.0
attrs==23.2.0
bidict==0.22.1
# Editable Git install with no remote (bigframes==0.19.2)
-e /tmpfs/src/github/python-bigquery-dataframes
black==23.12.1
bleach==6.1.0
cachetools==5.3.2
certifi==2023.11.17
cffi==1.16.0
cfgv==3.4.0
charset-normalizer==3.3.2
click==8.0.4
click-plugins==1.1.1
cligj==0.7.2
cloudpickle==3.0.0
colorlog==6.8.0
comm==0.2.1
coverage==7.4.0
cryptography==41.0.7
db-dtypes==1.2.0
decorator==5.1.1
distlib==0.3.8
docstring-inheritance==2.1.2
docutils==0.20.1
et-xmlfile==1.1.0
exceptiongroup==1.2.0
execnet==2.0.2
executing==2.0.1
filelock==3.13.1
fiona==1.9.5
frozenlist==1.4.1
fsspec==2023.12.2
gcp-docuploader==0.6.5
gcp-releasetool==1.17.0
gcsfs==2023.12.2.post1
geopandas==0.14.2
google-api-core==1.34.0
google-auth==2.26.2
google-auth-oauthlib==1.2.0
google-cloud-bigquery @ git+[https://github.com/googleapis/python-bigquery.git@1f96439b3dbd27f11be5e2af84f290ec6094d0a4](https://www.google.com/url?q=https://github.com/googleapis/python-bigquery.git@1f96439b3dbd27f11be5e2af84f290ec6094d0a4&sa=D)
google-cloud-bigquery-connection==1.14.0
-e git+[https://github.com/googleapis/python-bigquery-storage.git@fb0d6f07d4b7582cf01ec98fee3081ec34995603#egg=google_cloud_bigquery_storage](https://www.google.com/url?q=https://github.com/googleapis/python-bigquery-storage.git@fb0d6f07d4b7582cf01ec98fee3081ec34995603%23egg%3Dgoogle_cloud_bigquery_storage&sa=D)
google-cloud-core==2.4.1
google-cloud-functions==1.14.0
google-cloud-iam==2.13.0
google-cloud-resource-manager==1.11.0
google-cloud-storage==2.14.0
google-cloud-testutils==1.4.0
google-crc32c==1.5.0
google-resumable-media==2.7.0
googleapis-common-protos==1.62.0
greenlet==3.0.3
grpc-google-iam-v1==0.13.0
grpcio==1.60.0
grpcio-status==1.48.2
humanize==4.9.0
ibis-framework==7.1.0
identify==2.5.33
idna==3.6
importlib-metadata==7.0.1
importlib-resources==6.1.1
iniconfig==2.0.0
ipython==8.20.0
ipywidgets==8.1.1
jaraco.classes==3.3.0
jedi==0.19.1
jeepney==0.8.0
Jinja2==3.1.3
joblib==1.3.2
jupyterlab-widgets==3.0.9
keyring==24.3.0
markdown-it-py==3.0.0
MarkupSafe==2.1.4
matplotlib-inline==0.1.6
mdurl==0.1.2
mock==5.1.0
more-itertools==10.2.0
multidict==6.0.4
multipledispatch==1.0.0
mypy-extensions==1.0.0
nh3==0.2.15
nodeenv==1.8.0
nox==2023.4.22
numpy==1.26.3
oauthlib==3.2.2
openpyxl==3.1.2
packaging==23.2
pandas==2.2.0
pandas-gbq==0.20.0
parso==0.8.3
parsy==2.1
pathspec==0.12.1
pexpect==4.9.0
pins==0.8.4
pkginfo==1.9.6
platformdirs==4.1.0
pluggy==1.3.0
pooch==1.8.0
pre-commit==3.6.0
prompt-toolkit==3.0.43
proto-plus==1.23.0
protobuf==3.20.3
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==14.0.2
pyarrow-hotfix==0.6
pyasn1==0.5.1
pyasn1-modules==0.3.0
pycparser==2.21
pydata-google-auth==1.8.2
Pygments==2.17.2
PyJWT==2.8.0
pyperclip==1.8.2
pyproj==3.6.1
pytest==7.4.4
pytest-asyncio==0.23.3
pytest-cov==4.1.0
pytest-mock==3.12.0
pytest-retry==1.6.1
pytest-timeout==2.2.0
pytest-xdist==3.5.0
python-dateutil==2.8.2
pytz==2023.3.post1
PyYAML==6.0.1
readme-renderer==42.0
requests==2.31.0
requests-oauthlib==1.3.1
requests-toolbelt==1.0.0
rfc3986==2.0.0
rich==13.7.0
rsa==4.9
scikit-learn==1.4.0
scipy==1.12.0
SecretStorage==3.3.3
shapely==2.0.2
six==1.16.0
SQLAlchemy==2.0.25
sqlglot==19.9.0
stack-data==0.6.3
tabulate==0.9.0
threadpoolctl==3.2.0
tomli==2.0.1
toolz==0.12.0
tqdm==4.66.1
traitlets==5.14.1
twine==4.0.2
typing_extensions==4.9.0
tzdata==2023.4
urllib3==2.1.0
virtualenv==20.25.0
wcwidth==0.2.13
webencodings==0.5.1
widgetsnbextension==4.0.9
xarray==2024.1.0
xxhash==3.4.1
yarl==1.9.4
zipp==3.17.0

...

def test_read_gbq_after_session_start_must_comply_with_default_location(
        scalars_pandas_df_index,
        test_data_tables,
        test_data_tables_tokyo,
        dataset_id_permanent_tokyo,
        read_method,
        query_prefix,
    ):
        # Form query as a table name or a SQL depending on the test scenario
        query_tokyo = test_data_tables_tokyo["scalars"]
        query = test_data_tables["scalars"]
        if query_prefix:
            query_tokyo = f"{query_prefix} {query_tokyo}"
            query = f"{query_prefix} {query}"
    
        # Initially there is no location set in the bigquery options
        assert not bpd.options.bigquery.location
    
        # Starting user journey with anything other than read_gbq*, such as
        # read_pandas would bind the session to default location US
>       df = bpd.read_pandas(scalars_pandas_df_index)

[tests/system/small/test_pandas_options.py:143](https://cs.corp.google.com/piper///depot/google3/tests/system/small/test_pandas_options.py?l=143): 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[bigframes/pandas/__init__.py:570](https://cs.corp.google.com/piper///depot/google3/bigframes/pandas/__init__.py?l=570): in read_pandas
    return global_session.with_default_session(
[bigframes/core/global_session.py:65](https://cs.corp.google.com/piper///depot/google3/bigframes/core/global_session.py?l=65): in with_default_session
    return func(get_global_session(), *args, **kwargs)
[bigframes/session/__init__.py:849](https://cs.corp.google.com/piper///depot/google3/bigframes/session/__init__.py?l=849): in read_pandas
    return self._read_pandas(pandas_dataframe, "read_pandas")
[bigframes/session/__init__.py:889](https://cs.corp.google.com/piper///depot/google3/bigframes/session/__init__.py?l=889): in _read_pandas
    load_job = self.bqclient.load_table_from_dataframe(
[.nox/system_prerelease/lib/python3.11/site-packages/google/cloud/bigquery/client.py:2713](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/google/cloud/bigquery/client.py?l=2713): in load_table_from_dataframe
    dataframe.to_parquet(
[.nox/system_prerelease/lib/python3.11/site-packages/pandas/util/_decorators.py:333](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pandas/util/_decorators.py?l=333): in wrapper
    return func(*args, **kwargs)
[.nox/system_prerelease/lib/python3.11/site-packages/pandas/core/frame.py:3101](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pandas/core/frame.py?l=3101): in to_parquet
    return to_parquet(
[.nox/system_prerelease/lib/python3.11/site-packages/pandas/io/parquet.py:480](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pandas/io/parquet.py?l=480): in to_parquet
    impl.write(
[.nox/system_prerelease/lib/python3.11/site-packages/pandas/io/parquet.py:190](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pandas/io/parquet.py?l=190): in write
    table = self.api.Table.from_pandas(df, **from_pandas_kwargs)
[pyarrow/table.pxi:3869](https://cs.corp.google.com/piper///depot/google3/pyarrow/table.pxi?l=3869): in pyarrow.lib.Table.from_pandas
    ???
[.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py:613](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py?l=613): in dataframe_to_arrays
    arrays = [convert_column(c, f)
[.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py:613](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py?l=613): in <listcomp>
    arrays = [convert_column(c, f)
[.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py:600](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py?l=600): in convert_column
    raise e
[.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py:594](https://cs.corp.google.com/piper///depot/google3/.nox/system_prerelease/lib/python3.11/site-packages/pyarrow/pandas_compat.py?l=594): in convert_column
    result = pa.array(col, type=type_, from_pandas=True, safe=safe)
[pyarrow/array.pxi:340](https://cs.corp.google.com/piper///depot/google3/pyarrow/array.pxi?l=340): in pyarrow.lib.array
    ???
[pyarrow/array.pxi:82](https://cs.corp.google.com/piper///depot/google3/pyarrow/array.pxi?l=82): in pyarrow.lib._ndarray_to_array
    ???
[pyarrow/array.pxi:70](https://cs.corp.google.com/piper///depot/google3/pyarrow/array.pxi?l=70): in pyarrow.lib._ndarray_to_type
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pyarrow.lib.ArrowTypeError: ('Did not pass numpy.dtype object', 'Conversion failed for column geography_col with type geometry')

[pyarrow/error.pxi:91](https://cs.corp.google.com/piper///depot/google3/pyarrow/error.pxi?l=91): ArrowTypeError

...

=========================== short test summary info ============================
FAILED tests/system/small/test_dataframe.py::test_dataframe_bool_aggregates[all_axis0]
FAILED tests/system/small/test_dataframe.py::test_dataframe_bool_aggregates[any_axis0]
FAILED tests/system/small/test_dataframe.py::test_df_pivot[values2-int64_too-columns2]
FAILED tests/system/small/test_groupby.py::test_dataframe_groupby_analytic[cumprod]
FAILED tests/system/small/test_pandas_options.py::test_read_gbq_after_session_start_must_comply_with_default_location[read_gbq-on-table-name]
FAILED tests/system/small/test_pandas_options.py::test_read_gbq_after_session_start_must_comply_with_default_location[read_gbq-on-sql]
FAILED tests/system/small/test_pandas_options.py::test_read_gbq_after_session_start_must_comply_with_default_location[read_gbq_table-on-table-name]
FAILED tests/system/small/test_pandas_options.py::test_read_gbq_after_session_start_must_comply_with_default_location[read_gbq_query-on-sql]
FAILED tests/system/small/test_series.py::test_groupby_window_ops[cumprod] - ...
FAILED tests/system/small/test_dataframe.py::test_to_pandas_downsampling_option_override
FAILED tests/system/small/test_series.py::test_series_add_prefix - AssertionE...
FAILED tests/system/small/test_series.py::test_series_add_suffix - AssertionE...
FAILED tests/system/small/test_session.py::test_read_pandas - pyarrow.lib.Arr...
FAILED tests/system/small/test_session.py::test_read_pandas_multi_index - pya...
FAILED tests/system/small/test_session.py::test_read_pandas_rowid_exists_adds_suffix
FAILED tests/system/small/test_series.py::test_string_astype_int - AssertionE...
FAILED tests/system/small/test_session.py::test_read_pandas_tokyo - pyarrow.l...
FAILED tests/system/small/test_session.py::test_read_csv_local_default_engine[default_sep]
FAILED tests/system/small/test_session.py::test_read_csv_gcs_default_engine
FAILED tests/system/small/test_session.py::test_read_csv_local_default_engine[custom_sep]
20 failed, 1599 passed, 21 skipped, 7 xfailed, 804 warnings in 468.88s (0:07:48)
nox > Command py.test --quiet -n=20 --timeout=600 --junitxml=small_prerelease_3.11_sponge_log.xml --cov=bigframes --cov=tests/system/small --cov-append --cov-config=.coveragerc --cov-report=term-missing --cov-fail-under=0 tests/system/small failed with exit code 1
nox > Session system_prerelease failed.
nox > Ran multiple sessions:
nox > * unit_prerelease: success
nox > * system_prerelease: failed

https://fusion2.corp.google.com/invocations/f5604d0d-929d-456f-ab59-b7e292081af2/targets/bigframes%2Fpresubmit%2Fe2e/log
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. label Jan 22, 2024
@tswast tswast mentioned this issue Jan 22, 2024
Merged
@chelsea-lin chelsea-lin self-assigned this Jan 23, 2024
@chelsea-lin chelsea-lin mentioned this issue Jan 23, 2024
Open
@chelsea-lin
Copy link
Contributor

The issue could be solved by a fix in python client change: googleapis/python-bigquery#1796

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chelsea-lin chelsea-lin

Labels
api: bigquery Issues related to the googleapis/python-bigquery-dataframes API.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tswast @chelsea-lin