docs: update notebook for JSON subfields support in to_pandas_batches() (#2138)

* change to ai.generate

* perf: Default to interactive display for SQL in anywidget mode

Previously, SQL queries in anywidget mode would fall back to deferred execution, showing a dry run instead of an interactive table.

This change modifies the display logic to directly use the anywidget interactive display for SQL queries, providing a more consistent and responsive user experience. A test case has been added to verify this behavior.

* fix: resolve double printing issue in anywidget mode

* feat: Add test case for STRUCT column in anywidget

Adds a test case to verify that a DataFrame with a STRUCT column is
correctly displayed in anywidget mode.

This test confirms that displaying a STRUCT column does not raise an
exception that would trigger the fallback to the deferred representation.
It mocks `IPython.display.display` to capture the `TableWidget` instance
and asserts that the rendered HTML contains the expected string
representation of the STRUCT data.

* fix presubmit

* Revert accidental changes to test_function.py

* revert accidental change to blob.py

* change return type

* add todo and revert change

* Revert "add todo and revert change"

This reverts commit 153e1d2.

* Add todo

* Fix: Handle JSON dtype in anywidget display

This commit fixes an AttributeError that occurred when displaying a
DataFrame with a JSON column in anywidget mode. The dtype check
was incorrect and has been updated. Additionally, the SQL compilation
for casting JSON to string has been corrected to use TO_JSON_STRING.

* revert a change

* revert a change

* Revert: Restore bigframes/dataframe.py to state from 42da847

* remove anywidget from early return, allow execution proceeds to _repr_html_()

* remove unnecessary changes

* remove redundant code change

* code style change

* tescase update

* revert a change

* final touch of notebook

* fix presumbit error

* remove invlaid test with anywidget bug fix

* fix presubmit

* fix polar complier

* Revert an unnecessary change

* apply the workaround to i/O layer

* Revert scalar_op_registry.py chnage

* remove unnecessary import

* Remove duplicate conversation

* revert changes to test_dataframe.py

* notebook update

* call API on local data for complier.py

* add more testcase

* modfiy polars import

* fix failed tests

* chore: Migrate minimum_op operator to SQLGlot (#2205)

* chore: Migrate round_op operator to SQLGlot (#2204)

This commit migrates the `round_op` operator from the Ibis compiler to the SQLGlot compiler.

* fix: Improve error handling in blob operations (#2194)

* add error handling for audio_transcribe

* add error handling for pdf functions

* add eror handling for image functions

* final touch

* restore rename

* update notebook to better reflect our new code change

* return None on error with verbose=False for image functions

* define typing module in udf

* only use local variable

* Refactor code

* refactor: update geo "spec" and split geo ops in ibis compiler (#2208)

* feat: support INFORMATION_SCHEMA views in `read_gbq` (#1895)

* feat: support INFORMATION_SCHEMA tables in read_gbq

* avoid storage semi executor

* use faster tables for peek tests

* more tests

* fix mypy

* Update bigframes/session/_io/bigquery/read_gbq_table.py

* immediately query for information_schema tables

* Fix mypy errors and temporarily update python version

* snapshot

* snapshot again

* Revert: Unwanted code changes

* Revert "Revert: Unwanted code changes"

This reverts commit db5d8ea.

* revert 1 files to match main branch

* Correctly display DataFrames with JSON columns in anywidget

* add mis-deleted comment back

* revert unnecessary change

* move helper function to dtypes.py

* revert unnecessary testcase change

* Improve JSON type handling for to_gbq and to_pandas_batches

* Remove unnecessary comment

* Revert bigframes/dtypes.py and mypy.ini to main branch version

---------

Co-authored-by: jialuoo <[email protected]>
Co-authored-by: Tim Sweña (Swast) <[email protected]>

Author: 3 people

notebooks/dataframes/anywidget_mode.ipynb

@@ -35,7 +35,16 @@
    "execution_count": 2,
    "id": "ca22f059",
    "metadata": {},
-   "outputs": [],
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/usr/local/google/home/shuowei/src/python-bigquery-dataframes/venv/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.15) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.\n",
+      "  warnings.warn(message, FutureWarning)\n"
+     ]
+    }
+   ],
    "source": [
     "import bigframes.pandas as bpd"
    ]
@@ -142,9 +151,9 @@
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
-       "model_id": "aafd4f912b5f42e0896aa5f0c2c62620",
+       "model_id": "47795eaa10f149aeb99574232c0936eb",
        "version_major": 2,
-       "version_minor": 0
+       "version_minor": 1
       },
       "text/plain": [
        "TableWidget(page_size=10, row_count=5552452, table_html='<table border=\"1\" class=\"dataframe table table-stripe…"
@@ -205,16 +214,17 @@
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
-       "model_id": "5ec0ad9f11874d4f9d8edbc903ee7b5d",
+       "model_id": "8354ce0f82d3495a9b630dfc362f73ee",
        "version_major": 2,
-       "version_minor": 0
+       "version_minor": 1
       },
       "text/plain": [
        "TableWidget(page_size=10, row_count=5552452, table_html='<table border=\"1\" class=\"dataframe table table-stripe…"
       ]
      },
+     "execution_count": 7,
      "metadata": {},
-     "output_type": "display_data"
+     "output_type": "execute_result"
     }
    ],
    "source": [
@@ -283,8 +293,27 @@
     {
      "data": {
       "text/html": [
-       "✅ Completed. \n",
-       "    Query processed 171.4 MB in a moment of slot time.\n",
+       "\n",
+       "    Query started with request ID bigframes-dev:US.c45952fb-01b4-409c-9da4-f7c5bfc0d47d.<details><summary>SQL</summary><pre>SELECT\n",
+       "`state` AS `state`,\n",
+       "`gender` AS `gender`,\n",
+       "`year` AS `year`,\n",
+       "`name` AS `name`,\n",
+       "`number` AS `number`\n",
+       "FROM\n",
+       "(SELECT\n",
+       "  *\n",
+       "FROM (\n",
+       "  SELECT\n",
+       "    `state`,\n",
+       "    `gender`,\n",
+       "    `year`,\n",
+       "    `name`,\n",
+       "    `number`\n",
+       "  FROM `bigquery-public-data.usa_names.usa_1910_2013` FOR SYSTEM_TIME AS OF TIMESTAMP(&#x27;2025-10-30T21:48:48.979701+00:00&#x27;)\n",
+       ") AS `t0`)\n",
+       "ORDER BY `name` ASC NULLS LAST ,`year` ASC NULLS LAST ,`state` ASC NULLS LAST\n",
+       "LIMIT 5</pre></details>\n",
        "    "
       ],
       "text/plain": [
@@ -304,16 +333,17 @@
     {
      "data": {
       "application/vnd.jupyter.widget-view+json": {
-       "model_id": "651b5aac958c408183775152c2573a03",
+       "model_id": "59461286a17d4a42b6be6d9d9c7bf7e3",
        "version_major": 2,
-       "version_minor": 0
+       "version_minor": 1
       },
       "text/plain": [
        "TableWidget(page_size=10, row_count=5, table_html='<table border=\"1\" class=\"dataframe table table-striped tabl…"
       ]
      },
+     "execution_count": 9,
      "metadata": {},
-     "output_type": "display_data"
+     "output_type": "execute_result"
     }
    ],
    "source": [
@@ -323,11 +353,101 @@
     "print(f\"Small dataset pages: {math.ceil(small_widget.row_count / small_widget.page_size)}\")\n",
     "small_widget"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "added-cell-2",
+   "metadata": {},
+   "source": [
+    "### Displaying Generative AI results containing JSON\n",
+    "The `AI.GENERATE` function in BigQuery returns results in a JSON column. While BigQuery's JSON type is not natively supported by the underlying Arrow `to_pandas_batches()` method used in anywidget mode ([Apache Arrow issue #45262](https://github.com/apache/arrow/issues/45262)), BigQuery Dataframes automatically converts JSON columns to strings for display. This allows you to view the results of generative AI functions seamlessly."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "added-cell-1",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "✅ Completed. \n",
+       "    Query processed 85.9 kB in 14 seconds of slot time.\n",
+       "    "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/usr/local/google/home/shuowei/src/python-bigquery-dataframes/bigframes/dtypes.py:969: JSONDtypeWarning: JSON columns will be represented as pandas.ArrowDtype(pyarrow.json_())\n",
+      "instead of using `db_dtypes` in the future when available in pandas\n",
+      "(https://github.com/pandas-dev/pandas/issues/60958) and pyarrow.\n",
+      "  warnings.warn(msg, bigframes.exceptions.JSONDtypeWarning)\n"
+     ]
+    },
+    {
+     "data": {
+      "text/html": [
+       "✅ Completed. "
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "d1794b42579542a8980bd158e521bd3e",
+       "version_major": 2,
+       "version_minor": 1
+      },
+      "text/plain": [
+       "TableWidget(page_size=10, row_count=5, table_html='<table border=\"1\" class=\"dataframe table table-striped tabl…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/html": [],
+      "text/plain": [
+       "Computation deferred. Computation will process 0 Bytes"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "bpd._read_gbq_colab(\"\"\"\n",
+    "  SELECT\n",
+    "    AI.GENERATE(\n",
+    "      prompt=>(\\\"Extract the values.\\\", OBJ.GET_ACCESS_URL(OBJ.FETCH_METADATA(OBJ.MAKE_REF(gcs_path, \\\"us.conn\\\")), \\\"r\\\")),\n",
+    "      connection_id=>\\\"bigframes-dev.us.bigframes-default-connection\\\",\n",
+    "      output_schema=>\\\"publication_date string, class_international string, application_number string, filing_date string\\\") AS result,\n",
+    "    *\n",
+    "  FROM `bigquery-public-data.labeled_patents.extracted_data`\n",
+    "  LIMIT 5;\n",
+    "\"\"\")"
+   ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "3.10.18",
+   "display_name": "venv",
    "language": "python",
    "name": "python3"
   },
@@ -341,7 +461,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.18"
+   "version": "3.10.15"
   }
  },
  "nbformat": 4,