Skip to content

[SPARK-46135][PYTHON][DOCS] Fix table format error in ipynb docs #44049

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
panbingkun wants to merge 6 commits into from
Closed

[SPARK-46135][PYTHON][DOCS] Fix table format error in ipynb docs #44049

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6157db9
[SPARK-46135][PYTHON][DOCS] Fix table format error in ipynb docs
panbingkun Nov 28, 2023
899921e
[SPARK-46135][PYTHON][DOCS] Fix table format error in ipynb docs
panbingkun Nov 28, 2023
f52c1c8
[SPARK-46135][PYTHON][DOCS] Fix table format error in ipynb docs
panbingkun Nov 29, 2023
b34798c
Merge branch 'master' into SPARK-46135
panbingkun Nov 29, 2023
11f6454
Merge branch 'master' into SPARK-46135
panbingkun Nov 30, 2023
b6fb6db
[SPARK-46135][PYTHON][DOCS] Fix table format error in ipynb docs
panbingkun Nov 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion python/docs/source/getting_started/quickstart_connect.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
"metadata": {},
"outputs": [],
"source": [
"!$HOME/sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:$SPARK_VERSION"
"!$HOME/sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.13:$SPARK_VERSION"

@panbingkun panbingkun Nov 28, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, I made some changes because currently our scala version is 2.13, 2.12 is no longer supported.

]
},
{
Expand Down
110 changes: 26 additions & 84 deletions python/docs/source/getting_started/quickstart_df.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,27 +218,23 @@
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border='1'>\n",
"<tr><th>a</th><th>b</th><th>c</th><th>d</th><th>e</th></tr>\n",
"<tr><td>1</td><td>2.0</td><td>string1</td><td>2000-01-01</td><td>2000-01-01 12:00:00</td></tr>\n",
"<tr><td>2</td><td>3.0</td><td>string2</td><td>2000-02-01</td><td>2000-01-02 12:00:00</td></tr>\n",
"<tr><td>3</td><td>4.0</td><td>string3</td><td>2000-03-01</td><td>2000-01-03 12:00:00</td></tr>\n",
"</table>\n"
],
"text/plain": [
"DataFrame[a: bigint, b: double, c: string, d: date, e: timestamp]"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
"name": "stdout",
"output_type": "stream",
"text": [

@panbingkun panbingkun Nov 28, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's present the output results in text format instead of text/html format to avoid formatting errors.

@HyukjinKwon HyukjinKwon Nov 28, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, this example should show the output nicely as spark.sql.repl.eagerEval.enabled is enabled. Wonder if we can fix the docs instead.

@panbingkun panbingkun Nov 28, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HyukjinKwon Is the following presentation style appropriate for this special case?

  • Dark theme:
    image

  • Light theme:
    image

"+---+---+-------+----------+-------------------+\n",
"| a| b| c| d| e|\n",
"+---+---+-------+----------+-------------------+\n",
"| 1|2.0|string1|2000-01-01|2000-01-01 12:00:00|\n",
"| 2|3.0|string2|2000-02-01|2000-01-02 12:00:00|\n",
"| 4|5.0|string3|2000-03-01|2000-01-03 12:00:00|\n",
"+---+---+-------+----------+-------------------+\n",
"\n"
]
}
],
"source": [
"spark.conf.set('spark.sql.repl.eagerEval.enabled', True)\n",
"df"
"df.show()"
]
},
{
Expand Down Expand Up @@ -424,76 +420,22 @@
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>a</th>\n",
" <th>b</th>\n",
" <th>c</th>\n",
" <th>d</th>\n",
" <th>e</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>1</td>\n",
" <td>2.0</td>\n",
" <td>string1</td>\n",
" <td>2000-01-01</td>\n",
" <td>2000-01-01 12:00:00</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2</td>\n",
" <td>3.0</td>\n",
" <td>string2</td>\n",
" <td>2000-02-01</td>\n",
" <td>2000-01-02 12:00:00</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>3</td>\n",
" <td>4.0</td>\n",
" <td>string3</td>\n",
" <td>2000-03-01</td>\n",
" <td>2000-01-03 12:00:00</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" a b c d e\n",
"0 1 2.0 string1 2000-01-01 2000-01-01 12:00:00\n",
"1 2 3.0 string2 2000-02-01 2000-01-02 12:00:00\n",
"2 3 4.0 string3 2000-03-01 2000-01-03 12:00:00"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
"name": "stdout",
"output_type": "stream",
"text": [
"+----+-----+-----+---------+------------+---------------------+\n",
"| | a | b | c | d | e |\n",
"|----+-----+-----+---------+------------+---------------------|\n",
"| 0 | 1 | 2 | string1 | 2000-01-01 | 2000-01-01 12:00:00 |\n",
"| 1 | 2 | 3 | string2 | 2000-02-01 | 2000-01-02 12:00:00 |\n",
"| 2 | 4 | 5 | string3 | 2000-03-01 | 2000-01-03 12:00:00 |\n",
"+----+-----+-----+---------+------------+---------------------+\n"
]
}
],
"source": [
"df.toPandas()"
"from tabulate import tabulate\n",
"print(tabulate(df.toPandas(), headers = 'keys', tablefmt = 'psql'))"

@HyukjinKwon HyukjinKwon Nov 29, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, the output format looks fine but the whole point of using spark.sql.repl.eagerEval.enabled is to show a pretty table format without applying any other operations in the notebook.

Can you maybe just manually fix the output text/html to be compatible with both the sphinx dark theme and jupyter notebook?

@panbingkun panbingkun Nov 29, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Dark theme:
    image

  • Light theme:
    image

@panbingkun panbingkun Nov 29, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I understand what you mean.
For this document, I have modified the style to maintain using spark.sql.repl.eagerEval.enabled purpose to show a pretty table format without applying any other operations in the notebook.

@panbingkun panbingkun Nov 29, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For python/docs/source/getting_started/quickstart_df.ipynb, are we going to do something similar?
Because this example does not use spark.sql.repl.eagerEval.enabled.

@panbingkun panbingkun Nov 30, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The table format is also correct in jupyter notebook
image

]
},
{
Expand Down
Loading