Add a workflow that builds the docs and deploys them at staged or production #104

martin-g · 2022-12-05T22:29:59Z

Which issue does this PR close?

Closes #39.

Rationale for this change

Publish the documentation for the Datafusion Python bindings.

When changes are pushed to master then it will copy the generated HTMLs to asf-staging branch and update https://arrow.staged.apache.org/datafusion-python, i.e. the staging site. See Add a workflow that builds the docs and deploys them at staged or production #104 (comment)
When changes are pushed to a non-RC tag, e.g. 0.8.0, then it will copy the generated HTMLs to asf-site branch and update https://arrow.apache.org/datafusion-python, i.e. the production site. See Issue #39 - Add .asf.yaml for asf-site #105

What changes are included in this PR?

A new GitHub Actions workflow is introduced that builds the docs and deploys them at the appropriate site

Are there any user-facing changes?

No API changes!

martin-g · 2022-12-05T22:32:27Z

@andygrove Before merging this PR please create a new branch named asf-staging from asf-site and update its .asf.yaml file contents to be the same as in https://github.com/martin-g/arrow-datafusion-python/blob/asf-staging/.asf.yaml.
It will be used for the staging deployment.

martin-g · 2022-12-05T22:32:51Z

@kou Please review!

…duction Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

.github/workflows/docs.yaml

kou · 2022-12-06T01:52:01Z

.github/workflows/docs.yaml

+      - name: Copy & push the generated HTML
+        run: |
+          cd docs-target
+          find ./ -name "*.html" -type f -print0 | xargs -0 /bin/rm -f


This may not work for other extensions such as *.css. How about using rsync?

rsync \ -a \ --delete \ --exclude '/.git/' \ ../docs/build/html/ \ ./

Right! There are .css/.js/images/... !
I have to test your approach!
The important thing is to preserve anything that is not autogenerated, especially .asf.yaml

How about copying .asf.yaml from master to asf-site?
apache/arrow-site uses this approach: apache/arrow-site#116

This will work for asf-site, but it will break asf-staging, because it uses staging: instead of publish:

find ./ -not -name ".asf.yaml" | grep -v ".git" | xargs rm -rf seems to do the trick.

This will work for asf-site, but it will break asf-staging, because it uses staging: instead of publish:

Can we write both staging: and publish: in master's .asf.yaml?

staging: whoami: asf-staging subdir: datafusion-python publish: whoami: asf-site subdir: datafusion-python

I think this should work!
The only way to find out is to try it! :-)

I think so too. :-)

done with d5e32a2!
depends on #107

.github/workflows/docs.yaml

Co-authored-by: Sutou Kouhei <[email protected]>

Use special email address for the Git user, so that it has an avatar/icon. Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

kou

+1

This almost looks good to me. I think that we can merge this after we confirm that this works with 0.8.0-rc1 and 0.8.0 tags on your fork. (We need to merge this branch to master on your fork temporary to test these tags.)

.github/workflows/docs.yaml

kou · 2022-12-07T20:22:35Z

.github/workflows/docs.yaml

+on:
+  push:
+    branches:
+      - docs-workflow


We need to change this to master when we merge this.

Right! I used it to be able to test on my fork!

kou · 2022-12-07T20:41:22Z

.github/workflows/docs.yaml

+      - name: Checkout docs target branch
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+          ref: ${{ steps.target-branch.outputs.value }}
+          path: docs-target


It seems that we need to prepare asf-site and asf-staging branches for this: https://github.com/martin-g/arrow-datafusion-python/actions/runs/3633415170/jobs/6130403408

How about using git worktree instead?

diff --git a/.github/workflows/docs.yaml b/.github/workflows/docs.yaml index 806202a..bfafafa 100644 --- a/.github/workflows/docs.yaml +++ b/.github/workflows/docs.yaml @@ -26,12 +26,6 @@ jobs: fi - name: Checkout docs sources uses: actions/checkout@v3 - - name: Checkout docs target branch - uses: actions/checkout@v3 - with: - fetch-depth: 0 - ref: ${{ steps.target-branch.outputs.value }} - path: docs-target - name: Setup Python uses: actions/setup-python@v4 with: @@ -61,7 +55,12 @@ jobs: - name: Copy & push the generated HTML run: | set -x + git worktree add docs-target cd docs-target + if ! git checkout ${{ steps.target-branch.outputs.value }}; then + git checkout --orphan ${{ steps.target-branch.outputs.value }} + git rm -rf . + fi # delete anything but: 1) '.'; 2) '..'; 3) .git/ find ./ | grep -vE "^./$|^../$|^./.git" | xargs rm -rf cp ../.asf.yaml .

IMO this is more complex/confusing than the current solution.

A task before merging this PR is to create asf-staging branch.

Created: https://github.com/apache/arrow-datafusion-python/tree/asf-staging

.github/workflows/docs.yaml

Co-authored-by: Sutou Kouhei <[email protected]>

martin-g · 2022-12-07T21:00:22Z

This almost looks good to me. I think that we can merge this after we confirm that this works with 0.8.0-rc1 and 0.8.0 tags on your fork. (We need to merge this branch to master on your fork temporary to test these tags.)

Actually I already tested tags - see testN and test-rcN at https://github.com/martin-g/arrow-datafusion-python/actions
Those tags were created from docs-workflow branch, not from master.

kou · 2022-12-07T21:41:59Z

This almost looks good to me. I think that we can merge this after we confirm that this works with 0.8.0-rc1 and 0.8.0 tags on your fork. (We need to merge this branch to master on your fork temporary to test these tags.)

Actually I already tested tags - see testN and test-rcN at https://github.com/martin-g/arrow-datafusion-python/actions Those tags were created from docs-workflow branch, not from master.

Could you retry testN after you create asf-site branch on your fork?

martin-g · 2022-12-07T21:50:26Z

Could you retry testN after you create asf-site branch on your fork?

Done!
There is no Deploy DataFusion Python site for test-rc10 and there is such for test10.

kou · 2022-12-07T23:35:09Z

Thanks. Committed: martin-g@34ed8b8

I merge this.

martin-g force-pushed the docs-workflow branch from 6056d73 to a0e9bd6 Compare December 5, 2022 22:33

Add a workflow that builds the docs and deploys them at staged or pro…

aea783e

…duction Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

martin-g force-pushed the docs-workflow branch from a0e9bd6 to aea783e Compare December 5, 2022 22:36

martin-g mentioned this pull request Dec 5, 2022

Issue #39 - Add .asf.yaml for asf-site #105

Closed

kou reviewed Dec 6, 2022

View reviewed changes

martin-g and others added 2 commits December 6, 2022 09:19

Add a branch for unsupported branch/tag

91d97e6

Co-authored-by: Sutou Kouhei <[email protected]>

Improve the glob to ignored RCx tags

289f36e

martin-g force-pushed the docs-workflow branch 2 times, most recently from 52bcc20 to 64dc8d9 Compare December 6, 2022 20:38

"git commit" only if there are changed files

e0a6910

Use special email address for the Git user, so that it has an avatar/icon. Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

martin-g force-pushed the docs-workflow branch from 64dc8d9 to e0a6910 Compare December 6, 2022 20:49

martin-g added 4 commits December 6, 2022 23:00

Test building tags

617ead0

Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

Delete everything but .asf.yaml and .git/ when copying the new docs

832cc6d

Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

Remove the filtering by paths

8f46ef9

Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

Do not try to delete . and ..

822153e

Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

martin-g force-pushed the docs-workflow branch from d27c482 to 822153e Compare December 6, 2022 21:47

martin-g mentioned this pull request Dec 6, 2022

Add entries for publishing production (asf-site) and staging docs #107

Merged

Use .asf.yaml from master/tag

d5e32a2

Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

kou approved these changes Dec 7, 2022

View reviewed changes

martin-g and others added 2 commits December 7, 2022 22:55

Remove empty line

2a65b5d

Co-authored-by: Sutou Kouhei <[email protected]>

Fix wording

8b92e71

Co-authored-by: Sutou Kouhei <[email protected]>

Change the tested branch back to master

e0692fa

kou merged commit a110049 into apache:master Dec 7, 2022

kou mentioned this pull request Dec 7, 2022

Add DataFusion Python docs apache/arrow-site#279

Closed

martin-g deleted the docs-workflow branch December 8, 2022 09:38

kou mentioned this pull request Dec 13, 2022

chore: set up release process apache/arrow-adbc#174

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a workflow that builds the docs and deploys them at staged or production #104

Add a workflow that builds the docs and deploys them at staged or production #104

martin-g commented Dec 5, 2022 •

edited by kou

Loading

martin-g commented Dec 5, 2022

martin-g commented Dec 5, 2022

kou Dec 6, 2022

martin-g Dec 6, 2022

kou Dec 6, 2022

martin-g Dec 6, 2022

martin-g Dec 6, 2022

kou Dec 6, 2022

martin-g Dec 6, 2022

kou Dec 6, 2022

martin-g Dec 6, 2022 •

edited

Loading

kou left a comment

kou Dec 7, 2022

martin-g Dec 7, 2022

martin-g Dec 7, 2022

kou Dec 7, 2022

martin-g Dec 7, 2022

martin-g Dec 7, 2022

kou Dec 7, 2022

martin-g commented Dec 7, 2022

kou commented Dec 7, 2022

martin-g commented Dec 7, 2022 •

edited

Loading

kou commented Dec 7, 2022

Add a workflow that builds the docs and deploys them at staged or production #104

Add a workflow that builds the docs and deploys them at staged or production #104

Conversation

martin-g commented Dec 5, 2022 • edited by kou Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

martin-g commented Dec 5, 2022

martin-g commented Dec 5, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martin-g Dec 6, 2022 • edited Loading

Choose a reason for hiding this comment

kou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martin-g commented Dec 7, 2022

kou commented Dec 7, 2022

martin-g commented Dec 7, 2022 • edited Loading

kou commented Dec 7, 2022

martin-g commented Dec 5, 2022 •

edited by kou

Loading

martin-g Dec 6, 2022 •

edited

Loading

martin-g commented Dec 7, 2022 •

edited

Loading