Writing py5 tutorials and how-to documentation to support new creative coders

[marcovicci]

Hello, I'm Zelle! If you recognize that discussion title, it's the truncated form of "Writing py5 tutorials and how-to documentation to support creative coders new to Python and/or new to the Processing ecosystem", a project supported by the Processing Foundation alongside their Google Summer of Code 2022 offerings: https://medium.com/processing-foundation/announcing-google-summer-of-code-2022-projects-and-a-few-more-77043ab4d0b4

Learning how Jupyter Notebooks functions has been strange, but I return with progress and delight (it feels really good once you actually get into it). I've made the following placeholder pages in my fork of py5book's "tutorials" folder, representing eighteen different tutorials that will hopefully get just about anybody up and running in py5.

• Installing and using py5 (which can discuss how to use the live code functions on the documentation site, development environments like Thonny and Jupyter Notebooks with the right addons and kernels, and the difference between modes)
• Drawing 2D primitives (coordinates, color, fills, strokes)
• Variables and data types
• Drawing complex shapes (curves, vertices, contour functions)
• Working with text (strings and the functions and methods to manipulate them, visual display of text and fonts)
• Conditional statements (if, else if, else; and, or, not)
• Iteration with loops (for, while, break, continue)
• Randomness (using random functions with iteration to create tiled patterns)
  The following pages tentatively use the py5 kernel instead of py5bot, since this is where we introduce sketches with setup() and draw(). I believe there are other syntax differences that will need to be addressed, but I'll feel it out as we go...
• Animation and motion (use of non-static sketches, frames, etc)
• Mouse and keyboard interaction
• Transformations and matrixes
• Working with lists
• Reading external data (CSV files that can generate charts)
• Dictionaries (and in addition to the standard dictionary stuff, loading in external json data)
• Custom functions (args, return, etc)
• Trigonometry for animation
• Object-Oriented Programming
• py5vector movement (which utilizes OOP to do some really fun stuff)

Because I've been writing week-by-week, and in larger chunks than these pages represent (running through several pages worth of material each Monday), I have actual content to move into them up until Randomness and will soon have more. Not all these files will be populated in a linear order, since the structure of the class is a little different - notably, mouse and keyboard interaction will be filled in quite a bit later than the surrounding stuff. So it goes.

A good initial example of the content of these tutorials is the Conditional statements page, which has a first pass complete. (As each tutorial is moved into the correct page, I'll also be creating corresponding folders for them inside of the "images" folder, since that makes sense - and what even is py5 without visual examples?)

https://github.com/marcovicci/py5book/blob/main/tutorials/conditional_statements.md

None of this is any good unless there are eyes on it, of course, so it's all fair game for discussion and suggestions. I'll be continuing to move content into the right places - as of writing this, the beginner coders at Massey have learned about 2D primitives, variables/data types, drawing more complex shapes, and working with text, and they've taken to it amazingly well, which makes me very optimistic!

[tabreturn]

Thanks, Zelle. I think you should also amend the https://github.com/marcovicci/py5book/blob/main/_toc.yml file, so we can see how the contents will look ordered/arranged on the site.

It'll be a long list of items to add to the contents, and I recall discussions about accordion/expanding sections to accommodate things better. Also, it'll help us consider how we best arrange the content sections.

[marcovicci]

Done, and absolutely enormous! I definitely think an accordion style menu could be our saving grace. I wouldn't want to make what we're adding look more or less significant than the existing tutorials; perhaps everything here (new and old) can end up categorized and tucked away in those sensible categories.

[marcovicci]

Added first passes at Iteration with loops and Randomness, as well as updating https://github.com/marcovicci/py5book/blob/main/_toc.yml with this utterly huge new list of stuff.

https://github.com/marcovicci/py5book/blob/main/tutorials/iteration_with_loops.md

https://github.com/marcovicci/py5book/blob/main/tutorials/randomness.md

[hx2A]

Added first passes at Iteration with loops and Randomness,

I just read these and they look great! I very much like how you provide history and context to introduce the programming concepts. Your writing style is easy to read and I can see how this will be a hit with beginners.

this utterly huge new list of stuff.

I will look into what we can do about this to keep it organized. Other projects with larger Jupyter book websites would probably provide good ideas for how we can approach this. Currently there is something accordion-like happening with each class in the Reference documentation. Maybe we can do something like that here?

[hx2A]

I like your organization, this is a good outline of all of the topics.

(As each tutorial is moved into the correct page, I'll also be creating corresponding folders for them inside of the "images" folder, since that makes sense - and what even is py5 without visual examples?)

Let me know if you have issues with broken images, or images that work in the Jupyter notebook but aren't correct after Jupyter book gets built. The Makefile calls a post_build.py script to make some small adjustments and corrections. Currently it isn't adjusting any image links but I recall needing to do so in the past (I can't quite remember why though).

[marcovicci]

Thanks! There might be heaps - I haven't tried building the book at all yet while I muck around figuring it all out. I'll let you know when I find some problems...

[marcovicci]

Added first passes at Animation and Motion, and Transformation and Matrices (which is now spelled correctly)...

https://github.com/marcovicci/py5book/blob/main/tutorials/animation_and_motion.md

https://github.com/marcovicci/py5book/blob/main/tutorials/transformations_and_matrices.md

These ones are weird and lengthy, and especially when dealing with matrices, I'm a bit less confident in what I'm describing (life rarely asks me to do complex maths). The students will test it all soon enough!

Animation and Motion also includes an introduction to setup() and draw(), since all other tutorials so far have been using static mode. I suspect I may have to revisit that introduction in other tutorials that require it, in case people do a lot of jumping around.

[hx2A]

I like how github will add a little color swatch next to hex colors: #882222.

[villares]

First pass at Trigonometry for Animation added too. This one scares me and I'll revisit it later as well

See if you like any of my trigonometry examples...

https://github.com/villares/material-aulas/blob/main/Processing-Python-py5/seno_cosseno_atan2.md
(I haven't revised the Py.Processing to py5 conversion yet, code formatting is awful right now, as it came straight from the conversion helper)

[tabreturn]

How does one type an en dash in Jupyter notebook?

It seems Jupyter won't render -- as an en-dash (the width of an n character). As an aside, LaTeX will convert --- to an em-dash (the width of an m).

I'd probably just write all the en-dashes as as -- for Jupyter, then use a good ol' find-and-replace to switch those out for em-dash characters.

Also, I don't mind a double-hyphen (--), but hyphen (-) just looks wrong to me.

[hx2A]

I'd probably just write all the en-dashes as as -- for Jupyter, then use a good ol' find-and-replace to switch those out for em-dash characters.

The py5book build process has a post_build.py script for making adjustments like this. I'll open an issue to convert all -- and --- to en dashes and em dashes.

[hx2A]

py5coding/py5book#7

[villares]

Wow, wonderful stuff @marcovicci! Thank you all folks!

Should the binder thing work on the fork? Like this https://mybinder.org/v2/gh/marcovicci/py5book/HEAD?urlpath=lab ?

The colors part took me by surprise because I'm so used to fill(r, g, b) instead of the hex notation, maybe because I started on Processing Java, now I have to think if I should switch my stuff to hex!

[hx2A]

Should the binder thing work on the fork? Like this https://mybinder.org/v2/gh/marcovicci/py5book/HEAD?urlpath=lab ?

I would expect it to work but if you are the first person to attempt this, it will be slow as it builds the image. Are you seeing an error?

[villares]

It looked like it never finished loading properly, but I'm not sure.

…

On Sun, 31 Jul 2022, 14:57 hx2A, ***@***.***> wrote: Should the binder thing work on the fork? Like this https://mybinder.org/v2/gh/marcovicci/py5book/HEAD?urlpath=lab ? I would expect it to work but if you are the first person to attempt this, it will be slow as it builds the image. Are you seeing an error? — Reply to this email directly, view it on GitHub <#101 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4GADHRISODCIXMXNDRRTDVW25A3ANCNFSM543LRLMQ> . You are receiving this because you commented.Message ID: ***@***.***>

[hx2A]

Yes, sometimes it fails for odd reasons.

[villares]

Working now, yay!
https://hub-binder.mybinder.ovh/user/marcovicci-py5book-18jrna01/lab/tree/tutorials/color_and_shapes.ipynb

[marcovicci]

Long time no see! (Still writing, of course -- newest finished update is Mouse and keyboard interaction -- but I had a bit of a break where not much happened.) I'm back in the saddle and will be adapting the tutorial for custom functions soon. Yay!

@hx2A Tristan and I were wondering if I should do a pull request for one or two roughly finished tutorial pages? This could let us see how things are actually looking and whether something in my formatting is going to present problems overall, and give a pull request page to include in the Processing foundation's GSOC22 wrap-up post. Maybe the page on iteration/loops or randomness (as beginner introductions which use Static mode)? Or the Animation/Motion page? Let me know what you think I need to do to push them a bit farther along!

[hx2A]

Important: I worked on this PR for much of today. There were a bunch of problems & headaches because of things I didn't know about jupyterbook and other stuff. I fixed the problems, but now have these ridiculously long messages on github that point out problems that get solved by the end. This might be a bit confusing to read, and it would be hard to correct it. I'm going to leave this so perhaps you can see my thought process and why certain changes were made. I recommend reading this but gloss over anything that is confusing, everything should be OK by the end.

---

I'm a real greenhorn to contributing to projects on Github so I may have overlooked something in the PR!

That's OK, I'm new at reviewing PRs of this size. We'll figure out how to do this together!

I was able to fetch the PR code and needed to make some changes (discussed below) to make the build work. It doesn't seem I can commit and push to your PR but I did create a new branch in the py5book repo called zellegsoc that contains your PR plus my additions.

https://github.com/py5coding/py5book/tree/zellegsoc

I think a good plan is for both of us to make PRs against the same branch that we both pull from. Is that OK with you? I can also make PRs against your fork of py5book if you prefer. Both will get the job done. Which is easier for you?

I uploaded this build to http://py5dev.ixora.io/. There are some problems that we will need to put our heads together and solve.

I explored these issues with the animation page, and you can see the results of my changes on that tutorial.

http://py5dev.ixora.io/tutorials/intro_to_py5_and_python_animation_and_motion.html

Headache number 1

When jupyterbook runs the build, it will re-execute every notebook and store the results of each notebook cell. What appears in the build is its notebook executions, not yours. This will cause headaches for us because it will run one cell after another with no pauses. When you run the notebook, you will execute each cell with pauses, and for the cells that run a Sketch, you will let it run for a bit and then exit the Sketch. Unfortunately, jupyterbook does not know it needs to exit the Sketch. As a result, you will see this message:

Sketch is already running. To run a new sketch, exit the running sketch first.

in a lot of the notebooks (but not the animation one, because I fixed it there). The solve for this to explicitly call exit_sketch() after a small delay. That looks ugly when added to the documentation though. Happily, Myst Markdown has a solution for this that lets you add cells that either have their output hidden or cells that are completely removed.

https://jupyterbook.org/en/stable/reference/cheatsheet.html#myst-cheatsheet-code-cell-tags

In our case, you can add something like the following to the md files after run_sketch() gets called:

```{code-cell} ipython3
:tags: ["remove-cell"]
import time  # this only needs to be done once per notebook

# give the Sketch some time to run and then exit
time.sleep(5)
exit_sketch()
time.sleep(0.25) # important to pause again because exiting is not instant
# (conclude this block with 3 backticks, but I can't type that in github)

The above cell will be executed but will not appear in the generated html page. This solves headache number 1, but there are problems if the Sketch happens to use print statements.

Headache number 2

Print statements were a headache for py5 and Jupyter notebooks. Consider that a running py5 Sketch is using Java and is effectively multi-threaded. Q: if you run a Sketch with print statements in draw(), then move to a new cell and execute something simple in the new cell, where will the output of the print statements go? A: It will jump from the cell with the run_sketch() call and will now show up in the last executed cell. The output will follow you from one cell to the next.

Usually the solution for this is to use println(). This will "bind" to the cell that called run_sketch() and all output will only go there, regardless of what you do in the other cells.

However, all of this works differently when jupyterbook is executing the notebook. If you use the remove-cell idea discussed above and are using print, the print output will go to the cell that is getting removed. Nothing will appear in the html output. If you use println, for some unknown reason the output gets mangled. I tried some other idea (that I don't like) that moves the sleep command to the cell with run_sketch() that you can see on this page:

http://py5dev.ixora.io/tutorials/intro_to_py5_and_python_animation_and_motion.html

With this, the output also gets mangled with the print output grouped together in groups of 7 to 14 lines. If I used println, it would be groups of 1. Why does it do this??? I have no idea. I've been working on this all afternoon and am currently feeling frustrated.

The print statements look great for http://py5dev.ixora.io/tutorials/intro_to_py5_and_python_randomness.html but perhaps would benefit from output scrolling. Use the tag "output_scroll" for that, as described in the myst cheatsheet. These print statements are from py5bot sketches, which for whatever reason jupyterbook can properly handle.

So what to do about print statements? I don't want to ask you to not use print statements in the documentation. I don't like that mangled output and adding time.sleep() again and again is weird and confusing. We are going to have to put our heads together and figure out what to do about this. Reading the jupyterbook documentation carefully and experimenting with different ideas will be necessary. We'll have to find some kind of path forward that works with Jupyterbook and provides the quality documentation that we are striving for.

[hx2A]

There are a few other changes that I made that were necessary to make this work, plus some other comments I should make.

• I added a tutorials/misc.md file to be a placeholder for the group of misc tutorials. Without that placeholder, they don't appear in the menu sidebar on the left.
• I added code to scripts/post_build.py so that your tutorials/images/ would get included in the build.
• You added a page to developer/ that talks about your involvement in GSoC. This is a good page and I am glad you wrote it. I think it might be better to put it at the end of your tutorials? Or we should create a new section? I don't know. Right now it is not in _toc.yml so it does not appear in the build.
• The first heading in each tutorial becomes the page title and should have one #, not two. I think this is an accessibility issue where pages are harder to navigate with a screen reader if the page skips a heading level? I believe Jupyterbook fixes this for you, but it issues a warning. I made this adjustment to the animation page but not the others.
• Only the pages that match tutorials/intro_to_py5_and_python_* are included in the build because only those are mentioned in _toc.yml. The other tutorials like working_with_text, etc, need to be renamed to be included. This seems like an intentional choice. Did you intend to rename them later?
• Your writing and documentation are so imaginative! This is fun to read. I'm so excited for what you are working on.

@marcovicci , there's a lot ahead of us but we will work together and solve the problems. Perseverance and teamwork are the driving force of open source projects. I know I wrote a lot here. If anything I wrote is confusing, I'd be happy to get a video call to explain stuff better. Let me know.

[hx2A]

I did some digging into the jupyterbook documentation and I found this:

https://jupyterbook.org/en/stable/content/execute.html#exclude-files-from-execution

Apparently Jupyter book doesn't have to execute each notebook itself; certain notebooks can be left alone. It will then create the page with the notebook as you saved it.

Here is the solution to Headache number 1 and number 2. If we put this in _config.yml:

execute:
  execute_notebooks: cache
  exclude_patterns:
    - 'intro_to_py5_and_python*'

It will skip the execution of the tutorials. Then you can have print statements with outputs that appear as you intended them to appear, without Jupyterbook executing everything again and making a mess. However, I tested this and it only works if you remove the myst markdown files. If the myst markdown files are there, it will use those for the build instead, and you'll have no outputs.

I just tested this with the animation notebook and it works great.

[hx2A]

OK! I now I have the other intro_to_py5_and_python* tutorials appearing correctly in the output, without jupyterbook re-executing them. Progress! I had to remove the myst markdown files. Headaches are solved!

There are maybe some cosmetic issues in the tutorials, like some of the notebook cells that have many rows of print statement output could benefit from the :tags: ["output_scroll"] tag, as described in https://jupyterbook.org/en/stable/reference/cheatsheet.html#myst-cheatsheet-code-cell-tags. You'll need to add this to the notebook cell metadata, since the myst files are removed.

Everything pushed to the zellegsoc branch of the py5book repo:

https://github.com/py5coding/py5book/tree/zellegsoc

[hx2A]

@marcovicci sorry if all of these messages are confusing!

Bottom line, here are the things you'll want to do next:

• Pull from the zellegsoc branch, but prioritize your copies of the tutorials over what is in that branch if something looks off
• Remove the Myst markdown files the notebooks are paired with
• Execute the tutorial notebooks and save them with the output in each cell as you'd like it to appear in the final documentation, if this has not been done already
• Consider adding the "output_scroll" tag to the notebook cells where you think it would be helpful
• Also, the "raises-exception" tag is useful for cells that are supposed to throw an error
• Rename tutorials to fit the intro_to_py5_and_python* pattern when you want them to be in the table of contents
• Make sure the first heading in each tutorial has one #, not two

That's enough writing for me today. I need a break from my computer!

[hx2A]

@marcovicci I hope you are having a good day today!

I just merged your PR into a new branch zellegsoc. Then I ran the build and deployed to the dev site:

http://py5dev.ixora.io/

Please pull from the zellegsoc branch and continue your work from there, and make the next PR against that. Moving forward, I'll approve your PRs as you make them so we both can have a current branch to work with.

A few things you'll notice on the dev site:

• In the tutorials section, there is a folding chapter for your intro tutorials. There's also a second folding chapter for the other tutorials, which I gave the generic name "misc". I'll improve that later.
• Not all of your tutorials are included in the intro tutorials chapter because not all of them match the glob pattern in the _toc.yml file glob: tutorials/intro_to_py5_and_python_*. If you rename those files to match the pattern, they will be included. However, the tutorials are probably not in the right order. Jupyter book will sort the globbed filenames lexicographically when doing the build. You can address this by inserting a strategically placed zero padded number (01, 02, 03, etc) to each filename or by listing out the filenames individually in _toc.yml.
• Your tutorials when rendered don't have page titles at the top. Compare the iteration tutorial with the introduction to py5bot tutorial. To change that, you just need to change the first heading in each notebook to have one # instead of two (change ## iteration to # iteration).

Let me know if you have any git issues and I will help you! Have a good day, and thank you for your hard work!

[hx2A]

The git commands I believe you'll need are:

git remote add upstream [email protected]:py5coding/py5book.git
git checkout -b zellegsoc upstream/zellegsoc

Alternatively, you can also create a new fork and uncheck the "fork only main branch" checkbox that github presents you. That will give you a new repo with the zellegsoc branch.

When you get a chance, try making a new PR to the zellegsoc branch with just the heading changes so we know we both have the git workflow stuff operating correctly.