[DX/tests] - Update doc tutorials programatically for any PR change

[camilamacedo86]

What do you want to happen?

Description

Currently, we have the following samples which are used in the docs:

• Component Config: https://github.com/kubernetes-sigs/kubebuilder/tree/master/docs/book/src/component-config-tutorial/testdata
• CronJob Tutorial: https://github.com/kubernetes-sigs/kubebuilder/tree/master/docs/book/src/cronjob-tutorial/testdata
• Multiversion-tutorial: https://github.com/kubernetes-sigs/kubebuilder/tree/master/docs/book/src/multiversion-tutorial/testdata

You can see that the docs are using the samples and the info provided on them are comments in the source code generated.
However, it has been very thought we keep updated with the changes performed in the scaffolds. Currently, we need to:

• Run the helpers scripts under it samples dir (generate_cronjob.sh, generate_multiversion.sh, generate_componentconfig.sh )
• Run make all
• Then, compare the local branch with the master branch and then re-add the missing files and all code generated for the samples as the comments back
• Run make all again to ensure and compare again to see if we are not missing anything

Goal

The goal of this task is we programmatically ensure that all samples will be updated for any pull request sent to the project. It. will help us keep the docs and update and improve the DX for contributors. Also, it can help us check the changes in the review process which also help us to easily ensure them.

Note that we have a shell script generate.sh which is called via the makefile target [make generate] (https://github.com/kubernetes-sigs/kubebuilder/blob/master/Makefile#L65-L67) to re-gen all samples under the testdata dir.

See that in the CI we check if the samples are updated or not and we do not lot contributions that change the scaffold go forward and get merged without the authors ensuring that they executed the make generate locally to keep all updated. (You can check the GitHub Action: https://github.com/kubernetes-sigs/kubebuilder/blob/master/.github/workflows/testdata.yml)

Proposed Solution

• We use the Test Context utils to run all commands to generate the book testsdata samples, e.g: https://github.com/kubernetes-sigs/kubebuilder/blob/master/test/e2e/v3/generate_test.go#L132-L231
• Then, we use the utils to replace and insert the code source of the book samples as its comments back.

Note that we do something very similar in the SDK project using the same implementations to ensure that is samples under testdatawill have the required code changes. See: https://github.com/operator-framework/operator-sdk/blob/master/hack/generate/samples/internal/go/v3/memcached_with_webhooks.go

• We also need to call this implementation in the make generate target to ensure that it will be re-gen as tested in the CI.

Extra info

• We can have one PR to implement each book data to split the required work and make it easier
• I'd recommend we get started by automating the Component Config because this one is a small project in the book
• Be aware that for the samples CronJob Tutorial and Multiversion-tutorial the shell scripts generate_cronjob.sh, generate_multiversion.sh are missing steps and we usually need to add files back. In this case, would be nice to follow all the whole docs/tutorials and ensure that we are following the exact same steps that you are asking users to do. It will also help us ensure our guidance and please feel free to push PR's contributions to the docs if you find any issue with them.

Extra Labels

No response

Implementation solution for: #782

Current Status

• - Component Config is done: 📖 : update ComponentConfig tutorial #3181
• - CronJob Tutorial is done; 🌱 Update cronjob-tutorial automatically #3231
• - Multiversion-tutorial: TODO

[jmrodri]

This proposal makes sense to me. Anything that will improve keeping the samples updated is great.

[shuheiktgw]

Hi @camilamacedo86, can I work on this issue as my first contribution to kubebuilder project? 🙏

[camilamacedo86]

Sure, I'd recommend we begin by trying. to check this approach with the Component Config because this one is a smaller sample the docs
.

[shuheiktgw]

/assign

[shuheiktgw]

Sorry that I could not find time to work on this for the past few weeks, but I've finally started working on it 🙇

[camilamacedo86]

That is fine. No worry.

[shuheiktgw]

Sorry, let me clarify what we need to do here in the component-config's case as an example!

Then, we use the utils to replace and insert the code source of the book samples as its comments back.

I'm not sure where "the code source of the book samples" is located... Would you give me some pointers to it?

[AlmogBaku]

@shuheiktgw
If I understood correctly:

# enter the project
cd <kubebuilder>
# enter the docs src
cd docs/book/src/
# create testdata for each example
cd cronjob-tutorial/testdata/
./generate_cronjob.sh
cd ../../multiversion-tutorial/testdata/
./generate_multiversion.sh
cd ../../component-config-tutorial/testdata/
./generate_componentconfig.sh
# manually add/modify missing files and comments
cd ../../
git dif
echo "good luck with that! this is actually the hard task we trying to automate with the replace"
# remake all to ensure we didn't break the examples
cd cronjob-tutorial/testdata/ && make all && cd ../..
cd multiversion-tutorial/testdata/ && make all && cd ../..
cd component-config-tutoria/testdata/ && make all && cd ../..
# regenerate the proje
cd ../../../
make generate
echo "well done. clap yourself 👏🏽! and wish good luck for your reviewer"

[camilamacedo86]

Hi @AlmogBaku,

If you are looking to know if that are the steps required to update the samples inside of the doc:
Mainly yes.

• You do not need to run make generate: make generate generates the testdata samples based on the source code changes. In this case, you are just re-scaffolding the samples used in the doc and re-adding all examples/code and comments on top again.
  PS.: If you are looking to do this process I'd recommend you do project by project and not as you added ( first re-scaffold the cronjob, add bake all examples on top again and etc .. then move to the next one.

If you are asking if the automation will be like a shell script doing these steps:
No. That is not the idea or goal.
The proposed solution for automation will perform the steps and add the code/examples on top again very similar how we create the e2e tests in the project and as we do that in the SDK.

[camilamacedo86]

Hi @shuheiktgw,

Are you working in this one?
If not, could you please unsigned so it can let others try to achieve this goa?

[camilamacedo86]

Hi @ shuheiktgw,

Are you working on this one? If you need help please feel free to ping your questions into the kubebuilder slack channel. If not, could you please remove your assign so that others might want to looking on this one?

[shuheiktgw]

Sorry that I didn't notice the message and I'm not working on this issue so let me unassign myself 🙇
/unassign

[Kavinjsir]

@camilamacedo86 Thx for bringing this great idea! I'm trying to understand the workflow you suggest and list as below, would you like to fix me if anything wrong?

Idea

In brief, we hope every PR can keep the sample sync with the kubebuilder behavior.

---

Similar Scenario

Such attempt is similar as how we test the kubebuilder behavior through CI:

1. For each uploaded PR, the CI is triggered to execute make generate.
2. make generate removes /testdata and re-generate it.
3. After make generate is finished, CI would then check if the content in /testdata is the same as what it is on the current branch. (git diff)
4. If not, it means the kb behavior is changed in the PR, and the corresponding testdata is not sync with it.
5. The author of the PR should then run make generate locally to verify that the proposed kb behavior works on test data.

---

Proposed workflow

So, what proposed in the issue is: apart from /testdata, we would also like to see /docs with the corresponding samples in its sub-folder can be synced with every PR:

1. The author of the PR should update /docs (OR verify if there is any necessity to update /docs) based on his/her original code updates in the PR.
2. If there is any changes to the /docs, they should be included in the commit.
3. The CI should have additional steps to verify if /docs keeps sync with the updates of the PR.

---

Challenges

1. We need to initialize a new script that jumps between different folders under /docs to run various scripts to overwrite the files of the samples.
2. We may depend on the existing scripts (./generate_componentconfig.sh, ./generate_cronjob.sh), however, it is necessary to review these scripts:
   • Are they following the documentation?
   • Are they following the current main version of Kubebuilder?
3. We need to add new workflows in github action to run additional steps during CI to check /docs

[Sajiyah-Salat]

Thanks @Kavinjsir for your brief answer. I got some idea on how things work and what needs to change. Hey @camilamacedo86 is thier anything to add in @Kavinjsir comment.

[camilamacedo86]

HI @Kavinjsir,

That #2510 (comment) is a great summary 🥇 . The proposed solution to sort out the challenges would be to do scaffolds as operator-sdk does (using kb as lib) to generate the samples with code on top, see:

• code implementation is under: https://github.com/operator-framework/operator-sdk/tree/master/hack/generate/samples
• see the Memcached sample been generated and code added on top: https://github.com/operator-framework/operator-sdk/blob/master/hack/generate/samples/internal/go/memcached-with-customization/memcached_with_customization.go
• it is called in the make generate : https://github.com/operator-framework/operator-sdk/blob/master/Makefile#L42
• the result can be found in: https://github.com/operator-framework/operator-sdk/tree/master/testdata/go/v3

Therefore, we would no longer need the shell scripts. We would call kubebuilder binary, run the commands to generate the docs samples, then in golang inject the code on top ( using the methods to insert, and replace the string as we do in the plugins. example to insert code in the controller )

[Eileen-Yu]

HI @Kavinjsir,

That #2510 (comment) is a great summary 🥇 . The proposed solution to sort out the challenges would be to do scaffolds as operator-sdk does (using kb as lib) to generate the samples with code on top, see:

• code implementation is under: https://github.com/operator-framework/operator-sdk/tree/master/hack/generate/samples
• see the Memcached sample been generated and code added on top: https://github.com/operator-framework/operator-sdk/blob/master/hack/generate/samples/internal/go/memcached-with-customization/memcached_with_customization.go
• it is called in the make generate : https://github.com/operator-framework/operator-sdk/blob/master/Makefile#L42
• the result can be found in: https://github.com/operator-framework/operator-sdk/tree/master/testdata/go/v3

Therefore, we would no longer need the shell scripts. We would call kubebuilder binary, run the commands to generate the docs samples, then in golang inject the code on top ( using the methods to insert, and replace the string as we do in the plugins. example to insert code in the controller )

Thx @camilamacedo86 , just initialized a PR to start with the component-config, could you help with review?

[Eileen-Yu]

/assign

[camilamacedo86]

For we close this one we just need start to generate the Multiversion-tutorial see: https://github.com/kubernetes-sigs/kubebuilder/tree/master/docs/book/src/multiversion-tutorial/testdata/project as we do that for others now, more info: https://github.com/kubernetes-sigs/kubebuilder/tree/master/hack/docs

A help here is very welcome since if we be able to do that it will reduce a lot our effort to keep the things maintained and also help us to better ensure the project and that our samples are still working basically after the changes.

[camilamacedo86]

Hi @Eileen-Yu

You did an amazing job here. So, because it is only missing one tutorial to make easier others understand what we need to do I create an issue with all info here: #3880

If you wish to contribute within we will be more than happy to receive your contribution.
therefore, we are closing this one as done and in favor of : #3880

[Eileen-Yu]

yeah the most work of this task is done except for 1 tutorial.
My apology for not having sufficient time now to finish all of them, but I'll be super glad to help with review if anyone would like to contribute to it!