Conceptualizing Ideas for a Task V4

[ghostsquad]

I've been watching the discussions and issues rolling in. Taking a hard look at the current documentation and functionality, and I've decided that the best contribution I can make to Task is a re-envisioning. I'd like to keep task looking familiar, but some of these ideas will be backwards incompatible, thus, why I'm saying "v4".

I'm not quite ready to actually showcase anything yet. But I have a vacation from work coming up, nice 9 day vacation, which I intend on using to make progress on this. But I can say that I'm taking inspiration/functionality from several other tools which I'm familiar with:

1. Make - There are some features of Make that still don't exist, and I think those features, when implemented properly could really simplify things.
2. Github Actions Workflow - The syntax for github actions workflows is concise, simple, and I love the verbs they use. There's some functionality there too that I want to lean on.
3. ConcourseCI - There are some verbs and functionality I like from ConcourseCI.
4. Jsonnet - The ability to do lazy evaluation is something I'm in-love with in Jsonnet. I think that some of the current issues I've seen with Variable priority and when they are evaluated could be solved if a lazy paradigm is used.
5. Earthly - The ability to define your "environment" is an invaluable tool. Doing everything in a docker container for consistency has incredible value. Similar to why Github Actions is so great. I'm imagining native integration with running tasks in containers, similar to Earthly, or GHA's uses, and container fields/syntax.
6. Hugo - Hugo is built in go, and uses go templating as well to do amazing things. I want to really lean into the power of templating, and allow folks to do even more using this functionality.
7. Finally, Task itself. I love the simplicity of task, and I strongly believe in API designs that require less code, less configuration, not more, even when you want to use powerful features.

I would like to share more very soon. If you have any questions, let me know.

[per1234]

If you are making a hard fork, I recommend renaming your project (while still crediting its roots in Task of course). There should only be one Task, and that is the real Task which is maintained in this repository. A major version bump is not sufficient to differentiate your hard fork because the real Task may well bump to 4.x at some point as well.

Please contribute back any work you do in your project which is aligned with the direction of the Task project. Do it early and often because once your code base has diverged significantly it will become more difficult.

[ghostsquad]

@per1234 I don't intend on making this a hard fork. I intend on contributing this to this repo as a major version bump. Following Semver, major versions are the time to make backwards incompatible changes.

[andreynering]

Hi @ghostsquad,

First of all, make sure you also enjoy your vacations and rest. Coding is secondary, life is priority. 🙂

That said, some thoughts:

1. It's true that major versions allow backward incompatibilities, and we certainly have done that before. It's still interesting to keep it easy to upgrade to a newer version when possible, though. These decisions usually are decided on a case-by-case basis.
2. Specially for bigger features / more difficult work, it may be useful to discuss a bit before implementing. For small / easier things, it's fine to open the PR directly if you want.

The ability to do lazy evaluation is something I'm in-love with in Jsonnet. I think that some of the current issues I've seen with Variable priority and when they are evaluated could be solved if a lazy paradigm is used.

3. I'm curious about this one, as I never heard about Jsonnet before.

4. Regarding Docker support, there was a tentative to do that in the past, and the person eventually created hammerkit, his own Task alternative (in Typescript) with Docker integration. I didn't look too close, but hammerkit seems to have handled it in a nice way, I planned to get some inspiration from there if I ever worked on Docker support. I'm open to hear about other possible approaches, though.

There's a good number of opened PRs that could be reviewed already. Once you open yours, I'll try to give it some attention. 🙂

[ghostsquad]

Thank you for the kind words. My vacation will be filled with many fun things and relaxing "do nothing" days. I also really like to code, and this is one project I'm passionate about contributing to.

I decided to start this discussion thread now, because I'm starting to get some code down, and I'm excited about the possibilities I'm seeing.

I will definitely want to show case more when I'm able to finish a proof of concept.

I'm also considering the upgrade path for folks, and considering specifically what is required, or if a conversion utility is needed or would help. Something similar to what terraform has done, or the concept from kubernetes, called round-tripping, which is to the ability to convert an old schema to a new schema or vice versa, which is how they handle vXbetaY differences during development.

To be honest. A major reason I'm considering this v4 is because there is behavior that just doesn't exactly make sense. Obviously, changing that behavior would be a backwards incompatible change.

So I wanted to really take a hard look at what I wish task was really capable of. Simplicity of yaml, with logic using shell and templating.

I look forward to showing you more, and hope I can get you excited for this like I am. Stay tuned!

[ghostsquad]

I'll start dropping some hints & notes here.. the code is still very much a WIP.

I've started this first with just defining structs, names, verbs, and go doc comments to describe the desired functionality I'm aiming for (ReadMe driven development). Implementations will come later.

all of the new stuff currently exists under pkg sdk right now.

https://github.com/ghostsquad/task/tree/feat/v4/sdk

Here would be a good place to start looking around.
https://github.com/ghostsquad/task/blob/feat/v4/sdk/v4/taskfile/taskfile.go

** EDIT: 2022-04-06 - refactored to sdk/v4 directory.

✍️ Feedback is welcome! 😡 😐 😄 🧐 🤔

[andreynering]

Hi @ghostsquad,

Didn't have the time to read all of it (and maybe it's even not necessary yet), but I did a quick look at it.

1. I see many files have comments like this:

Copyright 2014 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Does this mean that you copied these files from the Kubernetes repository? If so, can you point me which repo so I can take a look? My knowledge about licenses is limited, but we'd need to make sure we can do that, and discuss an alternative otherwise.

2. I like much of the work you did on restructuring some directories, better documenting structs, etc. The project kinda needed it, since much of the structure is from the beginning of the project and only changed when needed.

I just wonder if having a big v4 branch with many changes is the way to go. That happened for v3 and I kinda regretted it in the end, although depending on the scenario it's a bit unavoidable. It would somehow be easier we if we had multiple PRs targeting master, one for each feature/change, so they can be reviewing independently. Those would be preparation for the release. Renaming github.com/go-task/task/v3 to v4 for example would only happen right before the final release.

But all this are just thoughts. I know it's just a WIP and if you want to advance a bit more before we decide that's fine. 🙂

[andreynering]

/cc @kerma @tylermmorton in case you have thoughts...

We may see a v4 soon. @tylermmorton started some proposals as part of #665, #667 and #668 that may somehow fit on this release.

[ghostsquad]

@andreynering regarding the license. Yes we absolutely can use the code, in fact, the license (attribution) is in the file because that's part of the requirements to use the code. You can find a TLDR of the license at tldrlegal and at snyk.io

Regarding the version bump, there are several considerations. The first is that, unless someone is actually using task code as a library, then much of what we do with the v3/v4 suffix doesn't matter. But for the use case that we want to allow a programmatic API, I'll assume this does matter.

The Task binary currently is able to parse v2, v3, and we'll soon want to support v4 yaml files, meaning that the binary itself should be backwards compatible, thus it will need access to the old versions of the engine/vm/behaviors. As such, it makes sense to refactor the code base into v2/v3/v4 subdirectories, since the binary must support all versions.

I think the branch based approach to major versions works fine when there's no promise that the CLI or library will support multiple versions of client code (in this case the yaml files).

For reference, if we were to go with branch based major versioning, I would allow master to always point to the latest major version, and would create branches for the preceding versions in order to allow new bugfixes to be applied to old versions.

As long as a tag is created for each release, I believe this works just fine.

[ghostsquad]

@andreynering I'll update comments with links to the original code as well (for the stuff I'm pulling from Kubernetes).

The label selectors, which is the primary feature, is here: https://github.com/kubernetes/apimachinery/blob/release-1.23/pkg/labels/selector.go

[kerma]

I've been thinking about v4 roadmap for a while now as well, but been a bit too busy to write it down. I'll put my ideas into a separate post here for better threading.

First, I think we and/or @andreynering needs to define the goal of the project if it's going to diverge from the existing one:

Task is a task runner / build tool that aims to be simpler and easier to use than, for example, GNU Make.

Adding features will probably make the tool more complex and harder to use, so I'm always careful of introducing new features. Specially if we want to keep that vision for task. If we do, I see v4 as a way to clean up some UX issues.

One of the biggest problems seems to be the confusion and usability of variables and environment variables: how they are evaluated, when loaded, used in preconditions, inherited between included files etc. This is probably an area I'd rethink completely from scratch for something like v4.

Second area which users (and I personally) seem to struggle with is task re-usability and modularity. There are known workarounds for most issues, but they are often cumbersome or require a lot of boilerplate code. As an example, a pattern I'm now using almost everywhere is to define reusable tasks without descriptions and higher level tasks calling those "internal" tasks for end users. Something like

deploy:
  preconditions:
  - sh: test -n "{{ .env }}"
    msg: "missing env variable"
  cmds:
  - sh script/deploy.sh {{ .env }}

dev:deploy:
  desc: "deploy to dev"
  cmds:
   - task: deploy
     vars:
       env: dev

But this gets very verbose and repetitive quite fast. It also pollutes the task list for the user (each deploy task is listed separately for each environment).

Third area, somewhat related to the previous one, is passing generated/computed/calculated data or values around. There are no elegant solutions for it atm and it's often error prone, partially due to mvdan/sh implementation details. I've solved it using temporary files in .task, similarily how the checksum implementation, but it would be nice to get a first class support for it in the tool.

That brings me to the last area which is the CLI part. Using task as a general purpose CLI builder would be pretty cool imo. So having a simple way to define tasks as

$ task job context action -- -key value -foo bar

would allow users to build custom CLI's. Technically you can do that today, but the amount of boilerplate required makes it less attractive.

To conclude, I think for v4 we should first define what are the main problems we'd like to solve there. Whether the solutions will be inspired by Github Actions, solved by lazy evaluation or more templating is secondary imho.

[tylermmorton]

+1 to UX cleanup like @kerma mentioned. Environment variables and task variables are a common source of confusion... I'm not sure how much sense this would make but perhaps eliminating task variables from v4 and providing more support/tooling around environment variable management could be a solution.

I'd like to see the usage of mvdan/sh change as mentioned in #668 -- Having all commands for a task run in the same shell could open up some possibilities for UX improvements.

[ghostsquad]

First, I think we and/or @andreynering needs to define the goal of the project if it's going to diverge from the existing one:

Task is a task runner / build tool that aims to be simpler and easier to use than, for example, GNU Make.

Adding features will probably make the tool more complex and harder to use, so I'm always careful of introducing new features. Specially if we want to keep that vision for task. If we do, I see v4 as a way to clean up some UX issues.

I'm very familiar and experienced with writing software that is simple to use. Simplicity and feature-set are not mutually exclusive. My goal is the same as above.

One of the biggest problems seems to be the confusion and usability of variables and environment variables: how they are evaluated, when loaded, used in preconditions, inherited between included files etc. This is probably an area I'd rethink completely from scratch for something like v4.

I agree. As part of my work, I'm thinking of this like a game. A game has rules, and everything, including players, MoBs, and NPCs follow those rules. With a simple system/set of rules, my desire is that it becomes much more intuitive than it currently is.

Second area which users (and I personally) seem to struggle with is task re-usability and modularity. There are known workarounds for most issues, but they are often cumbersome or require a lot of boilerplate code. As an example, a pattern I'm now using almost everywhere is to define reusable tasks without descriptions and higher level tasks calling those "internal" tasks for end users. Something like

deploy:
  preconditions:
  - sh: test -n "{{ .env }}"
    msg: "missing env variable"
  cmds:
  - sh script/deploy.sh {{ .env }}

dev:deploy:
  desc: "deploy to dev"
  cmds:
   - task: deploy
     vars:
       env: dev

But this gets very verbose and repetitive quite fast. It also pollutes the task list for the user (each deploy task is listed separately for each environment).

I also 100% agree with this, and this is also top of mind. I hope to get enough code written soon that I can show a proof of concept of how I think this can be improved.

Third area, somewhat related to the previous one, is passing generated/computed/calculated data or values around. There are no elegant solutions for it atm and it's often error prone, partially due to mvdan/sh implementation details. I've solved it using temporary files in .task, similarily how the checksum implementation, but it would be nice to get a first class support for it in the tool.

I'd like to go more into this with you. Do you have an example Taskfile I can play with?

I think though this can also be solved with lazy variable evaluation (like in jsonnet, or promises in JS).

That brings me to the last area which is the CLI part. Using task as a general purpose CLI builder would be pretty cool imo. So having a simple way to define tasks as

$ task job context action -- -key value -foo bar

would allow users to build custom CLI's. Technically you can do that today, but the amount of boilerplate required makes it less attractive.

YES YES. I've thought about this, and I think it would be pretty damn cool.

To conclude, I think for v4 we should first define what are the main problems we'd like to solve there. Whether the solutions will be inspired by Github Actions, solved by lazy evaluation or more templating is secondary imho.

This is a great start. Thank you. I originally thought about using problematic scenarios like this as my baseline, but decided to skip this step, because it helped my conceptualize a bigger picture, 100k m view. Of course this is very very helpful to get in writing. Teamwork makes the dream work. 🍻

[ghostsquad]

Oh.. I just found out that mentions of this discussion (from other issues) don't add links to those other issues here. 😭
If this was a standard issue, I'd be able to easily track those mentions. I might want to just start a PR or issue instead of discussion in order to get that feature of Github.

[andreynering]

Sad thing.

Found a feedback about that here: community/community#2971

Already added my 👍 there

[ghostsquad]

adding my 👍 too

[ghostsquad]

I've been looking at some really clean looking Makefiles recently. I think that's another reason I want to rename fields in v4. Just to cut down on the verbosity, without changing explicitness.

Example:

[ameergituser]

My main reason for switching from Make to Task is because Make can and generally does become "clean" yes, but unreadable (context: building C code). After not writing a Makefile for a couple of months it's very hard to figure out what is going on. That's why I appreciate the verbosity of Task.

But yes It can probably be improved.

If we can read and understand a Taskfile after not writing or reading one for many months, and not reading the documentation - I think this is a good metric/goal.

For this to be achieved, I believe Task should be intuitive, and the documentation should be embedded in the Taskfile. Which is to say it documents itself in a way.

[ghostsquad]

@ameergituser I 100% agree.

[therealjasonkenney]

A nice feature that could help with some of mine (and my teams) use of Task would be the ability to just specify a shell script to setup the environment, a dotenv file is nice, but especially in CI and environments where we need to fetch secret values, right now it makes for very verbose task calls. Something similar to how release.sh is used with elixir releases would be nice.

[ghostsquad]

A nice feature that could help with some of mine (and my teams) use of Task would be the ability to just specify a shell script to setup the environment, a dotenv file is nice, but especially in CI and environments where we need to fetch secret values, right now it makes for very verbose task calls. Something similar to how release.sh is used with elixir releases would be nice.

I feel like I'm missing something here, but also not familiar with elixir releases. How does this differ from either specifying a task with a multiline cmd? e.g.

tasks:
  setup:
     cmds:
       - |
         echo "step 1"
         echo "step 2"

Or deferring to an sh file, e.g.

tasks:
  setup:
     cmds:
       - ./setup.sh

Or do mean some way of essentially running a task without an Explicit Taskfile.yml? E.g.

task --url [email protected]/me/home.git setup

(Implicitly clone and looking for Taskfile.yml at the root of the repo)

Or

task --url https://example.local/task setup

An arbitrary url that returns application/yaml content