Support Actions v2

[Djiit]

Just to keep track of the work here :)

Thanks for you awesome tool !

[xmly]

Any available parser to parse the YAML syntax?

[eine]

Any available parser to parse the YAML syntax?

go-yaml/yaml

[cplee]

This one is gonna be fun! I'll be looking at this in the next few days.

[soderlind]

fwiw, you can use sclevine/yj to convert existing hcl to yaml

[eine]

fwiw, you can use sclevine/yj to convert existing hcl to yaml

Do you know the difference between a general solution such as the one you comment or github's own instructions (https://help.github.com/en/articles/migrating-github-actions-from-hcl-syntax-to-yaml-syntax)? Just wondering. If yj works, it seems much easier to install/use.

[soderlind]

Sorry, complete HCL n00b. I did a convert and it looks ok (quick and easy to):
https://gist.github.com/soderlind/97071f96f72a42f4767db205a3e48b8a

I'm on a mac, installed yj using brew: sclevine/yj#5 (comment)

[eine]

@soderlind thanks for the example. After having a look at it, I think that the main point is the semantics have changed, not only the syntax. See an actual example of a GHA v2 YAML file here: https://github.com/1138-4EB/ghdl-docker/blob/master/.github/workflows/base.yml Therefore, even though the output of yj is valid YAML, I'm afraid it won't be accepted by the backend in GitHub.

[soderlind]

@1138-4eb This might help, it's an official tool( written in go) from GitHub: https://github.com/actions/migrate

I converted the same .hcl file as above, here's the result:
https://gist.github.com/soderlind/2f5a0ffcc9a20cd5024a8aa4adcd4b2f#file-push-yaml

[eine]

@1138-4eb This might help, it's an official tool( written in go) from GitHub: https://github.com/actions/migrate

I converted the same .hcl file as above, here's the result:
https://gist.github.com/soderlind/2f5a0ffcc9a20cd5024a8aa4adcd4b2f#file-push-yaml

Yes. This is the 'official' solution, which is mentioned in the article I linked above.

[anoff]

One main challenge might be the fact that actions v2 are not necessarily dockerized. You would need to come up with act runtime that mirrors the agent behavior by running each job in separated processes mirroring the virtual environment.
Also the setup- actions (e.g. setup-python) might require some additional work to support those out of the box without mirroring each setup manually into the act ecosystem.

[cplee]

The point that @anoff made ☝️ is making this a very difficult exercise. Additionally, actions can be implemented in Javascript which requires act to replicate the runtime environment. Curious, how many folks would want to use act in this way? Would it make sense to just support the docker actions without the javascript actions?

[anoff]

I would be fine with a minimal implementation. Most of my personal workflows are actually bash based so spawning a docker and injecting script code into it would solve my needs.
Docker also makes a fine abstraction for any other tasks.

But I guess if the correct extension points are available the community will quickly provide support for nodeJS.

Maybe we can also get GitHub to pick up on this; having a way to dry-run workflows locally is a huge differentiator on the market. The only other CI service I know of is drone-ci.

With those big words being spoken; I would love to help out where I can. But I'm only a beginner in Go so you would need to guide the architecture 🙃

[case]

Dry-runs or even just syntax checking are what I used the most in the tool's previous incarnation, fwiw.

Thanks for your work on the project!

[eine]

The point that @anoff made ☝️ is making this a very difficult exercise. Additionally, actions can be implemented in Javascript which requires act to replicate the runtime environment.

While I find the point made by @anoff relevant and pertinent, I find it more difficult to understand how do you handle the context. Any GitHub Action written in JavaScript/TypeScript can be executed in a default node container. Should the action need to run additional containers, the socket on the host can be shared with the node container. However, how do you replicate the context variables for those workflows that depend on them?

Furthermore, I think that the name/title is misleading. AFAIU, this repository is about running workflows locally, not actions. I believe that executing actions locally is pretty straightfoward.

Maybe we can also get GitHub to pick up on this;

I totally agree. However, they seem to be having a hard time to give timely GHA related support.

[anoff]

Good point. Having docker as the runtime for each workflow (step?) should allow dealing with nodeJS functions as well.

What is the big difference between workflows and actions? I agree that we actually want to run an entire workflow and not a single action; but from a solution point of view there is not much difference except that workflows also need to support handling the workflow between multiple actions. Am I misunderstanding your point @1138-4eb

[eine]

What is the big difference between workflows and actions? I agree that we actually want to run an entire workflow and not a single action; but from a solution point of view there is not much difference except that workflows also need to support handling the workflow between multiple actions. Am I misunderstanding your point @1138-4eb

I think that the big difference is that actions don't require a parser and/or knowledge of complex semantics:

In order to execute an action, only action.yml needs to be read. While a full-featured YAML is useful, the syntax/semantics are so simple that some regex would also fit. There is a single decision to take:

• Execute natively.
  • JS / TS Actions, as defined by GitHub.
  • Shell scripts. The entrypoint to Docker Actions, but without using a container.
  • Any other bash, python, go... step.
• Execute in a container.
  • Docker Actions.
  • JS Actions in a node container.

Conversely, in workflows semantics are more complex, undocumented in a machine-readable format, and subject to change frequently. The complexity of reading the YAML file and/or executing individual tasks/actions/steps is negligible compared to understanding the workflows that users are trying to describe.

Overall, while I find the approach of executing workflow YAML files directly to be interesting/useful, I think it should be put aside until execution of actions/steps is properly supported. This is because execution of workflows is dependent on been able to execute steps.

I believe that execution of actions alone does pose some interesting issues/problems to work on:

• How to handle actions that are expected to be executed on windows-latest or macos-latest.
  • On windows, it is possible to run either linux or windows containers, but not both of them at the same time. Unfortunately, AFAIK, changing the type cannot be done programmatically without permanent damage (i.e. a hard reset of the docker installation). Therefore, windows-latest actions might need to be restricted to be executed on the host. As a result, I believe that setup-* actions should be blocked. Still, some other actions might do permanent modifications on the system, and it might be really hard to detect it. This is because actions are designed to be executed in a temporal environment, so many developers might not care about cleaning up.
• How to support actions that spawn docker containers, without using/requiring docker-in-docker. This is doable with or without volumes and with or without patching paths in the sources of actions.
• The contexts, as commented above. Atomic/independent actions can be easily executed. But, what about all those that depend on the contexts provided by GH? job, steps, runner, strategy and matrix can be replicated. However, github and secrets can be really hard to provide. Again, it is difficult to detect when a step/action needs any of these. This is specially so because the token is shared with all of them by default.

---

FTR, I'm working on 1138-4EB/issue-runner, which has a different target than this project but which shares some concepts/features. The focus of issue-runner is on executing MWEs that are defined in a markdown file (or the body of an issue in a github repo). It supports executing MWEs either natively or inside docker containers, but it is expected to be mostly used with containers (using nightly images provided by project maintainers). It is written in Go too.

Hence, some of the issues/problems I listed above must be solved for both projects. I don't expect to support neither windows containers nor macos hosts in issue-runner; but I'd be open to sharing some of the lower level task execution codebase, should that make any sense to maintainers of act.

---

Maybe we can also get GitHub to pick up on this; having a way to dry-run workflows locally is a huge differentiator on the market. The only other CI service I know of is drone-ci.

Upon further thought, I don't know if GitHub will want this project to ever take off. The point is that a fully functional act tool would allow to use GitHub actions not only locally, but also on absolutely any other CI service.

[paulriley]

I would argue for running everything natively, except where the user provides a container property on a job.

That way, if I want to run Node actions, I can choose whether to install Node locally or provide a container.

If the software is opinionated about this, then I no longer have a choice. This is particularly obstructive to those who (for whatever inexplicable reason) wish to build on Windows and run on Linux because, as you rightly point out, you can't have a docker service that runs both in containers. But it affects everyone's ability to run a similar setup locally and remotely.

My instinct is that GHA best practice will be to provide a build container to a job anyway, so that you're in control of updates.

[eine]

I would argue for running everything natively, except where the user provides a container property on a job.

Note that this sentence can be inverted and the result is the same:

I would argue for running everything in containers, except when the container image to be used cannot be determined.

I.e. some actions must be executed natively and others must be executed in containers. Any tool should support both types/options. IMHO, there is no default.

That way, if I want to run Node actions, I can choose whether to install Node locally or provide a container.

As commented above, I think that installing dependencies locally is not an issue. The main point about executing actions natively is security. Anyone can update the tag of an Action without prior notice, and you will be executing arbitrary code on your host. Currently there is no mechanism for you to check that the version (codebase) of the action is the same that you reviewd/checked. Docker containers are not a sandbox, but some isolation is provided at least.

[paulriley]

I.e. some actions must be executed natively and others must be executed in containers. Any tool should support both types/options. IMHO, there is no default.

Except where actions leave a choice. Simplistic example, but I can run these actions here on a container or not:

https://github.com/paulriley/github-actions-experiments/tree/master/.github/actions

Workflow with a container:
https://github.com/paulriley/github-actions-experiments/blob/master/.github/workflows/workflow.yml
Running this locally should launch the container to run all steps in.

Workflow without a container:
https://github.com/paulriley/github-actions-experiments/blob/build-native/.github/workflows/workflow.yml
Running this workflow locally leaves the question: Should each step run locally or in a container? I'm arguing it should be local, because it will be local (on an "ubuntu-latest" build server) on github. That's all.

As commented above, I think that installing dependencies locally is not an issue. The main point about executing actions natively is security. Anyone can update the tag of an Action without prior notice, and you will be executing arbitrary code on your host.

If you're not 100% confident in the source then you can use a commit ID instead of a tag. That's a problem that doesn't need solving.

https://help.github.com/en/articles/about-actions#versioning-your-action

[eine]

Except where actions leave a choice.

An Action merging the features of two types of Actions is technically the same as two independent Actions, one of each type. The user that writes the workflow will decide which type to use, and any runner tools just needs to support and honour it.

Running this workflow locally leaves the question: Should each step run locally or in a container? I'm arguing it should be local, because it will be local (on an "ubuntu-latest" build server) on github. That's all.

I don't want to install all the dependencies required to run a workflow by any means. If your host is an Ubuntu and you are willing to install all the tools that are available in ubuntu-latest, it's ok, and that's a use case that should be supported. However, for all of us using Fedora, Arch Linux or Windows hosts, running this type of workflows in a container or VM is a requirement, not a preference.

Once again, I'm not arguing against allowing to execute workflows natively. I'm just trying to explain why that's shortsighted and that this tools should account for other multiple use cases (at least, according to the current description). Precisely, that's the complexity, not only understanding the semantics of the workflows, but applying additional user-provided constraints.

If you're not 100% confident in the source then you can use a commit ID instead of a tag. That's a problem that doesn't need solving.

https://help.github.com/en/articles/about-actions#versioning-your-action

I thought that only tags and branches were supported. Thanks a lot for the reference!

[paulriley]

Once again, I'm not arguing against allowing to execute workflows natively. I'm just trying to explain why that's shortsighted and that this tools should account for other multiple use cases (at least, according to the current description). Precisely, that's the complexity, not only understanding the semantics of the workflows, but applying additional user-provided constraints.

I don't think we're necessarily disagreeing, I also think both options should be available. I guess the real question is the mechanism by which the user gets to decide. I'm suggesting that the mechanism already exists as a property of a job in the workflow and there's no need for any other method.

That is, if you're developing on Fedora but building on Ubuntu, you're probably better off using an Ubuntu (or indeed Fedora) container, using an image specified in the workflow, rather than running natively -- either local or remote.

All that said, I see your argument and I can see a case for a --image argument on act (or as a config setting), just in case someone wants to run actions in a container locally and natively on the build server. I just wouldn't worry about it for early development, because a mechanism is built into GHA Workflow syntax that needs supporting either way.

[aengelberg]

I've recently begun the lights-camera-action project (to support gh-actions-orb), which is in a super rough (but working) state at the moment. When it's feature-complete, it will handle the challenge of executing individual third-party actions. So given a repo-name@tag and some action inputs it will clone the repo, build any necessary Docker images, run Node, capture logging commands from stdout, etc. That sounds like one piece of what y'all are trying to do here, so just mentioning it in case it's helpful.

[cplee]

@aengelberg looks awesome...and just like what i'm needing. Thanks for sharing!

[BoforsKing]

Just curious about this package supporting YML syntax as GitHub Action v2 got rid of HCL.

[eine]

@BoforsKing, did you read the comments above?

[BoforsKing]

@BoforsKing, did you read the comments above?

I got confused half-way and didn't understood the other half to be honest. If it does, maybe some update to the readme could help to clarify how.

[prologic]

@eine I understand; but as act is a tool to run GHA workflows locally it would not be a very useful tools if users would have to go and manually muck around with the same workflow just to ge tthem to work locally. Yes I agree with you that piling everything into an image you need is going to be problematic (mostly size issues) but still. Also runniner DinD isn't altogether hard it just requires some special treatment.

That being said, this is shaping up to be a pretty good tool! I think the end goal should be to try and run existing GHA workflows without modification/fuss as much as possible :)

[prologic]

@cplee Ahh very nice! Thank you for this! I will definately check it out. Is it possible to have act configured by configuration file this way so users can just run act without the -P option?

[cplee]

I had considered that @prologic ...would you rather the act configuration be in ~/.config/act/ or in the repo as a file like .actconfig?

[prologic]

It might actually make sense to have the configuration per-repo. It would be feasible for example to go create custom images to replace/alias the ones in the workflow files that you build/publish yourself. Does that make sense?

[cplee]

It does...would you be interested in writing up your thoughts on how it'd work in a separate issue?

[eine]

I understand; but as act is a tool to run GHA workflows locally it would not be a very useful tools if users would have to go and manually muck around with the same workflow just to ge tthem to work locally.

I believe that this tool is usable because it is a semantic analyser that produces executable snippets from a non-executable YAML file. It is specially valuable because it allows not to learn each detail/syntax of each procedure.

I don't consider being aware of the dependencies to build/run a tool to be mucking around. Moreover, I'd consider quite irresponsible to do otherwise.

Of course, I don't mean that docker images should not be available. Just that an image of 18GB is absolutely opposite to the purpose of containers. Some years ago, it was not possible by default.

Yes I agree with you that piling everything into an image you need is going to be problematic (mostly size issues) but still. Also runniner DinD isn't altogether hard it just requires some special treatment.

The main point is that it is difficult to justify that handling problems because of image sizes or using DinD is easier than just selecting a good idiomatic solution.

It would be feasible for example to go create custom images to replace/alias the ones in the workflow files that you build/publish yourself. Does that make sense?

This is a different feature, isn't it? I thought we were talking about executing the workflow/steps in a container. Not about doing substitutions in the YAML for docker commands invoked in it.

[prologic]

It does...would you be interested in writing up your thoughts on how it'd work in a separate issue?

Yes I will do this :)

[prologic]

@eine I'm not disagreeing with you; you are right of course; I'm just merely saying that in order for act to a useful tool it should be easy to use. It's mostly about user expectations and what we want the tool to be capable of doing. Right now if I run act in a repo with existing GHA workflows, they fail. That's not great because now I have to go a) figure out why b) make modifications to something c) or something else.

I will file a separate issue and I fully intend to help contribute to this project / tool because we need this ourselves!

[eine]

I'm just merely saying that in order for act to a useful tool it should be easy to use.

I do think that act is useful and it is easy to use. The fact that it does not cover all corner cases doesn't make it less useful in many.

It's mostly about user expectations and what we want the tool to be capable of doing.

I think that this tool should not fulfill the expectations of all users. Developers should have their own expectations and make them clear and easy to find, so that users can know. However, developers should evaluate proposals and be careful no to widen the scope too much.

Right now if I run act in a repo with existing GHA workflows, they fail.

I'm afraid the opposite is also true. Any user with an Ubuntu 18 host which has the dependencies for the project whose workflow file is going to execute, will find that act can run them flawlessly without requiring a base container, and it can run containers in the steps without any issue. This is the same for Windows and macOs users.

The explanation in https://github.com/nektos/act#how-does-it-work sounds quite close to this environment. There is no explicit mention to running the workflow itself in a base container.

This is to say that, while I understand your point of view and I think that it is really useful for many users, I believe your expectations are currently out of scope. Of course, I think it'd be great to discuss it, and to extend/clarify act's scope. Nonetheless, creating a wrapper around act to implement the management of the (huge) base images and dind would also be a valid (and easier to understand) solution.

[prologic]

@eine I really think you're taking my point of view out of context and blowing it up a bit too much.

act is useful now. It needs to be as simple as a developer just running act in a pre-existing repo for me to be able to get others on my team to use this. That's it.

act professes to be able to run Github Workflow Actions locally. That's its mission in life; everything we can do to make this easier for developers and users alike (although I was only talking about developers here) should be discussed and potentially in-scope.

I filed #89 to discuss the first I hope of many proposals to enhance act(s) functionality.

[prologic]

@Djiit / @nektos-ci I think we can close this issue. Most of what's there for v2 / YAML support seems to work okay :)

[Kreyren]

Are actions v2 ready for production already? I would like to implement it in gitpod https://gitpod.io/

[eine]

@prologic, I'm sorry if my wording was misunderstood. I'm actually a Windows user and I'd love to be able to execute any ubuntu-latest job as is, without even reading the workflow I find in a repo. It's just that I don't understand that as being "local execution". Once a container (i.e. an intermediate layer) is required, it is implementation-dependent to use a VM or a remote instead. I'd like to discuss this so that the enhancement is not constrained by the requirements on your team.

[timfallmk]

I don't want to hijack the conversation, and I don't know if this belongs here or in a separate issue, so please yell at me if necessary 😄 .

When I attempt to run a workflow with a setup-go action in it, the necessary config files are not found:

Workflow files: https://github.com/ksync/ksync/blob/master/.github/workflows/test.yml

git remote -v
origin	https://github.com/ksync/ksync (fetch)
origin	https://github.com/ksync/ksync (push)

act -P ubuntu-18.04=nektos/act-environments-ubuntu:18.04
[Tests/Run tests] ⭐  Run actions/setup-go@v1
[Tests/Run tests]   ☁  git clone 'https://github.com/actions/setup-go' # ref=v1
[Tests/Run tests]   🐳  docker pull node:12-alpine
[Tests/Run tests]   🐳  docker run image=node:12-alpine entrypoint=[] cmd=["node" "/github/home//setup-go454447292/lib/setup-go.js"]
| internal/modules/cjs/loader.js:985
|   throw err;
|   ^
| 
| Error: Cannot find module '/github/home/setup-go454447292/lib/setup-go.js'
|     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:982:15)
|     at Function.Module._load (internal/modules/cjs/loader.js:864:27)
|     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:74:12)
|     at internal/main/run_main_module.js:18:47 {
|   code: 'MODULE_NOT_FOUND',
|   requireStack: []
| }
[Tests/Run tests]   ❌  Failure - actions/setup-go@v1
Error: exit with `FAILURE`: 1

I've also tried with the default environment and got the same result. I'm not sure if I'm understanding this right, but it seems like a default config file might need to be copied into the node image (in this case). Is this something that would be in the default runner?

act --version
act version 0.2.1

[prologic]

@prologic, I'm sorry if my wording was misunderstood. I'm actually a Windows user and I'd love to be able to execute any ubuntu-latest job as is, without even reading the workflow I find in a repo. It's just that I don't understand that as being "local execution". Once a container (i.e. an intermediate layer) is required, it is implementation-dependent to use a VM or a remote instead. I'd like to discuss this so that the enhancement is not constrained by the requirements on your team.

That's okay :)

So when we say "local" I mean this:

As a user I want to be able to run my Github Workflows in both Github and outside of Github locally on my development machine.

That's it. This is how I came across act in the first place.

What runners / containers / vms / whatever is used to run the workflow/action as stated in the workflow whether or not act replaces them with something else under the hood is irrelevant.

I think per-repo configuration will make this easier because I can write my Github Workflows once, write a .actrc configuration and have my workflows be capable of being run locally and on Github (via their Azure pipelines infra)

[cyberhck]

I was actually here to see if we can run GHA inside teamcity runners as we don't yet have GHA in enterprise 😄

[SvenDowideit]

I'm possibly weird as usual - I'd like to consider act as a replacement for make and a way to develop and test my CI/CD automations before committing them to the repository and as a way to debug and fix issues contemplate how it differs from the science workflow work we're also doing and to that end, I also prefer it to magically work, but then to allow me to customise very much looking forward to having time to play !

…

On Fri, Feb 21, 2020 at 2:45 PM Nishchal Gautam ***@***.***> wrote: I was actually here to see if we can run GHA inside teamcity runners as we don't yet have GHA in enterprise 😄 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#74?email_source=notifications&email_token=AAAG6TEOGBAT645E3QC2BDLRD5MABA5CNFSM4IKRT6U2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMRP5NY#issuecomment-589495991>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAG6TH4V4UO7GMDEBGZUBLRD5MABANCNFSM4IKRT6UQ> .

[cplee]

Closing this out now that we have a stable release v0.2.3.

Thanks all for the support! 🏁