Procfile comparison

[techtonik]

How recipes compare to procfiles https://honcho.readthedocs.io/en/latest/using_procfiles.html?

[riannucci]

Recipes fill a similar niche ("run a bunch of stuff!"), but have some additional features:

• They're cross-platform (meaning: they abstract things like running subprocesses, handling IO, paths, etc. with simple APIs where you can express the intent of the job with one API and have things turn out the right way on windows v. posix)
• They allow full workflow descriptions, meaning that you can change the behavior of the job based on the intermediate results of the steps in the build (e.g. process step output, handle failure conditions, etc.) using normal python constructs (loops, conditionals, exceptions).
• They integrate with continuous integration UI systems (today: buildbot annotations, tomorrow: some standalone UI apps) to show steps, statuses, summary text, links, etc.
• They allow builds to be expressed as a bunch of json-typed inputs (properties), which run and then return some json-typed data; this is going to allow us to build our CI systems around more of a "functional" approach. This approach will let us achieve better caching and composability when running many tasks in parallel on e.g. swarming.
• Since they're more than a simple list of steps, they allow cross-platform simulation (e.g. "if I run this recipe with these inputs on a windows machine, and the 'foo' step fails, what would happen?"), with code-coverage targets (because it's python, this catches a bunch of syntax errors / type-related bugs).
• At their root, they're really just python; you get the full¹ python toolset.

They're also designed to be federated across multiple repos; one repo (say, 'depot_tools') can provide recipe modules (like 'gclient') which are then used by other repos (say 'chromium'). The engine also has enough intelligence to know how to roll upstream changes in functionality downstream to other repos to help us keep the entire web of recipes up-to-date. This system also has full-pinning information, which means that if you run a historical build (e.g. from an old release branch, or from a bisect), you can have exactly the version of recipes that was used at that point in time.

If the question is then "that sounds complex! I want something simpler!" I would say, "That sounds good! Use something simpler! 😀". The luci projects that we're working on are very consciously not meant to assume that you're using recipes (and indeed, we're probably going to be supporting simple bash scripts and bare executables like gtest binaries). Recipes are just a tool that we've found to be really useful in codifying the way that our builds work, and making those encoded build processes maintainable and hackable by a large group of people.

Hopefully that answers your question?

---

1: though we remove portions of python that don't have cross-platform apis like 'os' and 'subprocess'; we provide replacement cross-platform implementations for those via recipe modules.

[techtonik]

This is very useful info. I think it has big potential if turned into slides. And for slides it will be interesting to see real world stories encountered through all this time..

This thing - https://github.com/luci/recipes-py/blob/master/doc/user_guide.md - is useful. And luci.github.io may be updated to point to it.

I can also see benefit in choosing this API - Python code that is independent on Python behavior and its libraries. It is a cool initiative. Can we also start doing language independent library of behavior - that will be sharing tests, will have public domain license (to be shared with other interpreters) and crediting? This may enable faster evolution of Python stdliib and other languages. Because right now people are reinventing things for their language.