Jsonnet Function Family for HCL

[mitchellh]

This introduces two new HCL functions jsonnetfile and jsonnetdir which behave similarly to our template functions: they process a file or directory, respectively, and convert Jsonnet-formatted files to JSON. They return a path to the processed file or directory, which is copied rather than updated in-place (behaving the same as our templating functions).

These functions can be used anywhere, but are created particularly to pair with #1357 since Jsonnet is a popular way to manage K8S configurations in certain environments.

Features

• Can process a single jsonnet file or a directory of files
• For directory processing, only files ending in .jsonnet are processed
• For both directory and single file processing, files in the result have a .json suffix added
• Full Jsonnet parameterization is supported -- users can specify custom variables.
• Unlike templating, HCL variables don't magically exist here. Users are encouraged to compose this with our templating functions to access that, or manually pass through the variables. We have to do this since type conversions are a mess.

Example

Building on #1357

app "example-nodejs" {
  # ... 

  deploy {
    use "kubernetes-apply" {
      // Note this complex case where we compose template and jsonnet, this lets us first 
      // template our ".jsonnet" files and then process them. The user could also just use
      // the `ext_vars` approach or something.
      path = jsonnetdir(templatedir("${path.app}/k8s"), {
        ext_vars = { 
          environment = "production"
        }
      })

      prune_label = "app=myapp"
    }
  }
}

[mitchellh]

I deleted these tests cause they were overly dependent on an exact file structure and the functionality was already tested by other cases. The "overly dependent" property was resulting in failures due to test files introduced for the jsonnet tests.

[evanphx]

Just one question about what deletes the tempdir. It doesn't look like anything does, which means users will accidentally create a ton of them?

[evanphx]

What deletes the tempdir?

[mitchellh]

Nobody, this is behaving the same way as template* functions at the moment where we dangle the temp dirs. The hope is it goes into a system temp dir that gets cleared at some point currently (mine does on both macOS and Linux, at least). I think in the future we can try some process-wide cleanup if you wanted but I think that should be a separate PR.

[evanphx]

Oh zoinks. I want to fix that then. Go ahead with this and I’ll fix the tempdir issue in the 0.4 cycle.

On Mon, Apr 19, 2021 at 6:51 PM Mitchell Hashimoto ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In internal/config/funcs/jsonnet.go <#1360 (comment)>: > +// stored in a temporary directory that is returned; the original files are +// untouched. +var JsonnetDirFunc = function.New(&function.Spec{ + Params: []function.Parameter{ + { + Name: "dir", + Type: cty.String, + }, + { + Name: "options", + Type: cty.DynamicPseudoType, + }, + }, + Type: function.StaticReturnType(cty.String), + Impl: func(args []cty.Value, retType cty.Type) (cty.Value, error) { + td, err := ioutil.TempDir("", "waypoint") Nobody, this is behaving the same way as template* functions at the moment where we dangle the temp dirs. The hope is it goes into a system temp dir that gets cleared at some point currently (mine does on both macOS and Linux, at least). I think in the future we can try some process-wide cleanup if you wanted but I think that should be a separate PR. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#1360 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAAB2DAQNELHKIOMMFSYDTJTM3HANCNFSM43GXB5HA> .

-- Evan Phoenix // ***@***.***

[mitchellh]

Sounds good. My recommendation would be to maintain a global var of cleanup things and call a package-level cleanup at some point. We need to keep these directories around longer than expected because plugins may use them all the way until Waypoint exits.

[evanphx]

Yup! I was also thinking about making a tempdir on start and setting it to TMPDIR so that any plugins that use tempdir will be in the one we cleanup on exit.

On Mon, Apr 19, 2021 at 7:06 PM Mitchell Hashimoto ***@***.***> wrote: Sounds good. My recommendation would be to maintain a global var of cleanup things and call a package-level cleanup at some point. We need to keep these directories around longer than expected because plugins may use them all the way until Waypoint exits. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#1360 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAAB5H5BA4P2PBVK4U4PTTJTORVANCNFSM43GXB5HA> .

-- Evan Phoenix // ***@***.***