Special file type for `locals` or `overrides`

[djryanj]

Terraform Version

1.13.0

Use Cases

Consider the following:

• Your Terraform configuration has multiple .tfvars files, each one dedicated to a specific environment
• Your Terraform configuration requires locals, because some resource configurations need dynamic inputs based on other variables or references to data sources, etc.
• Your Terraform configuration cannot be generalized enough such that global locals and environment-specific tfvars are sufficient without significant use of conditionals or other functions

For example, say I'm creating an Application Gateway in Azure, and I have listeners that are different in dev and prod and require me to reference different web application firewall policy ID's on a per-listener basis; my listeners are also not the same in dev and prod. Take the following (very simplified) configuration:

main.tf:

resource "azurerm_application_gateway" "network" {
  name                = "example-appgateway"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  firewall_policy_id  = azurerm_web_application_firewall_policy.default.id

  dynamic "http_listener" {
    for_each = local.http_listeners

    content {
      frontend_ip_configuration_name = coalesce(http_listener.value.frontend_ip_configuration_name, local.frontend_ip_configuration_name, local.frontend_ip_configuration_private_name)
      frontend_port_name   = http_listener.value.frontend_port_name
      name                 = http_listener.value.name
      protocol             = http_listener.value.ssl_certificate_name == null ? "Http" : "Https"
      firewall_policy_id   = http_listener.value.firewall_policy_id
    }
  }
}

dev.tfvars:

http_listeners = {
  listener_1 = {
    frontend_ip_configuration_name = "feip-1"
    frontend_port_name             = "fep-80"
    name                           = "listener1"
  }
}

Desired final configuration:

http_listeners = {
  listener_1 = {
    frontend_ip_configuration_name = "feip-1"
    frontend_port_name             = "fep-80"
    name                           = "listener1"
  }
}

prod.tfvars:

http_listeners = {
  listener_3 = {
    frontend_ip_configuration_name = "feip-1"
    frontend_port_name             = "fep-80"
    name                           = "listener3"
  }
  listener_4 = {
    frontend_ip_configuration_name = "feip-1"
    frontend_port_name             = "fep-80"
    name                           = "listener3"
  }
}

If I use locals, like this:

locals.tf:

locals {
  http_listeners = {
    for k, v in var.http_listeners : k => {
      name                           = v.name
      frontend_port_name             = v.frontend_port_name
      frontend_ip_configuration_name = v.frontend_ip_configuration_name
      firewall_policy_id             = v.firewall_policy_id
    }
  }
}

And my desired final configuration for local.http_listeners would then be:

dev:

http_listeners = {
  listener_1 = {
    frontend_ip_configuration_name = "feip-1"
    frontend_port_name             = "fep-80"
    name                           = "listener1"
  }
}

prod:

http_listeners = {
  listener_3 = {
    frontend_ip_configuration_name = "feip-1"
    frontend_port_name             = "fep-80"
    name                           = "listener3"
    firewall_policy_id             = azurerm_web_application_firewall_policy.listener_3.id
  }
  listener_4 = {
    frontend_ip_configuration_name = "feip-1"
    frontend_port_name             = "fep-80"
    name                           = "listener3"
    firewall_policy_id             = azurerm_web_application_firewall_policy.listener_4.id
  }
}

As you can see I have no easy way of defining the var.http_listeners input variable in a .tfvars file in such a way that I could correctly reference any of the azurerm_web_application_firewall_policy resources and have the final output be unique for dev and prod or whatever else I decide to create in the future.

Attempted Solutions

A few solutions I've considered:

• Hard code ID's into the variables, which doesn't work if this module is also creating those resources plus it risks encoding information into the code that may be considered secret (like subscription IDs)
• Really complicated conditionals and other things. This is hard to read, and doesn't scale well partly due to the lack of switch statements (hey, there's another idea for a feature) but even then, the scale is very limited
• Use a module for this resource, and call it multiple times, but using a conditional where it only creates that configuration if e.g., environment = dev. This violates DRY, and forces configuration out of variables and into the code directly.
• Use _override.tf files, which would be reasonable in an environment where pipelines are in use as they could be created/dropped in per environment, but causes difficulties when run locally
• Requesting that .tfvars files be able to reference resources created in the configuration, which seems like the sort of thing that would lead to impossible race conditions, cycle errors, and other nightmares

Proposal

Currently, Terraform allows us to specify which tfvars file(s) to use by passing a -var-file CLI parameter (to some/most commands). The proposal here is to either:

1. Extend that same behaviour into locals by defining a new file type, .tflocals, into which locals are placed, and a CLI parameter, -local-file, which works the same as -var-file.
   • Importantly, this should not break regular locals usage; just like with tfvars files, Terraform should ignore their existence unless they are specifically passed into the CLI at runtime except e.g., terraform.tfvars and similarly terraform.tflocals or perhaps locals.tflocals.
2. Alternatively, do the same thing with a new .tfoverride file type, which works just like _override.tf files except again, is only processed by Terraform when explicitly passed in via an override-file CLI parameter. There should be no change to regular _override.tf file processing.

It's relevant to call out that I stated above when considering multiple modules called conditionally that

This violates DRY, and forces configuration out of variables and into the code directly

which arguably could apply here too if one considers locals to be part of the code rather than part of the variable definitions for the code. Personally I take the latter view, which is why I think creating a .tflocals file type for locals makes a certain amount of sense.

Thank you for considering this.

References

No response

[apparentlymart]

When I've been in this sort of situation before I've typically solved it by having values in one input variable refer to keys from a map in another input variable. Something like this, perhaps:

variable "http_listeners" {
  type = list(object({
    frontend_ip_configuration_name = optional(string)
    frontend_port_name             = string
    name                           = string
    firewall_policy_key            = optional(string)
  }))
  
  # Optional: custom validation rule checking that every
  # firewall_policy_key value corresponds to one of the
  # keys in var.firewall_policies, instead of just letting
  # it fail in the `firewall_policy_id` expression itself.
}

variable "firewall_policies" {
  type = map(object({
    # ...whatever settings you want to allow callers to customize for these...
  }))
}

resource "azurerm_application_gateway" "network" {
  name                = "example-appgateway"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  firewall_policy_id  = azurerm_web_application_firewall_policy.default.id

  dynamic "http_listener" {
    for_each = var.http_listeners

    content {
      frontend_ip_configuration_name = coalesce(http_listener.value.frontend_ip_configuration_name, local.frontend_ip_configuration_name, local.frontend_ip_configuration_private_name)
      frontend_port_name   = http_listener.value.frontend_port_name
      name                 = http_listener.value.name
      protocol             = http_listener.value.ssl_certificate_name == null ? "Http" : "Https"
      firewall_policy_id   = (
        http_listener.value.firewall_policy_id != null ?
        azurerm_web_application_firewall_policy.example[http_listener.value.firewall_policy_id] :
        null
      )
    }
  }
}

resource "azurerm_web_application_firewall_policy" "example" {
  for_each = var.firewall_policies
  
  # ...settings based on attributes of each.value...
}

This means that every call to this module can have its own set of HTTP listeners and its own set of firewall policies and its own set of relationships between them, without the caller needing to know about internal details like the resource name of the azurerm_application_gateway resource.

I'm sharing this mainly because I'm curious where this sits in relation to the "Attempted Solutions" you listed. Is what I've described here something like what you intended one of those items to mean? e.g. is that definition of firewall_policy_id an example of what you meant by "really complicated conditionals and other things"?

I'm asking only out of curiousity about how far away an approach like the above is from acceptable, and whether there's something smaller that could change to make this sort of thing easier to model with normal expressions. I'm not meaning to suggest that my example above ought to already be an acceptable answer for your situation.

[jbardin]

Thanks for filing the issue @djryanj.
I'm also curious what about your experience is with the above approach demonstrated by @apparentlymart, which more aligns with the current design. There is a bit of a contradiction in the request, if one were to consider the local values as only part of the variable definitions, then they would be reduced to only referencing each other or input variables, narrowing their effective scope. A hurdle there is that this synthetic scope would need to be constrained in a way that Terraform doesn't currently recognize; maybe by a union of variables declarations and input file scope? If you want to retain the ability to reference other object types, then locals are going to inherently remain part of the overall configuration.

I don't think changing the definition of locals is an optimum solution, so maybe that means we're looking at something more akin to overrides here. I can see some benefit to being able to swap out the composition of variables along with the variables themselves, but finding a clear and maintainable way to do that might be challenging.

[djryanj]

@apparentlymart - thanks for looking at my suggestion.

Actually, I forget to mention in the "Attempted Solutions" section exactly what you're talking about. I did try that, but that approach has limitations:

1. If you ever need to reference an object from somewhere that is not common to the rest of them, this breaks. For example, if I need to do this:

    resource "azurerm_web_application_firewall_policy" "place_1" {
      for_each = var.firewall_policies
      provider = azurerm.subscription1
      # ...settings based on attributes of each.value...
    }
   
    resource "azurerm_web_application_firewall_policy" "place_2" {
      for_each = var.firewall_policies
      provider = azurerm.subscription2
      # ...settings based on attributes of each.value...
    }

   Then I am out of luck. Some kind of custom object could be created to handle every other parameter of a common azurerm_web_application_firewall_policy except that one (though it gets unwieldy). While I realize this is somewhat specific to the azurerm provider, I suspect that it's not difficult to find other instances where this would also be a problem.
2. Yes, this sort of fits into "really complicated conditionals and other things". While I realize that it's up to the users to be able to work within the language, using (from your example) firewall_policy_key in the variable definition means that they have to get that key exactly right to whatever is used as the key in the for_each azurerm_web_application_firewall_policy definition. terraform validate will find this error, but it's a frustrating and non-obvious coding requirement. The whole thing feels so unwieldy to write and maintain.

Locals solves this nicely, by making the configuration a lot easier to read.

@jbardin I agree that perhaps a different method is required. I hesitated to phrase it the original way I thought of it, which was to allow variables to reference attributes from other parts of the configuration which is what I think you mean by:

I can see some benefit to being able to swap out the composition of variables along with the variables themselves,

However based on what I could find searching the issues that seems to have been an explicit design goal of locals from the beginning; it just doesn't quite handle this edge case, and I suspect I'm not the first to request exactly that: reference stuff in the configuration from variables directly.

The more I think about it the more I think you're correct in that extending overrides might be the best way forwards; could you put locals in overrides?

What about a new, middle type, something like scoped_locals within the new tflocals file type, that is only available to use via CLI parameter?

I also recognize that this might be such a weird edge case that it's not worth exploring. It might even be a case of a bad "code smell" which is likely indicative of poor design or using anti-patterns when deploying the resources to have to do this. However, despite that I can see times when this might be helpful.

I appreciate the discussion here.