-
Notifications
You must be signed in to change notification settings - Fork 614
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ext/dynblock: Preserve marks from for_each expression into result #679
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously if the for_each expression was marked then expansion would fail because marked expressions are never directly iterable. Now instead we'll allow marked for_each and preserve the marks into the values produced by the resulting block as much as we can. This runs into the classic problem that HCL blocks are not values themselves and so cannot carry marks directly, but we can at least make sure that the values of any leaf arguments end up marked.
Similar to the previously-added UnknownBody, the new optional interface MarkedBody allows hcl.Body implementations to suggest a set of marks that ought to be applied to any value that's generated to represent the content of that body. The dynblock extension then uses this to get hcldec to mark the whole object representing any block that was generated by a dynamic block whose for_each was marked, for a better representation of the fact that a block's existence was decided based on a marked value.
jbardin
approved these changes
May 9, 2024
wata727
added a commit
to terraform-linters/tflint
that referenced
this pull request
Jan 4, 2025
Follow up of hashicorp/hcl#679 Previously, for_each in dynamic blocks did not allow marked values such as sensitive. However, hashicorp/hcl#679 now supports this by propagating the marks to expanded children. The reason behind this is to add a new mark called "ephemeral", so we'll pull the changes to support Terraform 1.10. Note that tfhcl's dynamic block support has incomplete mark propagation since marked values resolve to unknown values. This is because in the past the marked values could not be sent over the wire protocol, and may be fixed in the near future.
wata727
added a commit
to terraform-linters/tflint
that referenced
this pull request
Jan 4, 2025
Follow up of hashicorp/hcl#679 Previously, for_each in dynamic blocks did not allow marked values such as sensitive. However, hashicorp/hcl#679 now supports this by propagating the marks to expanded children. The reason behind this is to add a new mark called "ephemeral", so we'll pull the changes to support Terraform 1.10. Note that tfhcl's dynamic block support has incomplete mark propagation since marked values resolve to unknown values. This is because in the past the marked values could not be sent over the wire protocol, and may be fixed in the near future.
wata727
added a commit
to terraform-linters/tflint
that referenced
this pull request
Jan 11, 2025
Follow up of hashicorp/hcl#679 Previously, for_each in dynamic blocks did not allow marked values such as sensitive. However, hashicorp/hcl#679 now supports this by propagating the marks to expanded children. The reason behind this is to add a new mark called "ephemeral", so we'll pull the changes to support Terraform 1.10. Note that tfhcl's dynamic block support has incomplete mark propagation since marked values resolve to unknown values. This is because in the past the marked values could not be sent over the wire protocol, and may be fixed in the near future.
wata727
added a commit
to terraform-linters/tflint
that referenced
this pull request
Jan 11, 2025
Follow up of hashicorp/hcl#679 Previously, for_each in dynamic blocks did not allow marked values such as sensitive. However, hashicorp/hcl#679 now supports this by propagating the marks to expanded children. The reason behind this is to add a new mark called "ephemeral", so we'll pull the changes to support Terraform 1.10. Note that tfhcl's dynamic block support has incomplete mark propagation since marked values resolve to unknown values. This is because in the past the marked values could not be sent over the wire protocol, and may be fixed in the near future.
wata727
added a commit
to terraform-linters/tflint
that referenced
this pull request
Jan 11, 2025
* Bump tflint-plugin-sdk to v0.22.0 * Add `ephemeralasnull` function Follow up of hashicorp/terraform#35363 Follow up of hashicorp/terraform#35652 * Introduce ephemeral input variables Follow up of hashicorp/terraform#35273 Follow up of hashicorp/terraform#35985 Terraform throws an error if you use ephemeral values for the count meta-argument, but TFLint does not. This is because the reason for throwing an error is that plan cannot have ephemeral values, which is not an issue in the context of static analysis. In the future, we can throw an error if we need to match this behavior. * Functions that allow marks must also deal with unknown values Follow up of hashicorp/terraform#35985 * Add `terraform.applying` symbol Follow up of hashicorp/terraform@7c928fc * Introduce ephemeral resources Follow up of hashicorp/terraform#35727 Follow up of hashicorp/terraform#35728 Ephemeral resource addresses are like resources in that they always resolve to unknown values, but they differ in that they are marked as ephemeral, which can have a subtle effect on the return value of the ephemeralasnull function. * `issensitive` must return unknown for unknown args without `sensitive` Follow up of hashicorp/terraform#36012 * Fix `templatefile` function for unknown/marked values Follow up of hashicorp/terraform#36118 Follow up of hashicorp/terraform#36127 * Update collections to use for-range method Follow up of hashicorp/terraform#35818 * Include context when variable default has nested problem Follow up of hashicorp/terraform#35465 * Allow marked values in dynamic block `for_each` Follow up of hashicorp/hcl#679 Previously, for_each in dynamic blocks did not allow marked values such as sensitive. However, hashicorp/hcl#679 now supports this by propagating the marks to expanded children. The reason behind this is to add a new mark called "ephemeral", so we'll pull the changes to support Terraform 1.10. Note that tfhcl's dynamic block support has incomplete mark propagation since marked values resolve to unknown values. This is because in the past the marked values could not be sent over the wire protocol, and may be fixed in the near future. * Do not return ephemeral values to unsupported plugins Because ephemeral values are likely to contain secrets, return ErrSensitive for plugins that do not support it to prevent unintended disclosure. * Add E2E tests for ephemeral values and marked dynamic blocks * Update Terraform compatibility guide
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously if the
for_each
expression was marked then expansion would fail because marked expressions are never directly iterable. (The failure also had a very confusing error message, because it wasn't directly testing whether the value was marked and was just assuming that any non-iterable value is non-iterable because it's of the wrong type.)Now instead we'll allow marked for_each and preserve the marks into the values produced by the resulting block as much as we can.
This runs into the classic problem that HCL blocks are not values themselves and so cannot carry marks directly, but we can at least make sure that the values of any leaf arguments end up marked. We also ensure that applications that use
hcldec
(which includes Terraform) will see marks on the objects that represent the blocks themselves, thereby improving the marking accuracy.This also fixes a nearby bug that I found along the way:
dynblock
would previously panic if the value specified for a block label was marked. It will now return an error message, although the error message is not very good quality because HCL doesn't understand what any marks represent. If this becomes a problem in practice then perhaps in future we'll allow applications to specify their own label value validator functions so that they can customize the error messages, but dynamic labels are so rarely used right now that it doesn't seem worth that complexity.Terraform-specific note: Terraform currently only uses its "sensitive" mark and configures
dynblock
to reject anyfor_each
value that carries that mark, so this change effectively does nothing for today's Terraform because the new codepaths are unreachable anyway.I've implemented this for hashicorp/terraform#35078, because that introduces a new mark called "ephemeral" which represents that the value isn't required to be consistent between plan and apply and is never persisted in the state. It isn't appropriate to reject values with that mark in dynamic block
for_each
, because we want to be able to use ephemeral values in provider configuration blocks and several heavily-used providers use nested blocks in their configuration schemas that might need to vary based on an ephemeral value.(For example, consider an ephemeral input value for providing a JWT to use for AWS's "AssumeRoleWithWebIdentity". A JWT is a good example of something that's likely to vary between plan and apply and so would be declared as ephemeral, but it would be reasonable to use the nullness of that value to choose dynamically whether or not to include the
assume_role_with_web_identity
block, which requires that we allow thefor_each
result to be derived from the ephemeral value and therefore be ephemeral itself.)