Failed to parse a heredoc expression as it is

[wata727]

HCL Template

foo = <<EOF
{}
EOF

Expected behavior

After parsing the above file as a config, if you take bytes from the range of the "foo" expression (<<EOF\n{}\nEOF) and parse it again as an expression based on its start position and the bytes, you expect to get the expression correctly.

Actual behavior

Failed to parse bytes as an expression. It says "Unterminated template string; No closing marker was found for the string."

Steps to reproduce

Run the following code:

package main

import (
	"fmt"
	"io/ioutil"

	"github.com/hashicorp/hcl/v2"
	"github.com/hashicorp/hcl/v2/hclsyntax"
)

func main() {
	src, err := ioutil.ReadFile("config.hcl")
	if err != nil {
		panic(err)
	}
	file, diags := hclsyntax.ParseConfig(src, "config.hcl", hcl.InitialPos)
	if diags.HasErrors() {
		panic(diags)
	}
	attributes, diags := file.Body.JustAttributes()
	if diags.HasErrors() {
		panic(diags)
	}
	expr := attributes["foo"].Expr

	exprSrc := expr.Range().SliceBytes(src)
	fmt.Println("Source:")
	fmt.Println(string(exprSrc))
	fmt.Print("\n")

	parsedExpr, diags := hclsyntax.ParseExpression(exprSrc, "config.hcl", expr.Range().Start)
	if diags.HasErrors() {
		panic(diags)
	}

	fmt.Printf("original=%#v\n", expr)
	fmt.Printf("parsed=%#v\n", parsedExpr)
}

References

• Request: Make json structs exported #332
  • We are using ParseExpression to serialize hcl.Expression and communicate over RPC between processes.
• Plugins fail when evaluating an attribute with a heredoc value terraform-linters/tflint#1029
  • An issue reported in our repository due to this issue.
• https://github.com/terraform-linters/tflint/blob/v0.23.0/plugin/server.go#L31-L32
  • Send expression to plugin processes.
• https://github.com/terraform-linters/tflint-plugin-sdk/blob/v0.7.0/tflint/client/decode.go#L67
  • Receive expression bytes and parse it.

[wata727]

But it's understandable that it fails to parse this. When you retrieve an expression based on the range, the last newline is always lost. This is invalid as heredoc.

<<EOF
{}
EOF
\ No newline at end of file

In addition, I noticed that ParseExpression always parses the bytes in "ignore newlines" mode.

hcl/hclsyntax/public.go

Lines 46 to 48 in 56e58b6

	// Bare expressions are always parsed in "ignore newlines" mode, as if
	// they were wrapped in parentheses.
	parser.PushIncludeNewlines(false)

The problem I want to solve with this issue is that I want to know if I should fix the ParseExpression for this problem or if it's the user's responsibility to always add a newline at the end of the passed bytes.