plugin: gRPC-based new plugin system

[wata727]

This PR introduces a new plugin system based on gRPC. The new plugin system is a complete replacement of the existing plugin system and all users will need to rebuild rulesets to comply with this API. This goal is to resolve some issues with the traditional net/rpc based plugin system.

The first issue is to mitigate the impact of Terraform v1.0 package internalization. This SDK provided a way to access data compatible with structures defined in Terraform's internal packages such as WalkResources. For this reason, it had to maintain its own compatible structure, and TFLint had to decode Resources, Variables, etc. from HCL files as before.

However, the schema to be inspected is known to the rule side (for instance, the aws_instance_invalid_type rule knows that the instance_type attribute exists inside the resource block), so TFLint does not need to decode in the early stage. This eliminates the need for LoadConfigFile and the compatible structures. As a side benefit, it makes more robust to the Terraform version because it eliminates the need to check for schemas that are not needed for inspection. See also terraform-linters/tflint#937 (comment) for more information on this issue.

The second issue is to send hcl.Body over a wire protocol. Previous APIs sent hcl.Body to plugins to use structures like configs.Resource in the SDK. However, hcl.Body is an interface, and interface cannot be encoded by gob as it is, so the corresponding range and bytes are sent instead.

The problem with this way is that there is no correct way to extract the corresponding range from hcl.Body. TFLint extracts the range by dirty hacks, but as a result, there are some bugs. See also terraform-linters/tflint-ruleset-google#129, #97
It also includes the fatal problem of not being able to send the merged config properly. See also terraform-linters/tflint#1019, #89

The last issue is that we get a runtime error when sending a value that cannot be encoded by gob. The encoding/gob is easy to use with net/rpc and has the advantage of being able to get started quickly as it can implicitly encode/decode many structures and primitives. On the other hand, using a value that includes an interface without gob.Register causes a runtime error, which makes requires careful testing. See also terraform-linters/tflint-ruleset-aws#48

To resolve all these issues, the new plugin system makes the following changes:

Introduces new Runner API GetResourceContent instead of WalkResource*

Previously, the code to access a particular attribute of a resource block was:

runner.WalkResourceAttributes("aws_instance", "instance_type", func (attribute *hcl.Attribute) error {
    attr.Expr // => An expression of `instance_type`
})

This changes as follows:

resources, _ = runner.GetResourceContent("aws_instance", &hclext.BodySchema{
    Attributes: []hclext.AttributeSchema{{Name: "instance_type"}},
}, nil)

for _, resource := range resources.Blocks {
    resource.Body.Attributes["instance_type"].Expr // => An expression of `instance_type`
}

As you can see, the API will change to a schema-based that allows you to retrieve arbitrary attributes and blocks at once. This schema is inspired by hcl.BodySchema and Content and can be used in much the same way. By changing to a schema-based design, TFLint only needs to have hcl.File and will be able to send merged configs.

The hclext is an extension package of hashicorp/hcl introduced to allow you to declare the schema of nested blocks. For example, the create_before_destroy attribute in a lifecycle block can be extracted as follows:

resources, _ := runner.GetResourceContent("aws_instance", &hclext.BodySchema{
    Blocks: []hcl.BlockSchema{
        {
            Type: "lifecycle",
            Body: &hclext.BodySchema{Attributes: []hclext.AttributeSchema{{Name: "create_before_destroy"}}},
        },
    },
}, nil)

for _, resource := range resources.Blocks {
    for _, lifecycle := range resource.Body.Blocks {
        lifecycle.Body.Attributes["create_before_destroy"].Expr // => An expression of `create_before_destroy`
    }
}

Based on this, the WalkResourceAttributes, WalkResourceBlocks, and WalkResources APIs have been removed. Use the GetResourceContent API.

GetResourceContent is a shorthand for GetModuleContent. You can use this API if you want to retrieve structures other than resources.

providers, _ := runner.GetModuleContent(&hclext.BodySchema{
    Blocks: []hclext.BlockSchema{
        {
            Type: "provider",
            LabelNames: []string{"name"},
            Body: &hclext.BodySchema{
                Attributes: []hclext.AttributeSchema{{Name: "region"}},
            },
        },
    },
}, nil)

for _, provider := range providers.Blocks {
    provider.Body.Attributes["region"].Expr // => An expression of `region`
}

Based on this, the WalkModuleCalls, Backend, and Config APIs have been removed. Use the GetModuleContent API.

These behaviors can be changed by options. For example, if you want to access the root module from a child module during inspection, you can change the target by setting ModuleCtx as follows. The default is your own module.

runner.GetModuleContent(
    &hclext.BodySchema{ ... },
    &tflint.GetModuleContentOption{ModuleCtx: tflint.RootModuleCtxType},
)

Similarly, you can switch the scope of expression evaluation from your own module to the root module. This removed the RootProvider and EvaluateExprOnRootCtx. Use GetModuleContent and EvaluateExpr with the tflint.RootModuleCtxType option.

There are some other changes related to the Runner API, so I'll list them here:

• IsNullExpr is removed. Use EvaluateExpr with cty.Value.
  • If you pass a pointer of cty.Value as the second argument, you can evaluate the expression without error even if it is null or unknown. cty.Value has IsNull method that determine if the value is null.
• File and Files are renamed to GetFile and GetFiles.
• DecodeRuleConfig identifies the hclext tag, not the hcl tag.
  • Previously, we used hashicorp/hcl's DecodeBody in this method, but the new API requires us to use hclext.DecodeBody, so we need to rename the tag. Note that the remain tag is not available here.
• EmitIssueOnExpr is removed. Use EmitIssue API.
  • Previously, we had defined a dedicated API to determine if the issue was emitted on an expression, but it's very confusing. See also rules: Use EmitIssueOnExpr when emitting an issue for an expression tflint-ruleset-aws#195 In the new plugin system, the plugin does not have to use a different API to determine if it is an issue for an expression from the range of the issue.

From net/rpc + gob to gRPC + Protocol Buffers

Protocol Buffers make the encoding process explicit, allowing type checking to detect errors that previously occurred at runtime. In addition, since the interface defined by proto can automatically generate the corresponding request/response structures by grpc-go, gRPC is also adopted.

The new plugin system defines the plugin/host2plugin and plugin/plugin2host packages to visualize the bi-directional communication. Each package has a gRPC server and a client defined, and encoding/decoding to proto is done explicitly in the toproto and fromproto packages.

The first server-client communication performed by go-plugin is host2plugin, and a plugin serves as a gRPC server. Then, if necessary, the host serves a gRPC server as plugin2host, and the plugin queries as a client. This way is the same as net/rpc.

Structured Logging

Previously, all logs output by plugins were at debug level, regardless of log level.

log.Println("[ERROR] log from plugin")

2022-12-30T23:11:04.764-0800 [DEBUG] plugin.tflint-ruleset-template: 2021/12/30 23:11:04 [ERROR] log from plugin

The new plugin supports structured logging by using the logger package, and the plugin can also output logs at the correct log level.

logger.Error("log from plugin")

22:14:49 [ERROR] rules/rule.go:27 log from plugin: module=tflint-ruleset-template

Other Changes

Several other changes have been made:

• RuleSet's ApplyConfig receives hclext.BodyContent instead of Config.
  • Previously, ApplyConfig was received a Config containing hcl.Body in a plugin block, but now it is changed to a schema-based implementation, and receives a BodyContent based on the passed schema.
  • The schema inside the plugin block must be defined with the ConfigSchema method.
• Custom rulesets must embed BuildinRuleSet.
  • Added mustEmbedBuiltinRuleSet method to guarantee the embedding.
• Added Metadata method to Rule interface.
  • This allows you to declare metadata to identify the rule type in your custom ruleset.
• All rules must embed DefaultRule.
  • Added mustEmbedDefaultRule method to guarantee the embedding.
• Severity() must be return tflint.Severity instead of string.