At its core, terraform-validator is a thin layer on top of config-validator, a shared library that takes in a policy library and a set of CAI assets and reports back any violations of the specified policies.
terraform-validator consumes a Terraform plan and uses it to build CAI Assets, which then get run through config-validator. These built Assets only exist locally, in memory.
If an existing bundle (for example, CIS v1.1) doesn't support a check you need, please consider contributing a new constraint template to the policy-library repository.
The first step in determining if a GCP resource is supported is to figure out the name of the corresponding Terraform resource. You can often do this by searching for the GCP resource name in the Terraform google provider documentation.
A resource is "supported" by terraform-validator if it has an entry in resource_converters.go. For example, you could search resource_converters.go for google_compute_disk to see if that resource is supported.
Adding support for a resource has four steps:
- Make changes to Magic Modules to add any necessary code to terraform-google-conversion.
- Add tests for the new resource to terraform-validator, and run them against a locally-generated copy of terraform-google-conversion.
- Make PRs for Magic Modules & terraform-validator with your changes. The reviewer will double-check that your code works and then merge the Magic Modules PR.
- Once the Magic Modules PR is merged, it will automatically update terraform-google-conversion with your changes. Update your terraform-validator PR to use the new version of terraform-google.conversion.
Each of these is discussed in more detail below.
Note: terraform-validator can only support resources that are supported by the GA terraform provider, not beta resources.
Magic Modules uses a shared code base to generate terraform-google-conversion and the google and google-beta Terraform providers.
Most Terraform resources are represented as yaml files which are grouped by product.
Each product has an api.yaml file (which defines the basic API schema) and a terraform.yaml file (which defines any terraform-specific overrides.)
A terraform.yaml file can specify exclude_validator: true on a resource to skip terraform-google-conversion autogeneration, or exclude_resource: true to skip autogeneration for both terraform-google-conversion and the providers.
Auto-generating terraform-google-conversion code based on yaml files is strongly preferred. If this does not automatically add an entry to resource_converters.go, you can manually add an entry to resource_converters.go.erb using the autogenerated resource converter function.
If an autogenerated converter is not possible, you can instead place a handwritten file in the magic-modules/mmv1/third_party/validator folder.
Most resources will only need a resource converter with a conversion func, which should look something like:
// The type comes from https://cloud.google.com/asset-inventory/docs/supported-asset-types
const WhateverResourceAssetType string = "compute.googleapis.com/Route"
func resourceConverterWhateverResource() ResourceConverter {
return ResourceConverter{
AssetType: WhateverResourceAssetType,
Convert: GetWhateverResourceCaiObject,
}
}
func GetWhateverResourceCaiObject(d TerraformResourceData, config *Config) ([]Asset, error) {
// This function does basic conversion of a Terraform resource to a CAI Asset.
// The asset path (name) will substitute in variables from the Terraform resource.
// The format should match what is specified at https://cloud.google.com/asset-inventory/docs/supported-asset-types
name, err := assetName(d, config, "//whatever.googleapis.com/projects/{{project}}/whatevers/{{name}}")
if err != nil {
return []Asset{}, err
}
if obj, err := GetWhateverResourceApiObject(d, config); err == nil {
return []Asset{{
Name: name,
Type: WhateverResourceAssetType,
Resource: &AssetResource{
Version: "v1", // or whatever the correct version is
DiscoveryDocumentURI: "https://www.googleapis.com/path/to/rest/api/docs",
DiscoveryName: "Whatever", // The term used to refer to this resource by the official documentation
Data: obj,
},
}}, nil
} else {
return []Asset{}, err
}
}
func GetWhateverResourceApiObject(d TerraformResourceData, config *Config) (map[string]interface{}, error) {
obj := make(map[string]interface{})
// copy values from the terraform resource to obj
// return any errors encountered
// ...
return obj, nil
}You will also need to add an entry to resource_converters.go.erb, which is used to generate the resource_converters.go file in terraform-google-conversion. Each entry in resource_converters.go.erb maps a terraform resource name to a function that returns a ResourceConverter - in your case, resourceConverterWhateverResource.
To generate terraform-google-conversion code locally, run the following from the root of the magic-modules repository:
make validator OUTPUT_PATH="/path/to/your/terraform-google-conversion"
You can then run make test inside your terraform-google-conversion repository to make sure those tests pass.
Now that you have a local copy of terraform-google-conversion that has been generated from Magic Modules, you need to make Terraform Validator use it for local testing. You can do this with a replace directive:
replace github.com/GoogleCloudPlatform/terraform-google-conversion => /path/to/your/terraform-google-conversion
You can now build the binary (with make build) and test it. One way to do this would be to create a test project following the instructions in the policy library user guide (but using the binary you just built.) It's easiest to use a GCPAlwaysViolatesConstraintV1 constraint for testing new resources; this is what the tests do. terraform-validator convert tfplan.json can show you what terraform-validator thinks the converted Asset looks like.
Be sure to add test cases to test/cli_test.go and test/read_test.go. The test names refer to files in testdata/templates. You will generally need to add the following files:
- A .tf file.
- A .tfplan.json file.
- A .json file (representing the output of
terraform-validator convert)
See Getting started for details on running tests.
Try to get your tests passing locally before proceeding. (But you can also go ahead and open PRs if you're running into issues you can't figure out how to resolve.)
Now that you have your code working locally, open PRs for Magic Modules and terraform-validator. The reviewer will make sure your code works as expected.
For the Magic Modules PR, the most important check is terraform-google-conversion-test - as long as that's passing, you're probably fine. If it is failing, go back to step 1 and try running make test for terraform-google-conversion to reproduce & fix the failure.
For terraform-validator, the CI tests will not pass at this point, because the terraform-google-conversion dependency has not yet been updated. As long as the tests are passing locally for you, it should be fine.
After the Magic Modules PR is merged, and the terraform-google-conversion repository contains your changes, update the terraform-google-conversion dependency in your terraform-validator PR. This command will make the necessary changes:
go get github.com/GoogleCloudPlatform/terraform-google-conversionIf the CI tests are still failing after you make this change, double-check that you're able to run make test locally inside the terraform-validator repository using the updated (not replaced) dependency.