Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how to retrieve or iterate all files #119

Closed
universam1 opened this issue May 27, 2021 · 18 comments
Closed

how to retrieve or iterate all files #119

universam1 opened this issue May 27, 2021 · 18 comments
Labels
enhancement New feature or request

Comments

@universam1
Copy link

universam1 commented May 27, 2021

I need to low-level iterate all files for style checking, but I can't see the list being accessible.
I read the intention of File(string) is such a use case, also there is an unexported list under runner.data.Files available, but I'm struggling to grasp their access.

Use case is to determine if a heredoc syntax has been used.

Thank you for help @wata727

@universam1
Copy link
Author

Feels like a runner.WalkFiles() or simpliy a runner.Files() accessor is missing

@wata727
Copy link
Member

wata727 commented May 29, 2021

Currently, there is no API provided to access the low-level HCL body, but I think it is possible to provide such an API.

@wata727 wata727 added the enhancement New feature or request label May 29, 2021
@universam1
Copy link
Author

thank you for the reply!
Would you accept an PR or what would be the plan @wata727 ?

@wata727
Copy link
Member

wata727 commented Jun 5, 2021

Sure! This issue will be resolved someday, but if you need features sooner, please open a PR and I'll be happy to review it.

@jonathansp
Copy link
Contributor

I've just opened this PR terraform-linters/tflint#1132 which I imagine is the first step to have Files() available to the sdk. If the implementation is correct and approved, then I will add this to the client side.

@wata727
Copy link
Member

wata727 commented Jun 12, 2021

Nice! It's a good idea to merge the sdk changes first and then update the TFLint side PR.

@nlamirault
Copy link

nlamirault commented Jul 2, 2021

Hi,
I try this code :

func (rule *TerraformPortefaixStandardStructureRule) checkDirectories(runner tflint.Runner) error {
	files, _ := runner.Files()
	log.Printf("[INFO] Files: %d", len(files))
	allowedFiles := map[string]bool{"providers.tf": true, "main.tf": true}

	for name := range files {
		_, filename := path.Split(name)
		log.Printf("[INFO] OK: %s %s", name, filename)
		if _, exists := allowedFiles[filename]; !exists {
			message := fmt.Sprintf("File %s is not allowed here.", filename)

			return runner.EmitIssue(rule, message, hcl.Range{Start: hcl.InitialPos})
		}
	}
}

But i've got no files. Any idea ?

❯ ls -aRl
drwxr-xr-x nicolas users  4.0 KB Mon Jun 28 16:10:26 2021  .
drwxr-xr-x nicolas users  4.0 KB Wed Mar  3 10:57:09 2021  ..
.rw-r--r-- nicolas users    6 B  Mon Jun 28 16:10:26 2021  .terraform-version
.rwxr-xr-x nicolas users  2.5 KB Fri Mar 19 17:49:02 2021  .terraform.lock.hcl
drwxr-xr-x nicolas users  4.0 KB Fri Mar 19 17:49:02 2021  backend-vars
.rw-r--r-- nicolas users  691 B  Fri Mar 19 17:49:02 2021  backend.tf
.rw-r--r-- nicolas users  682 B  Fri Mar 19 17:49:02 2021  data.tf
.rw-r--r-- nicolas users 1013 B  Fri Mar 19 17:49:02 2021  dns.tf
.rwxr-xr-x nicolas users  772 B  Thu Jun 17 09:14:10 2021  main.tf
.rw-r--r-- nicolas users  615 B  Fri Mar 19 17:49:02 2021  outputs.tf
.rw-r--r-- nicolas users  685 B  Fri Mar 19 17:49:02 2021  provider.tf
drwxr-xr-x nicolas users  4.0 KB Fri Mar 19 17:49:02 2021  tfvars
.rw-r--r-- nicolas users  1.4 KB Fri Mar 19 17:49:02 2021  variables.tf

./backend-vars:
drwxr-xr-x nicolas users 4.0 KB Fri Mar 19 17:49:02 2021  .
drwxr-xr-x nicolas users 4.0 KB Mon Jun 28 16:10:26 2021  ..
.rw-r--r-- nicolas users 649 B  Fri Mar 19 17:49:02 2021  prod.tfvars

./tfvars:
drwxr-xr-x nicolas users  4.0 KB Fri Mar 19 17:49:02 2021  .
drwxr-xr-x nicolas users  4.0 KB Mon Jun 28 16:10:26 2021  ..
.rw-r--r-- nicolas users 1017 B  Fri Mar 19 17:49:02 2021  prod.tfvars

❯ tflint -c ../../../../.tflint.hcl --loglevel=trace
09:53:14 config.go:96: [INFO] Load config: ../../../../.tflint.hcl
09:53:14 config.go:311: [DEBUG] Config loaded
09:53:14 config.go:312: [DEBUG]   Module: false
09:53:14 config.go:313: [DEBUG]   Force: false
09:53:14 config.go:314: [DEBUG]   IgnoreModules: map[string]bool{}
09:53:14 config.go:315: [DEBUG]   Varfiles: []string{}
09:53:14 config.go:316: [DEBUG]   Variables: []string{}
09:53:14 config.go:317: [DEBUG]   DisabledByDefault: false
09:53:14 config.go:318: [DEBUG]   Rules: map[string]*tflint.RuleConfig{"terraform_portefaix_standard_structure":(*tflint.RuleConfig)(0xc0005866c0)}
09:53:14 config.go:319: [DEBUG]   Plugins: map[string]*tflint.PluginConfig{"portefaix":(*tflint.PluginConfig)(0xc0005866f0)}
09:53:14 option.go:49: [DEBUG] CLI Options
09:53:14 option.go:50: [DEBUG]   Module: false
09:53:14 option.go:51: [DEBUG]   Force: false
09:53:14 option.go:52: [DEBUG]   IgnoreModules: map[string]bool{}
09:53:14 option.go:53: [DEBUG]   EnableRules: []string(nil)
09:53:14 option.go:54: [DEBUG]   DisableRules: []string(nil)
09:53:14 option.go:55: [DEBUG]   Only: []string(nil)
09:53:14 option.go:56: [DEBUG]   EnablePlugins: []string(nil)
09:53:14 option.go:57: [DEBUG]   Varfiles: []string{}
09:53:14 option.go:58: [DEBUG]   Variables: []string{}
09:53:14 loader.go:57: [INFO] Initialize new loader
09:53:14 loader.go:82: [INFO] Load configurations under .
09:53:14 loader.go:90: [INFO] Module inspection is disabled. Building a root module without children...
09:53:14 loader.go:170: [INFO] Load values files
09:53:14 runner.go:50: [INFO] Initialize new runner for root
09:53:14 terraform.go:158: [INFO] TF_VAR_* environment variable found: key=TF_VAR_master_authorized_networks
09:53:14 discovery.go:68: [INFO] Plugin `portefaix` found
2021-07-02T09:53:14.891+0200 [DEBUG] plugin: starting plugin: path=/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix args=[/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix]
2021-07-02T09:53:14.891+0200 [DEBUG] plugin: plugin started: path=/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix pid=1378465
2021-07-02T09:53:14.891+0200 [DEBUG] plugin: waiting for RPC address: path=/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix
2021-07-02T09:53:14.896+0200 [DEBUG] plugin.tflint-ruleset-portefaix: plugin address: address=/tmp/nicolas/plugin540353126 network=unix timestamp=2021-07-02T09:53:14.896+0200
2021-07-02T09:53:14.896+0200 [DEBUG] plugin: using plugin: version=8
09:53:14 provider.go:62: [INFO] Prepare rules
09:53:14 provider.go:90: [INFO]   3 default rules enabled
09:53:14 terraform_deprecated_interpolation.go:48: [TRACE] Check `terraform_deprecated_interpolation` rule for `root` runner
09:53:14 terraform_module_pinned_source.go:72: [TRACE] Check `terraform_module_pinned_source` rule for `root` runner
09:53:14 terraform_module_pinned_source.go:81: [DEBUG] Walk `dns.source` attribute
09:53:14 terraform_workspace_remote.go:48: [TRACE] Check `terraform_workspace_remote` rule for `root` runner
2021-07-02T09:53:14.900+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 09:53:14 [ERR] Check `terraform_portefaix_standard_structure` rule for runner
2021-07-02T09:53:14.900+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 09:53:14 [INFO] Files: 0
2021-07-02T09:53:14.900+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 09:53:14 [ERR] plugin: plugin server: accept unix /tmp/nicolas/plugin540353126: use of closed network connection
2021-07-02T09:53:14.901+0200 [DEBUG] plugin: plugin process exited: path=/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix pid=1378465
2021-07-02T09:53:14.901+0200 [DEBUG] plugin: plugin exited

@jonathansp
Copy link
Contributor

jonathansp commented Jul 2, 2021

Hi @nlamirault could you please provide which version of tflint and tflint-plugin-sdk you currently have installed? according to the docs, tflint does not recursively find files, but it should list files from the root/current working dir:

Does TFLint check modules recursively?

No. TFLint always checks only the current root module (no recursive check). However, you can check calling child modules based on module arguments by enabling Module Inspection. This allows you to check that you are not passing illegal values to the module.
Note that if you want to recursively inspect local modules, you need to run them in each directory. This is a limitation that occurs because Terraform always works for one directory. TFLint tries to emulate Terraform's semantics, so cannot perform recursive inspection.

@nlamirault
Copy link

$ cat go.mod
module github.com/nlamirault/tflint-ruleset-portefaix

go 1.16

require (
        github.com/agext/levenshtein v1.2.2 // indirect
        github.com/apparentlymart/go-textseg/v12 v12.0.0 // indirect
        github.com/fatih/color v1.10.0 // indirect
        github.com/golang/protobuf v1.4.3 // indirect
        github.com/google/go-cmp v0.5.6 // indirect
        github.com/hashicorp/go-plugin v1.4.1 // indirect
        github.com/hashicorp/go-version v1.3.0 // indirect
        github.com/hashicorp/hcl/v2 v2.10.0 // indirect
        github.com/hashicorp/yamux v0.0.0-20181012175058-2f1d1f20f75d // indirect
        github.com/kr/text v0.2.0 // indirect
        github.com/kylelemons/godebug v1.1.0 // indirect
        github.com/mitchellh/go-testing-interface v1.14.1 // indirect
        github.com/mitchellh/go-wordwrap v1.0.0 // indirect
        github.com/sergi/go-diff v1.2.0 // indirect
        github.com/stretchr/testify v1.7.0 // indirect
        github.com/terraform-linters/tflint-plugin-sdk v0.8.3-0.20210614125323-8364139f3745 // indirect
        github.com/zclconf/go-cty v1.8.3 // indirect
        golang.org/x/net v0.0.0-20210326060303-6b1517762897 // indirect
        golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
        google.golang.org/appengine v1.6.6 // indirect
        google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d // indirect
        google.golang.org/grpc v1.32.0 // indirect
        google.golang.org/protobuf v1.25.0 // indirect
        gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
)

and :

❯ tflint --version
TFLint version 0.28.1
+ ruleset.azurerm (0.8.2)
+ ruleset.google (0.5.1)
+ ruleset.aws (0.2.1)

@jonathansp
Copy link
Contributor

jonathansp commented Jul 2, 2021

missing a return nil at the end could be the cause? after the for loop

@nlamirault
Copy link

@jonathansp i've got it :) juste copy paste missing on github.

@jonathansp
Copy link
Contributor

  1. I've just created a tf-files dir:
ls -la
total 0
drwxr-xr-x   5 jonathan 1052847317  160 Jul  2 09:30 .
drwxr-xr-x  13 jonathan 1052847317  416 Jul  2 09:30 ..
-rw-r--r--   1 jonathan 1052847317    0 Jul  2 09:30 dns.tf
-rw-r--r--   1 jonathan 1052847317    0 Jul  2 09:30 main.tf
-rw-r--r--   1 jonathan 1052847317    0 Jul  2 09:30 providers.tf
  1. created a new plugin called test, using your code.

  2. $ make install

  3. $ tflint --enable-plugin=test tf-files --only=test_rule

  4. the result was

1 issue(s) found:

Error: File dns.tf is not allowed here. (test_rule)

  on  line 1:
   (source code not available)

do you think it could be caused for any other reason, like relative paths or anything else I can do to reproduce the problem?

@nlamirault
Copy link

I push my code here : https://github.com/nlamirault/tflint-ruleset-portefaix
Could you try it ? on the same directory.

@jonathansp
Copy link
Contributor

jonathansp commented Jul 2, 2021

$ git clone https://github.com/nlamirault/tflint-ruleset-portefaix.git
Cloning into 'tflint-ruleset-portefaix'...
remote: Enumerating objects: 38, done.
remote: Counting objects: 100% (38/38), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 38 (delta 9), reused 25 (delta 7), pack-reused 0
Unpacking objects: 100% (38/38), done.

$ cd tflint-ruleset-portefaix

$ make install
go build
mkdir -p ~/.tflint.d/plugins
mv ./tflint-ruleset-portefaix ~/.tflint.d/plugins

$ tflint-ruleset-portefaix git:(main)  tflint --enable-plugin=portefaix ../tf-files --only=terraform_portefaix_standard_structure
1 issue(s) found:

Error: File dns.tf is not allowed here. (terraform_portefaix_standard_structure)

  on  line 1:
   (source code not available)

Reference: https://github.com/nlamirault/tflint-ruleset-portefaix/blob/master/README.md
$ cat go.mod

module github.com/nlamirault/tflint-ruleset-portefaix

go 1.16

require (
	github.com/agext/levenshtein v1.2.3 // indirect
	github.com/fatih/color v1.12.0 // indirect
	github.com/google/go-cmp v0.5.6 // indirect
	github.com/hashicorp/go-hclog v0.16.1 // indirect
	github.com/hashicorp/go-plugin v1.4.2 // indirect
	github.com/hashicorp/go-version v1.3.0 // indirect
	github.com/hashicorp/hcl/v2 v2.10.0
	github.com/hashicorp/yamux v0.0.0-20210316155119-a95892c5f864 // indirect
	github.com/mattn/go-isatty v0.0.13 // indirect
	github.com/mitchellh/go-testing-interface v1.14.1 // indirect
	github.com/mitchellh/go-wordwrap v1.0.1 // indirect
	github.com/oklog/run v1.1.0 // indirect
	github.com/terraform-linters/tflint-plugin-sdk v0.8.3-0.20210614125323-8364139f3745
	github.com/vmihailenco/tagparser v0.1.2 // indirect
	github.com/zclconf/go-cty v1.8.4 // indirect
	golang.org/x/net v0.0.0-20210614182718-04defd469f4e // indirect
	golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c // indirect
	google.golang.org/appengine v1.6.7 // indirect
	google.golang.org/genproto v0.0.0-20210701191553-46259e63a0a9 // indirect
	google.golang.org/grpc v1.39.0 // indirect
)
$ tflint --version
TFLint version 0.29.0

I noticed you're using tflint 0.28, which doesn't expose Files() method yet. Update to 0.29 will solve this issue.

@nlamirault
Copy link

$ ls -alFrt
drwxr-xr-x nicolas users 4.0 KB Fri Jul  2 14:23:58 2021  ../
.rw-r--r-- nicolas users   0 B  Fri Jul  2 14:24:03 2021  foo.tf
.rw-r--r-- nicolas users   0 B  Fri Jul  2 14:24:07 2021  doe.tf
drwxr-xr-x nicolas users 4.0 KB Fri Jul  2 14:24:07 2021  ./

$ tflint --version                                                                                         
TFLint version 0.29.1

$ tflint . --loglevel=debug --enable-plugin=portefaix --enable-rule=terraform_portefaix_standard_structure 
14:27:11 config.go:105: [INFO] Load config: .tflint.hcl
14:27:11 config.go:117: [INFO] Default config file is not found. Ignored
14:27:11 config.go:126: [INFO] Load fallback config: /home/nicolas/.tflint.hcl
14:27:11 config.go:134: [INFO] Fallback config file is not found. Ignored
14:27:11 config.go:136: [INFO] Use default config
14:27:11 option.go:50: [DEBUG] CLI Options
14:27:11 option.go:51: [DEBUG]   Module: false
14:27:11 option.go:52: [DEBUG]   Force: false
14:27:11 option.go:53: [DEBUG]   IgnoreModules: map[string]bool{}
14:27:11 option.go:54: [DEBUG]   EnableRules: []string{"terraform_portefaix_standard_structure"}
14:27:11 option.go:55: [DEBUG]   DisableRules: []string(nil)
14:27:11 option.go:56: [DEBUG]   Only: []string(nil)
14:27:11 option.go:57: [DEBUG]   EnablePlugins: []string{"portefaix"}
14:27:11 option.go:58: [DEBUG]   Varfiles: []string{}
14:27:11 option.go:59: [DEBUG]   Variables: []string{}
14:27:11 loader.go:57: [INFO] Initialize new loader
14:27:11 loader.go:82: [INFO] Load configurations under .
14:27:11 loader.go:90: [INFO] Module inspection is disabled. Building a root module without children...
14:27:11 loader.go:170: [INFO] Load values files
14:27:11 runner.go:50: [INFO] Initialize new runner for root
14:27:11 terraform.go:158: [INFO] TF_VAR_* environment variable found: key=TF_VAR_master_authorized_networks
14:27:11 discovery.go:90: [DEBUG] Find plugin path: /home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix
14:27:11 discovery.go:54: [INFO] Plugin `portefaix` found
2021-07-02T14:27:11.255+0200 [DEBUG] plugin: starting plugin: path=/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix args=[/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix]
2021-07-02T14:27:11.255+0200 [DEBUG] plugin: plugin started: path=/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix pid=1693657
2021-07-02T14:27:11.255+0200 [DEBUG] plugin: waiting for RPC address: path=/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix
2021-07-02T14:27:11.262+0200 [DEBUG] plugin.tflint-ruleset-portefaix: plugin address: network=unix address=/tmp/nicolas/plugin443987081 timestamp=2021-07-02T14:27:11.262+0200
2021-07-02T14:27:11.262+0200 [DEBUG] plugin: using plugin: version=8
14:27:11 provider.go:62: [INFO] Prepare rules
14:27:11 provider.go:90: [INFO]   3 default rules enabled
2021-07-02T14:27:11.266+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 14:27:11 [DEBUG] Check `terraform_portefaix_standard_structure` rule for runner
2021-07-02T14:27:11.266+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 14:27:11 [DEBUG] Files: 0
2021-07-02T14:27:11.266+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 14:27:11 [ERR] plugin: plugin server: accept unix /tmp/nicolas/plugin443987081: use of closed network connection
2021-07-02T14:27:11.268+0200 [DEBUG] plugin: plugin process exited: path=/home/nicolas/.tflint.d/plugins/tflint-ruleset-portefaix pid=1693657
2021-07-02T14:27:11.268+0200 [DEBUG] plugin: plugin exited

I have no idea why it does not work :(

@jonathansp
Copy link
Contributor

I think I may have been using tflint from master branch, which would explain that. @wata727 can you please confirm if this feature was release already or will be?

@nlamirault
Copy link

It works with master branch !
Cool.

2021-07-02T15:05:42.924+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 15:05:42 [ERR] Check `terraform_portefaix_standard_structure` rule for runner
2021-07-02T15:05:42.933+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 15:05:42 [INFO] Files: 7
2021-07-02T15:05:42.933+0200 [DEBUG] plugin.tflint-ruleset-portefaix: 2021/07/02 15:05:42 [INFO] OK: iac/gcp/dns/terraform/backend.tf backend.tf
1 issue(s) found:

Error: File backend.tf is not allowed here. (terraform_portefaix_standard_structure)

  on  line 1:
   (source code not available)

Reference: https://github.com/nlamirault/tflint-ruleset-portefaix/blob/master/README.md

@wata727
Copy link
Member

wata727 commented Jul 3, 2021

TFLint v0.30 / Plugin SDK v0.9 has been released.

@wata727 wata727 closed this as completed Jul 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Development

No branches or pull requests

4 participants