New option: --append to allow importing to existing workspace

[magodo]

For local backend, aztfy will by default ensure the output directory is empty at the very begining. This is to avoid any conflicts happen for existing user files, including the terraform configuration, provider configuration, the state file, etc. As a result, aztfy generates a pretty new workspace for users.

One limitation of doing so is users can't import resources to existing state file via aztfy. To support this scenario, this PR adds a new option --append. This option will make aztfy skip the empty guarantee for the output directory. If the output directory is empty, then it has no effect. Otherwise, it will ensure the provider setting (create a file for it if not exists). Then it proceeds the following steps.

This means if the output directory has an active Terraform workspace, i.e. there exists a state file, any resource imported by the aztfy will be imported into that state file. Especially, the file generated by aztfy in this case will be named differently than normal, where each file will has .aztfy suffix before the extension (e.g. main.aztfy.tf), to avoid potential file name conflicts.

• Fix Do not require target directory to be empty #135
• Fix Support import to existing state #132

Test

❯ AZTFY_E2E=1 go test -timeout=30m -v -run=TestMutateOutputDir ./internal/test
=== RUN   TestMutateOutputDir
Initializing...
List resources...
Import resources...
Importing /subscriptions/67a9759d-d099-4aa8-8675-e6cfd669c3f4/resourceGroups/aztfy-rg-budmjjug1 as azurerm_resource_group.round1_0
Generating Terraform configurations...
Initializing...
List resources...
Import resources...
Importing /subscriptions/67a9759d-d099-4aa8-8675-e6cfd669c3f4/resourceGroups/aztfy-rg-budmjjug2 as azurerm_resource_group.round2_0
Generating Terraform configurations...
--- PASS: TestMutateOutputDir (133.90s)
PASS
ok      github.com/Azure/aztfy/internal/test    134.036s

[magodo]

@stemaMSFT and @grayzu Despite the implementaions, I'm not so sure whether the option name --mutate-output-dir is appropriate here, as it sounds too verbose. Do you have any good naming for this?

[stemaMSFT]

I don't have a great alternative name for this, but I do agree that the name seems a little verbose. @grayzu let me know if you have any thoughts.

[magodo]

@stemaMSFT Shall we just call this --append (meaning the output directory will be kept as is, but just "appending" new stuff), to keep the same style as the --overwrite option (meaning the output directory will be removed and recreated)?

[ms-henglu]

LGTM

[stemaMSFT]

Hey @magodo, @grayzu and I were wondering why this behavior should not be by default available for the user. Rather than having to specify --append, maybe by default appending to a directory and having to specify --overwrite instead. What are your thoughts? This may be a potential revert to modify the behavior while still addressing the overarching issues.

[magodo]

@stemaMSFT The reason is "appending" to an existing directory might affecting user's data, so is regarded as a potential dangerous action. The default behavior is neither --append nor --overwrite, it is:

• In interactive mode: check whether the directory is empty, will prompt users to clean it up if not.
• In batch mode: check whether the directroy is empty, just error out if not.

One potential improvement is to make both modes consistent to be: check whether the directroy is empty. If not, tell the users to either use --overwrite to clean up the stuff, or use --append to append the artifacts to the dir. WDYT?

[koudaiii]

The reason is "appending" to an existing directory might affecting user's data, so is regarded as a potential dangerous action.

Just Idea: If aztfy has an option such as the --dry-run or --diff, we can understand what will be changed. If we know what will be changed, we might not need empty dir.

[magodo]

@koudaiii In order to understand the change that is to happen, that needs to complete the full process, including TF resource addresses specifying, importing and cfg generation. That seems not what --dry-run usually means. Whilst I still appreaciate your idea!

[koudaiii]

@magodo, I see !! I misunderstood it. Thank you for your help 😄