1122 API requests only to list VMs

[mritzmann]

We manage several hundred servers with Terraform. A terraform plan lasts longer than an hour, which is not very convenient. That's why we took a look at Terraform's DEBUG Log.

env TF_LOG=DEBUG TF_LOG_PATH="/tmp/terraform.log" terraform plan

First, some interesting statistics: Terraform executes 1404 API requests.

$ grep -i "/api/nutanix/" /tmp/terraform.log | wc -l
1404

Of these, 1122 are only for listing VMs.

$ grep -i "/api/nutanix/v3/vms/list" /tmp/terraform.log | wc -l
1122

I don't know golang enough, but I tried to find out where all the requests come from.

1. There is a function named resourceNutanixVirtualMachineExists. I assume this function is called once for each individual VM, to check whether a VM exists.
2. Function resourceNutanixVirtualMachineExists calls a function named ListAllVM.
3. Function ListAllVM calls a function named ListVM.
4. Function ListVM makes multiple API requests to /api/nutanix/v3/vms/list (multiple requests because of pagination).

Summary:

• For each VM, the entire list of all VMs is retrieved each time.
• Because of pagination this leads to several requests per server
• A request takes 5-15 seconds depending on the load.

A terraform state with ~500 servers thus leads to ~2500 requests

500 Servers * (1x5 pagination) = 2500 Requests
2500 Requests * ~ 7s = 17500s

Maybe there's a good reason why you're doing this... From my point of view it can never scale that way.

Expected behavior
The list with all servers should not be fetched every time anew. A single time per terraform plan is sufficient. In this example, only five requests would be necessary instead of 2500.

Logs
I'll be happy to deliver more if necessary.

Versions

• OS: Debian 8.11
• Terraform: v0.11.14
• Nutanix Cluster (Prism Element / AOS): Version 5.10.6 LTS
• Nutanix Prism Central: 5.10.6
• Terraform provider version: v1.0.2

[marinsalinas]

@mritzmann, could you test it again in the master branch to see if the issue is solved?

[mritzmann]

I tested it yesterday with version v1.1.0. Seems to be fixed.

• terraform plan bevor: longer than an hour
• terraform plan now: three minutes

@marinsalinas thanks for your work! 🙇