Add "DISTRIBUTED_ONLY" deployment mode support to NSX-T Edge Gateway

[Didainius]

Closes #1299

This PR adds new field - deployment_mode that supports two values ACTIVE_STANDBY (the default that the edge gateway worked always) and DISTRIBUTED_ONLY . A new feature of VCD 10.6

NON-DISTRIBUTED Edge Gateway

Edge Gateways (backed by Tier-1 NSX GWs) can now also be deployed in the Distributed only mode without the SR (services router) component. In such case the Tier-1 GW does not provide services that run on SR such as firewalling, NAT, VPN, DNS forwarding or static routes. The distributed nature guarantees high N/S data throughput (no hairpinning of traffic to single edge node running the active SR component). Load balancing, rate limiting or DHCP service in network mode is still supported. The other use case beside performance is to limit the NSX licensing costs.

Edge Gateway in NON_DISTRIBUTED mode supports only a small subset of configurations. The error is not completelly clear therefore there is an additional effort to make the error more clear if it matches the error for known resources that fail.

Error: [nsx-t firewall create/update] error creating NSX-T Firewall Rules: error setting
NSX-T Firewall: error in HTTP PUT request: ACCESS_TO_RESOURCE_IS_FORBIDDEN - [
19-2024-08-05-15-02-04-250--9e6beec5-5b47-4797-ab0d-162fed8d1401 ] Either you need some or
all of the following rights [ORG_VDC_GATEWAY_VIEW_FIREWALL] to perform operations
[GATEWAY_VIEW_FIREWALL_NSX_T] for 5f1fc518-865a-4c43-8b13-408c11ed8c06 or the target entity
is invalid.

to

│ Error: error setting NSX-T Firewall: error in HTTP PUT request: ACCESS_TO_RESOURCE_IS_FORBIDDEN - [ 19-2024-08-06-11-14-20-775--09cd9edc-5d2f-458d-9a98-608a3d178004 ] Either you need some or all of the following rights [ORG_VDC_GATEWAY_VIEW_FIREWALL] to perform operations [GATEWAY_VIEW_FIREWALL_NSX_T] for ccf22499-c009-47ab-9472-a05dfba80391 or the target entity is invalid.
│ 
│ vcd_nsxt_firewall cannot be configured on DISTRIBUTED_ONLY NSX-T Edge Gateway
│ 
│   with vcd_nsxt_firewall.testing,
│   on vcd.TestAccVcdNsxtEdgeGateway.tf line 52, in resource "vcd_nsxt_firewall" "testing":
│   52: resource "vcd_nsxt_firewall" "testing" {

Tested on 10.4.0 and 10.6.0 (nsxt tag)

[adambarreiro]

Quick scan

[lvirbalas]

I can't seem to find docs for this feature in the PR :)

[Didainius]

I can't seem to find docs for this feature in the PR :)

Indeed. Added

[lvirbalas]

Just a tiny polish comment.