Sync main branch into FEATURE-BRANCH-provider-functions (#10032)

* fixes: permadiff issue if event trigger region is not specified (#9989) * fixes: permadiff issue if event trigger region is not specified - fixes hashicorp/terraform-provider-google#17161 * chore: Updated cloudfunction2 test with no explicit trigger_region * Fix GKE front matter, bad field docs (#10018) * Fix GKE front matter, bad field docs * Update container_cluster.html.markdown * added default value for minPortsPerVM field at "google_compute_router_nat" (#9712) * added default value for statis allocation * setting the default from API * add newline removal bash command in guidelines (#9734) * Update hashicorp/terraform-plugin-framework, hashicorp/terraform-plugin-mux, hashicorp/terraform-plugin-sdk/v2 (#10009) * Adding Data Source Forwarding Rules (#10004) * Fix broken terraform datasource google_compute_machine_types example (#10020) * Add check for Environment proto field before accessing in Dataflow provider (#10016) * Add comments to skipped sweeper (#10023) * Add comments for skipped sweepers (#10024) * C3 version schema (#9986) * add support for build number in composerEnvironmentVersionRegexp and composerImageVersionDiffSuppress * make build number optional * regroup regex, cleaner comparison of versions * correction * Add Resource Manager Tags support to 'google_container_cluster' (#9531) * resourceManagerTags added to Cluster Node Config schema * update beta tag * add cluster and node proto tests * add expand and flatten proto * removed beta tag * added to documentation * added resource manager tags to auto pilot * migrating resourceManagerTags tests * migrating node_pools test * migrating additional tests * minor fixes * fixing tests * add in-place update support * fixed tests * fixed annotations * validated clusters and node pools tests. Isolated node pool auto config * isolated resource manager tags from docs * fixed permission issue * fixed spaces * fixed non determinism on tag keys * removed auto_pilot rmts * fixed time_sleep * add depends_on to IAM policies * Add volume replication support for Google Cloud NetApp Volumes (#9816) * Initial replication commit * Cleanup work - Renamed a lot of files to make clear which resource the belong to - Updated documentation for resource fields - Renamed a few resource fields and changed some types - Disabled the custom code for now. Needs to be discussed first * Update example file * Updated example file * Major updates - Reorganisation of block - Reorganisation of fields to match API documentation - Updated example parameters - Added missing API fields - Improved descriptions - * For replication deletion, stop replication first * Add support for deleting destination volume on replication delete * Make volumes deletable in presence of snapshots. This change will be PRed for volume resource independently. Adding it here while it is not in main. * Improving debug error message * yaml check and format fix * Add wait for mirror to initialize. Required to run destroy shortly after create. * Wait on destroy, not on create * Make deleting a replication more robust - doc improvements - started to implement stop/resume. More work required. - renamed a few files to better reflect what they are good for * adding support for stop/resume * yamlformat and lint * Add force delete to delete volumes with nested snapshots * resource test first version * More changes to make tests solid - Introduced new parameter to wait for mirror_status==MIRRORED - more mirror state reconciliation * Test updates * few cleanups * Make virtual field verifies happy * Minor test improvements * More fine tuning - Remove merge conflict in volume.yaml - make generated test work - make output field work - ignore_read for virtual fields * Resource name change as suggested by @slevenick * Remove snapshot code block and fix typo * Detect manual stop/resume actions * Remove ignore_read for deletion_policy * - Made destinationVolumeParameters immutable. It still requires ignore_read. - removed ignore_read from virtual_fields * destinationVolumeParameters are only evaluated at create. Make the immutable. * Name cleanups and comment improvements * removed comment Co-authored-by: Shuya Ma <[email protected]> * tabs to spaces in resource block Co-authored-by: Shuya Ma <[email protected]> * Updates to address review comments - make wait_for_mirror also work for stop/resume, additionally to create - convert tabs in test resource blocks to spaces - fix typos * Rewording of comments Co-authored-by: Shuya Ma <[email protected]> --------- Co-authored-by: G-NamanGupta <[email protected]> Co-authored-by: Shuya Ma <[email protected]> * Ensured that beta runs in TeamCity use only beta paths (#10025) * Ensured that beta runs in TeamCity use only beta paths * Added tests for sweeper package path --------- Co-authored-by: Deepak Kumar <[email protected]> Co-authored-by: Riley Karson <[email protected]> Co-authored-by: Max Portocarrero CI&T <[email protected]> Co-authored-by: Mauricio Alvarez Leon <[email protected]> Co-authored-by: Jesse Liddle <[email protected]> Co-authored-by: Wessel Blokzijl <[email protected]> Co-authored-by: Jack McCluskey <[email protected]> Co-authored-by: hao-nan-li <[email protected]> Co-authored-by: Salome Papiashvili <[email protected]> Co-authored-by: Oliver Krause <[email protected]> Co-authored-by: G-NamanGupta <[email protected]> Co-authored-by: Shuya Ma <[email protected]> Co-authored-by: Stephen Lewis (Burrows) <[email protected]>
GoogleCloudPlatform · Feb 22, 2024 · 566f948 · 566f948
1 parent a9f2961
commit 566f948
Show file tree

Hide file tree

Showing 40 changed files with 2,002 additions and 226 deletions.
diff --git a/mmv1/products/alloydb/User.yaml b/mmv1/products/alloydb/User.yaml
@@ -24,6 +24,7 @@ references: !ruby/object:Api::Resource::ReferenceLinks
     'AlloyDB': 'https://cloud.google.com/alloydb/docs/'
   api: 'https://cloud.google.com/alloydb/docs/reference/rest/v1/projects.locations.clusters.users/create'
 import_format: ['projects/{{project}}/locations/{{location}}/clusters/{{cluster}}/users/{{user_id}}']
+# Skipping the sweeper because instances will be deleted during cluster sweeps
 skip_sweeper: true
 autogen_async: true
 custom_code: !ruby/object:Provider::Terraform::CustomCode

diff --git a/mmv1/products/cloudfunctions2/Function.yaml b/mmv1/products/cloudfunctions2/Function.yaml
@@ -594,6 +594,7 @@ properties:
           events originating in this region. It can be the same
           region as the function, a different region or multi-region, or the global
           region. If not provided, defaults to the same region as the function.
+        default_from_api: true
       - !ruby/object:Api::Type::String
         name: 'eventType'
         description: 'Required. The type of event to observe.'

diff --git a/mmv1/products/compute/RouterNat.yaml b/mmv1/products/compute/RouterNat.yaml
@@ -236,7 +236,8 @@ properties:
   - !ruby/object:Api::Type::Integer
     name: minPortsPerVm
     description: |
-      Minimum number of ports allocated to a VM from this NAT.
+      Minimum number of ports allocated to a VM from this NAT. Defaults to 64 for static port allocation and 32 dynamic port allocation if not set.
+    default_from_api: true
   - !ruby/object:Api::Type::Integer
     name: maxPortsPerVm
     description: |

diff --git a/mmv1/products/firebase/AndroidApp.yaml b/mmv1/products/firebase/AndroidApp.yaml
@@ -52,6 +52,7 @@ import_format:
     '{{app_id}}',
   ]
 autogen_async: true
+# Sweeper skipped as this resource has customized deletion.
 skip_sweeper: true
 identity:
   - appId

diff --git a/mmv1/products/firebase/AppleApp.yaml b/mmv1/products/firebase/AppleApp.yaml
@@ -52,6 +52,7 @@ import_format:
     '{{app_id}}'
   ]
 autogen_async: true
+# Sweeper skipped as this resource has customized deletion.
 skip_sweeper: true
 identity:
   - appId

diff --git a/mmv1/products/firebase/WebApp.yaml b/mmv1/products/firebase/WebApp.yaml
@@ -52,6 +52,7 @@ import_format:
     '{{app_id}}',
   ]
 autogen_async: true
+# Sweeper skipped as this resource has customized deletion.
 skip_sweeper: true
 identity:
   - appId

diff --git a/mmv1/products/netapp/VolumeReplication.yaml b/mmv1/products/netapp/VolumeReplication.yaml
@@ -0,0 +1,289 @@
+# Copyright 2023 Google Inc.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+---
+!ruby/object:Api::Resource
+name: "VolumeReplication"
+description: |
+  Volume replication creates an asynchronous mirror of a volume in a different location. This capability
+  lets you use the replicated volume for critical application activity in case of a location-wide outage
+  or disaster.
+
+  A new destination volume is created as part of the replication resource. It's content is updated on a
+  schedule with content of the source volume. It can be used as a read-only copy while the mirror is
+  enabled, or as an independent read-write volume while the mirror is stopped. A destination volume will
+  also contain the snapshots of the source volume. Resuming a mirror will overwrite all changes on the
+  destination volume with the content of the source volume. While is mirror is enabled, all configuration
+  changes done to source or destination volumes are automatically done to both. Please note that the
+  destination volume is not a resource managed by Terraform.
+
+  Reversing the replication direction is not supported through the provider.
+references: !ruby/object:Api::Resource::ReferenceLinks
+  guides:
+    "Documentation": "https://cloud.google.com/netapp/volumes/docs/protect-data/about-volume-replication"
+  api: "https://cloud.google.com/netapp/volumes/docs/reference/rest/v1/projects.locations.volumes.replications"
+base_url: projects/{{project}}/locations/{{location}}/volumes/{{volume_name}}/replications
+self_link: projects/{{project}}/locations/{{location}}/volumes/{{volume_name}}/replications/{{name}}
+create_url: projects/{{project}}/locations/{{location}}/volumes/{{volume_name}}/replications?replicationId={{name}}
+update_url: projects/{{project}}/locations/{{location}}/volumes/{{volume_name}}/replications/{{name}}
+update_verb: :PATCH
+update_mask: true
+autogen_async: true
+async: !ruby/object:Api::OpAsync
+  operation: !ruby/object:Api::OpAsync::Operation
+    base_url: "{{op_id}}"
+id_format: "projects/{{project}}/locations/{{location}}/volumes/{{volume_name}}/replications/{{name}}"
+import_format:
+  [
+    "projects/{{project}}/locations/{{location}}/volumes/{{volume_name}}/replications/{{name}}",
+  ]
+examples:
+  - !ruby/object:Provider::Terraform::Examples
+    name: "netapp_volume_replication_create"
+    primary_resource_id: "test_replication"
+    vars:
+      source_pool_name: "source-pool"
+      destination_pool_name: "destination-pool"
+      volume_name: "source-volume"
+      replication_name: "test-replication"
+      destination_volume: "destination-volume"
+      network_name: "test-network"
+    ignore_read_extra:
+      - "delete_destination_volume"
+      - "replication_enabled"
+      - "force_stopping"
+      - "wait_for_mirror"
+    test_vars_overrides:
+      network_name: 'acctest.BootstrapSharedServiceNetworkingConnection(t, "gcnv-network-config-1", acctest.ServiceNetworkWithParentService("netapp.servicenetworking.goog"))'
+parameters:
+  - !ruby/object:Api::Type::String
+    name: "location"
+    required: true
+    immutable: true
+    url_param_only: true
+    description: |
+      Name of region for this resource. The resource needs to be created in the region of the destination volume.
+  - !ruby/object:Api::Type::String
+    name: "volumeName"
+    description: The name of the existing source volume.
+    required: true
+    immutable: true
+    url_param_only: true
+  - !ruby/object:Api::Type::String
+    name: "name"
+    description: The name of the replication. Needs to be unique per location.
+    required: true
+    immutable: true
+    url_param_only: true
+properties:
+  - !ruby/object:Api::Type::Enum
+    name: "state"
+    description: |
+      Indicates the state of replication resource. State of the mirror itself is indicated in mirrorState.
+    values:
+      - :STATE_UNSPECIFIED
+      - :CREATING
+      - :READY
+      - :UPDATING
+      - :DELETING
+      - :ERROR
+    output: true
+  - !ruby/object:Api::Type::String
+    name: "stateDetails"
+    description: |
+      State details of the replication resource.
+    output: true
+  - !ruby/object:Api::Type::Enum
+    name: "role"
+    description: |
+      Reverting a replication can swap source and destination volume roles. This field indicates if the `location` hosts
+      the source or destination volume. For resume and revert and resume operations it is critical to understand
+      which volume is the source volume, since it will overwrite changes done to the destination volume.
+    values:
+      - :REPLICATION_ROLE_UNSPECIFIED
+      - :SOURCE
+      - :DESTINATION
+    output: true
+  - !ruby/object:Api::Type::Enum
+    name: "replicationSchedule"
+    description: |
+      Specifies the replication interval.
+    values:
+      - :EVERY_10_MINUTES
+      - :HOURLY
+      - :DAILY
+    required: true
+  - !ruby/object:Api::Type::Enum
+    name: "mirrorState"
+    description: |
+      Indicates the state of the mirror between source and destination volumes. Depending on the amount of data
+      in your source volume, PREPARING phase can take hours or days. mirrorState = MIRRORED indicates your baseline
+      transfer ended and destination volume became accessible read-only. TRANSFERRING means a MIRRORED volume
+      currently receives an update. Updated every 5 minutes.
+    values:
+      - :MIRROR_STATE_UNSPECIFIED
+      - :PREPARING
+      - :MIRRORED
+      - :STOPPED
+      - :TRANSFERRING
+    custom_flatten: templates/terraform/custom_flatten/netapp_volume_replicaton_mirror_state.go.erb
+    output: true
+  - !ruby/object:Api::Type::String
+    name: "createTime"
+    description: |
+      Create time of the active directory. A timestamp in RFC3339 UTC "Zulu" format. Examples: "2023-06-22T09:13:01.617Z".
+    output: true
+  - !ruby/object:Api::Type::String
+    name: "destinationVolume"
+    description: |
+      Full resource name of destination volume with format: `projects/{{project}}/locations/{{location}}/volumes/{{volumeId}}`
+    output: true
+  - !ruby/object:Api::Type::NestedObject
+    name: "transferStats"
+    description: |-
+      Replication transfer statistics. All statistics are updated every 5 minutes.
+    output: true
+    properties:
+      - !ruby/object:Api::Type::String
+        name: "transferBytes"
+        description: |
+          Number of bytes transferred so far in current transfer.
+        output: true
+      - !ruby/object:Api::Type::String
+        name: "totalTransferDuration"
+        description: |
+          Total time taken so far during current transfer.
+        output: true
+      - !ruby/object:Api::Type::String
+        name: "lastTransferBytes"
+        description: |
+          Size of last completed transfer in bytes.
+        output: true
+      - !ruby/object:Api::Type::String
+        name: "lastTransferDuration"
+        description: |
+          Time taken during last completed transfer.
+        output: true
+      - !ruby/object:Api::Type::String
+        name: "lagDuration"
+        description: |
+          The elapsed time since the creation of the snapshot on the source volume that was last replicated
+          to the destination volume. Lag time represents the difference in age of the destination volume
+          data in relation to the source volume data.
+        output: true
+      - !ruby/object:Api::Type::String
+        name: "updateTime"
+        description: |
+          Time when progress was updated last. A timestamp in RFC3339 UTC "Zulu" format. Examples: "2023-06-22T09:13:01.617Z".
+        output: true
+      - !ruby/object:Api::Type::String
+        name: "lastTransferEndTime"
+        description: |
+          Time when last transfer completed. A timestamp in RFC3339 UTC "Zulu" format. Examples: "2023-06-22T09:13:01.617Z".
+        output: true
+      - !ruby/object:Api::Type::String
+        name: "lastTransferError"
+        description: |
+          A message describing the cause of the last transfer failure.
+        output: true
+  - !ruby/object:Api::Type::KeyValueLabels
+    name: "labels"
+    description: |
+      Labels as key value pairs. Example: `{ "owner": "Bob", "department": "finance", "purpose": "testing" }`
+  - !ruby/object:Api::Type::NestedObject
+    name: "destinationVolumeParameters"
+    description: |-
+      Destination volume parameters.
+    # destinationVolumeParameters is only used on CREATE. Will not be returned on READ.
+    ignore_read: true
+    properties:
+      - !ruby/object:Api::Type::String
+        name: "storagePool"
+        description: |
+          Name of an existing storage pool for the destination volume with format: `projects/{{project}}/locations/{{location}}/storagePools/{{poolId}}`
+        immutable: true
+        required: true
+      - !ruby/object:Api::Type::String
+        name: "volumeId"
+        description: |
+          Name for the destination volume to be created. If not specified, the name of the source volume will be used.
+        immutable: true
+        default_from_api: true
+      - !ruby/object:Api::Type::String
+        name: "shareName"
+        description: |
+          Share name for destination volume. If not specified, name of source volume's share name will be used.
+        immutable: true
+        default_from_api: true
+      - !ruby/object:Api::Type::String
+        name: "description"
+        description: |
+          Description for the destination volume.
+        immutable: true
+  - !ruby/object:Api::Type::String
+    name: "sourceVolume"
+    description: |
+      Full resource name of source volume with format: `projects/{{project}}/locations/{{location}}/volumes/{{volumeId}}`
+    output: true
+  - !ruby/object:Api::Type::Boolean
+    name: "healthy"
+    description: |
+      Condition of the relationship. Can be one of the following:
+        - true: The replication relationship is healthy. It has not missed the most recent scheduled transfer.
+        - false: The replication relationship is not healthy. It has missed the most recent scheduled transfer.
+    output: true
+  - !ruby/object:Api::Type::String
+    name: "description"
+    description: |
+      An description of this resource.
+virtual_fields:
+  - !ruby/object:Api::Type::Boolean
+    name: "delete_destination_volume"
+    description: |
+      A destination volume is created as part of replication creation. The destination volume will not became
+      under Terraform management unless you import it manually. If you delete the replication, this volume
+      will remain.
+      Setting this parameter to true will delete the *current* destination volume when destroying the
+      replication. If you reversed the replication direction, this will be your former source volume!
+      For production use, it is recommended to keep this parameter false to avoid accidental volume
+      deletion. Handle with care. Default is false.
+    default_value: false
+  - !ruby/object:Api::Type::Boolean
+    name: "replication_enabled"
+    description: |
+      Set to false to stop/break the mirror. Stopping the mirror makes the destination volume read-write
+      and act independently from the source volume.
+      Set to true to enable/resume the mirror. WARNING: Resuming a mirror overwrites any changes
+      done to the destination volume with the content of the source volume.
+    default_value: true
+  - !ruby/object:Api::Type::Boolean
+    name: "force_stopping"
+    description: |
+      Only replications with mirror_state=MIRRORED can be stopped. A replication in mirror_state=TRANSFERRING
+      currently receives an update and stopping the update might be undesirable. Set this parameter to true
+      to stop anyway. All data transferred to the destination will be discarded and content of destination
+      volume will remain at the state of the last successful update. Default is false.
+    default_value: false
+  - !ruby/object:Api::Type::Boolean
+    name: "wait_for_mirror"
+    description: |
+      Replication resource state is independent of mirror_state. With enough data, it can take many hours
+      for mirror_state to reach MIRRORED. If you want Terraform to wait for the mirror to finish on
+      create/stop/resume operations, set this parameter to true. Default is false.
+    default_value: false
+custom_code: !ruby/object:Provider::Terraform::CustomCode
+  constants: templates/terraform/constants/netapp_volume_replication.go.erb
+  post_create: templates/terraform/post_create/netapp_volume_replication_post_create.go.erb
+  pre_delete: templates/terraform/pre_delete/netapp_volume_replication_stop_before_delete.go.erb
+  post_delete: templates/terraform/post_delete/netapp_volume_replication_delete_destination_volume.go.erb
+  post_update: templates/terraform/post_update/netapp_volume_replication_mirror_state.go.erb
diff --git a/mmv1/templates/terraform/constants/netapp_volume_replication.go.erb b/mmv1/templates/terraform/constants/netapp_volume_replication.go.erb
@@ -0,0 +1,51 @@
+// Custom function to wait for mirrorState target states
+func NetAppVolumeReplicationWaitForMirror(d *schema.ResourceData, meta interface{}, targetState string) error {
+	config := meta.(*transport_tpg.Config)
+	userAgent, err := tpgresource.GenerateUserAgentString(d, config.UserAgent)
+	if err != nil {
+		return err
+	}
+
+	url, err := tpgresource.ReplaceVars(d, config, "{{NetappBasePath}}projects/{{project}}/locations/{{location}}/volumes/{{volume_name}}/replications/{{name}}")
+	if err != nil {
+		return err
+	}
+
+	billingProject := ""
+
+	project, err := tpgresource.GetProject(d, config)
+	if err != nil {
+		return fmt.Errorf("Error fetching project for volume replication: %s", err)
+	}
+	billingProject = project
+
+	// err == nil indicates that the billing_project value was found
+	if bp, err := tpgresource.GetBillingProject(d, config); err == nil {
+		billingProject = bp
+	}
+
+	for {
+		res, err := transport_tpg.SendRequest(transport_tpg.SendRequestOptions{
+			Config:    config,
+			Method:    "GET",
+			Project:   billingProject,
+			RawURL:    url,
+			UserAgent: userAgent,
+		})
+		if err != nil {
+			return transport_tpg.HandleNotFoundError(err, d, fmt.Sprintf("NetappVolumeReplication %q", d.Id()))
+		}
+
+		log.Printf("[DEBUG] waiting for mirrorState. actual: %v, target: %v", res["mirrorState"], targetState)
+
+		if res["mirrorState"] == targetState {
+			break
+		}
+
+		time.Sleep(30 * time.Second)
+		// This method can potentially run for days, e.g. when setting up a replication for a source volume
+		// with dozens of TiB of data. Timeout handling yes/no?
+	}
+
+	return nil
+}