Skip to content

Ensure empty required object properties are in requests#2856

Merged
danielrbradley merged 11 commits into
masterfrom
required-containers
Oct 27, 2023
Merged

Ensure empty required object properties are in requests#2856
danielrbradley merged 11 commits into
masterfrom
required-containers

Conversation

@danielrbradley

@danielrbradley danielrbradley commented Oct 26, 2023

Copy link
Copy Markdown
Contributor
  1. Add "required containers" to the metadata to ensure the object always exists in the request even if no properties are set within it.
  2. Parse the required containers from the schema based on the property of the container being required in the object it's within.
  3. Add manually curated list of containers which are required even when not listed in the specification
  4. When constructing the containers in the provider, ensure required containers exist.

Fixes #2190
Fixes #2578

Based on a query of the generated metadata, this affects 337 resources (939 when considering multiple versions of the same resource):

alertsmanagement: PrometheusRuleGroup
alertsmanagement: TenantActivityLogAlert
apimanagement: ApiManagementService
apimanagement: ApiSchema
apimanagement: WorkspaceApiSchema
app: ManagedEnvironment
authorization: RoleAssignment
authorization: ManagementLockAtResourceGroupLevel
authorization: ManagementLockAtResourceLevel
authorization: ManagementLockAtSubscriptionLevel
authorization: ManagementLockByScope
authorization: PolicyExemption
authorization: Variable
authorization: VariableAtManagementGroup
authorization: VariableValue
authorization: VariableValueAtManagementGroup
automation: Webhook
automation: SoftwareUpdateConfigurationByName
automation: Certificate
automation: Connection
automation: ConnectionType
automation: Credential
automation: DscConfiguration
automation: HybridRunbookWorker
automation: JobSchedule
automation: Module
automation: Python2Package
automation: Python3Package
automation: Runbook
automation: Schedule
automation: SourceControl
automation: Variable
avs: Cluster
azureactivedirectory: B2CTenant
azurestack: LinkedSubscription
azurestack: Registration
azurestackhci: GuestAgent
azurestackhci: HybridIdentityMetadatum
blueprint: Assignment
blueprint: Blueprint
blueprint: PolicyAssignmentArtifact
blueprint: PublishedBlueprint
blueprint: RoleAssignmentArtifact
blueprint: TemplateArtifact
cache: Redis
cache: FirewallRule
cache: LinkedServer
cache: PatchSchedule
confluent: Organization
connectedvmwarevsphere: Cluster
connectedvmwarevsphere: Datastore
connectedvmwarevsphere: GuestAgent
connectedvmwarevsphere: Host
connectedvmwarevsphere: HybridIdentityMetadatum
connectedvmwarevsphere: InventoryItem
connectedvmwarevsphere: ResourcePool
connectedvmwarevsphere: VCenter
connectedvmwarevsphere: VirtualMachine
connectedvmwarevsphere: VirtualMachineTemplate
connectedvmwarevsphere: VirtualNetwork
connectedvmwarevsphere: VMInstanceGuestAgent
connectedvmwarevsphere: VirtualMachineInstance
containerinstance: ContainerGroup
containerservice: TrustedAccessRoleBinding
databox: Job
databoxedge: User
databoxedge: BandwidthSchedule
databoxedge: Container
databoxedge: IoTAddon
databoxedge: MonitoringConfig
databoxedge: Share
databoxedge: StorageAccount
databoxedge: StorageAccountCredential
databoxedge: ArcAddon
databoxedge: FileEventTrigger
databoxedge: PeriodicTimerEventTrigger
databricks: VNetPeering
databricks: Workspace
datafactory: ChangeDataCapture
datafactory: Pipeline
datalakeanalytics: Account
datalakeanalytics: ComputePolicy
datalakeanalytics: FirewallRule
datalakestore: FirewallRule
datalakestore: TrustedIdProvider
datalakestore: VirtualNetworkRule
datashare: ADLSGen2StorageAccountDataSet
datashare: ADLSGen2StorageAccountDataSetMapping
datashare: BlobStorageAccountDataSet
datashare: BlobStorageAccountDataSetMapping
datashare: ADLSGen1FileDataSet
datashare: ADLSGen1FolderDataSet
datashare: ADLSGen2FileDataSet
datashare: ADLSGen2FileDataSetMapping
datashare: ADLSGen2FileSystemDataSet
datashare: ADLSGen2FileSystemDataSetMapping
datashare: ADLSGen2FolderDataSet
datashare: ADLSGen2FolderDataSetMapping
datashare: BlobContainerDataSet
datashare: BlobContainerDataSetMapping
datashare: BlobDataSet
datashare: BlobDataSetMapping
datashare: BlobFolderDataSet
datashare: BlobFolderDataSetMapping
datashare: KustoClusterDataSet
datashare: KustoClusterDataSetMapping
datashare: KustoDatabaseDataSet
datashare: KustoDatabaseDataSetMapping
datashare: KustoTableDataSet
datashare: KustoTableDataSetMapping
datashare: ScheduledSynchronizationSetting
datashare: ScheduledTrigger
datashare: ShareSubscription
datashare: SqlDBTableDataSetMapping
datashare: SqlDWTableDataSetMapping
datashare: SynapseWorkspaceSqlPoolTableDataSet
datashare: SynapseWorkspaceSqlPoolTableDataSetMapping
dbformariadb: FirewallRule
dbformysql: FirewallRule
dbforpostgresql: FirewallRule
dbforpostgresql: Role
deploymentmanager: Rollout
deploymentmanager: Service
deploymentmanager: ServiceTopology
deploymentmanager: ServiceUnit
desktopvirtualization: ApplicationGroup
desktopvirtualization: HostPool
desktopvirtualization: Application
desktopvirtualization: MSIXPackage
desktopvirtualization: ScalingPlan
desktopvirtualization: ScalingPlanPooledSchedule
desktopvirtualization: ScalingPlanPersonalSchedule
deviceupdate: Instance
deviceupdate: PrivateEndpointConnection
devspaces: Controller
devtestlab: ArtifactSource
devtestlab: CustomImage
devtestlab: Disk
devtestlab: Environment
devtestlab: Formula
devtestlab: GlobalSchedule
devtestlab: NotificationChannel
devtestlab: Policy
devtestlab: Schedule
devtestlab: Secret
devtestlab: ServiceFabric
devtestlab: ServiceFabricSchedule
devtestlab: VirtualMachine
devtestlab: VirtualMachineSchedule
documentdb: CassandraResourceCassandraKeyspace
documentdb: CassandraResourceCassandraTable
documentdb: CassandraResourceCassandraView
documentdb: DatabaseAccount
documentdb: GraphResourceGraph
documentdb: GremlinResourceGremlinDatabase
documentdb: GremlinResourceGremlinGraph
documentdb: MongoClusterFirewallRule
documentdb: MongoDBResourceMongoDBCollection
documentdb: MongoDBResourceMongoDBDatabase
documentdb: SqlResourceSqlContainer
documentdb: SqlResourceSqlDatabase
documentdb: SqlResourceSqlStoredProcedure
documentdb: SqlResourceSqlTrigger
documentdb: SqlResourceSqlUserDefinedFunction
documentdb: TableResourceTable
edgeorder: AddressByName
edgeorder: OrderItemByName
edgeorder: Address
edgeorder: OrderItem
elasticsan: ElasticSan
elasticsan: PrivateEndpointConnection
elasticsan: Volume
elasticsan: VolumeSnapshot
hardwaresecuritymodules: DedicatedHsm
hdinsight: PrivateEndpointConnection
healthcareapis: IotConnectorFhirDestination
hybridcontainerservice: HybridIdentityMetadatum
hybriddata: DataStore
hybriddata: JobDefinition
insights: AlertRule
insights: LogProfile
insights: MetricAlert
insights: ScheduledQueryRule
insights: GuestDiagnosticsSetting
insights: GuestDiagnosticsSettingsAssociation
insights: PrivateLinkScope
insights: AutoscaleSetting
kubernetes: ConnectedCluster
labservices: Lab
labservices: LabPlan
labservices: Schedule
labservices: User
logic: Agreement
logic: Certificate
logic: Map
logic: Partner
logic: RosettaNetProcessConfiguration
logic: Schema
logic: Session
logic: IntegrationAccountAgreement
logic: IntegrationAccountCertificate
logic: IntegrationAccountMap
logic: IntegrationAccountPartner
logic: IntegrationAccountSchema
logic: IntegrationAccountSession
machinelearningservices: Workspace
managednetworkfabric: AccessControlList
managednetworkfabric: ExternalNetwork
managednetworkfabric: InternalNetwork
managednetworkfabric: IpPrefix
managednetworkfabric: NetworkInterface
managednetworkfabric: NetworkRack
managednetworkfabric: RoutePolicy
managednetworkfabric: InternetGateway
managednetworkfabric: InternetGatewayRule
managednetworkfabric: IpCommunity
managednetworkfabric: IpExtendedCommunity
managednetworkfabric: L2IsolationDomain
managednetworkfabric: L3IsolationDomain
managednetworkfabric: NeighborGroup
managednetworkfabric: NetworkDevice
managednetworkfabric: NetworkFabric
managednetworkfabric: NetworkFabricController
managednetworkfabric: NetworkPacketBroker
managednetworkfabric: NetworkTap
managednetworkfabric: NetworkTapRule
managednetworkfabric: NetworkToNetworkInterconnect
migrate: Assessment
migrate: Group
mobilenetwork: PacketCoreControlPlane
mobilenetwork: Sim
mobilenetwork: AttachedDataNetwork
mobilenetwork: MobileNetwork
mobilenetwork: PacketCoreDataPlane
mobilenetwork: Service
mobilenetwork: SimGroup
mobilenetwork: SimPolicy
mobilenetwork: Slice
mobilenetwork: PacketCapture
netapp: BackupPolicy
netapp: Volume
netapp: Backup
netapp: Pool
netapp: SnapshotPolicy
network: LocalNetworkGateway
network: VirtualNetworkGateway
network: VirtualNetworkGatewayConnection
network: ConnectionMonitor
network: PacketCapture
network: DnsForwardingRuleset
network: DnsResolver
network: ForwardingRule
network: InboundEndpoint
network: OutboundEndpoint
network: PrivateResolverVirtualNetworkLink
networkcloud: Cluster
networkcloud: DefaultCniNetwork
networkcloud: HybridAksCluster
networkcloud: AgentPool
networkcloud: BareMetalMachine
networkcloud: BareMetalMachineKeySet
networkcloud: BmcKeySet
networkcloud: ClusterManager
networkcloud: Console
networkcloud: KubernetesCluster
networkcloud: L2Network
networkcloud: L3Network
networkcloud: MetricsConfiguration
networkcloud: Rack
networkcloud: StorageAppliance
networkcloud: TrunkedNetwork
networkcloud: VirtualMachine
networkcloud: Volume
notificationhubs: NotificationHub
operationalinsights: LinkedService
operationalinsights: QueryPack
operationalinsights: LinkedStorageAccount
operationalinsights: SavedSearch
orbital: Contact
orbital: ContactProfile
orbital: Spacecraft
policyinsights: AttestationAtResource
policyinsights: AttestationAtResourceGroup
policyinsights: AttestationAtSubscription
relay: HybridConnectionAuthorizationRule
relay: NamespaceAuthorizationRule
relay: WCFRelayAuthorizationRule
resources: TemplateSpecVersion
resources: AzureCliScript
resources: AzurePowerShellScript
scvmm: Cloud
scvmm: GuestAgent
scvmm: HybridIdentityMetadata
scvmm: InventoryItem
scvmm: VirtualMachine
scvmm: VirtualMachineTemplate
scvmm: VirtualNetwork
scvmm: VmmServer
scvmm: VMInstanceGuestAgent
scvmm: VirtualMachineInstance
scvmm: AvailabilitySet
security: AdaptiveApplicationControl
security: JitNetworkAccessPolicy
securityinsights: AutomationRule
securityinsights: IncidentTask
servicefabricmesh: Application
servicefabricmesh: Gateway
servicefabricmesh: SecretValue
servicefabricmesh: Volume
servicelinker: Linker
servicelinker: Connector
signalrservice: SignalRCustomCertificate
signalrservice: SignalRCustomDomain
solutions: Application
solutions: ApplicationDefinition
storage: BlobContainerImmutabilityPolicy
storagemover: Agent
storagemover: JobDefinition
storagepool: DiskPool
storagepool: IscsiTarget
storsimple: AccessControlRecord
storsimple: BackupPolicy
storsimple: BackupSchedule
storsimple: BandwidthSetting
storsimple: StorageAccountCredential
storsimple: Volume
storsimple: VolumeContainer
timeseriesinsights: AccessPolicy
timeseriesinsights: ReferenceDataSet
timeseriesinsights: Gen2Environment
timeseriesinsights: EventHubEventSource
timeseriesinsights: Gen1Environment
timeseriesinsights: IoTHubEventSource
webpubsub: WebPubSubCustomCertificate
webpubsub: WebPubSubCustomDomain
workloads: SAPVirtualInstance
workloads: Connector

Add a single manual required container until we can derive from the spec.
We're interested in the request to make, when constructing the containers, rather than the response.
@danielrbradley danielrbradley self-assigned this Oct 26, 2023
@github-actions

Copy link
Copy Markdown
Contributor

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

@codecov

codecov Bot commented Oct 26, 2023

Copy link
Copy Markdown

Codecov Report

Merging #2856 (02acdf7) into master (4d9bd40) will increase coverage by 0.10%.
Report is 1 commits behind head on master.
The diff coverage is 84.31%.

@@            Coverage Diff             @@
##           master    #2856      +/-   ##
==========================================
+ Coverage   58.28%   58.38%   +0.10%     
==========================================
  Files          54       55       +1     
  Lines       10005    10056      +51     
==========================================
+ Hits         5831     5871      +40     
- Misses       3651     3660       +9     
- Partials      523      525       +2     
Files Coverage Δ
provider/pkg/gen/schema.go 76.12% <100.00%> (+0.09%) ⬆️
provider/pkg/resources/resources.go 68.58% <ø> (ø)
provider/pkg/gen/requiredContainers.go 60.00% <60.00%> (ø)
provider/pkg/gen/properties.go 72.49% <89.28%> (+1.46%) ⬆️
provider/pkg/provider/provider.go 49.74% <78.57%> (+0.06%) ⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@pulumi-bot pulumi-bot added the needs-triage Needs attention from the triage team label Oct 26, 2023
This fails without the required containers.
@danielrbradley danielrbradley marked this pull request as ready for review October 26, 2023 19:19
@danielrbradley danielrbradley removed the needs-triage Needs attention from the triage team label Oct 26, 2023
@danielrbradley danielrbradley changed the title Required containers Ensure empty required properties are sent Oct 26, 2023
- Make: Skip expanding then passing the list of go packages. `./...` does this for us.
- Use gotestfmt in CI/CD for better output. Still set same coverage flags as in make.
- Combine provider and arm2pulumi prebuild steps to make CI prebuild easier.
- Increase explicit `-parallel` from program test in CI to 16. Tests are not bound on local resources but rather on waiting for external services.
- Add actionlint.yaml config to silence warnings about self-hosted runners.

@thomas11 thomas11 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with two not strictly required suggestions.

Comment thread provider/pkg/gen/requiredContainers.go
Comment thread provider/pkg/gen/schema.go Outdated
- When merging containers, concatenate each entry so we can index it, then only add if we've not seen it before.
- Add docs for the additional required containers.
Managed Environments take >8 minutes to create and destroy. Query Pack takes less than 2 minutes.
- Rename the test to indicate its purpose.
- Tested locally that reverting the provider fix causes the test to fail with "Value cannot be null.\r\nParameter name: 'properties' property must be specified"
@danielrbradley danielrbradley changed the title Ensure empty required properties are sent Ensure empty required object properties are in requests Oct 27, 2023
@danielrbradley danielrbradley enabled auto-merge (squash) October 27, 2023 09:58
@danielrbradley danielrbradley merged commit afdb0cc into master Oct 27, 2023
@danielrbradley danielrbradley deleted the required-containers branch October 27, 2023 10:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v2] Can't create App.ManagedEnvironment: "Must contain properties" Can not create Log Analytics Query Pack

3 participants