Skip to content

[Fix] Remove single-node validation from jobs clusters#4216

Merged
shreyas-goenka merged 3 commits into
mainfrom
remove/job-cluster-validation
Nov 13, 2024
Merged

[Fix] Remove single-node validation from jobs clusters#4216
shreyas-goenka merged 3 commits into
mainfrom
remove/job-cluster-validation

Conversation

@shreyas-goenka
Copy link
Copy Markdown
Contributor

@shreyas-goenka shreyas-goenka commented Nov 12, 2024 •
edited
Loading

Changes

Fixes databricks/cli#1896. Introducing this validation has caused a regression for both DABs and TF customers. This PR removes the validation for job clusters.

Tests

Unit tests.

shreyas-goenka
shreyas-goenka commented Nov 12, 2024
View reviewed changes
Comment thread clusters/clusters_api.go
@shreyas-goenka shreyas-goenka marked this pull request as ready for review November 12, 2024 16:23
@shreyas-goenka shreyas-goenka requested review from a team as code owners November 12, 2024 16:23
@shreyas-goenka shreyas-goenka changed the title Remove single-node validation from jobs clusters [fix] Remove single-node validation from jobs clusters Nov 12, 2024
@shreyas-goenka shreyas-goenka changed the title [fix] Remove single-node validation from jobs clusters [Fix] Remove single-node validation from jobs clusters Nov 12, 2024
@alexott
Copy link
Copy Markdown
Contributor

alexott commented Nov 13, 2024

Do we still need #4206?

@pietern
Copy link
Copy Markdown
Contributor

pietern commented Nov 13, 2024

@shreyas-goenka Please update this as well:

terraform-provider-databricks/clusters/resource_cluster.go

Lines 133 to 135 in e3b2561

// This method is a duplicate of Validate() in clusters/clusters_api.go that uses Go SDK.
// Long term, Validate() in clusters_api.go will be removed once all the resources using clusters are migrated to Go SDK.
func Validate(cluster any) error {

@tanmay-db Can you confirm if this aligns with the direction you're taking for these resources?

@shreyas-goenka
Copy link
Copy Markdown
Contributor Author

shreyas-goenka commented Nov 13, 2024

@alexott Yeah we still need #4206 since it tackles the issue of policy not working from clusters. I need to update that PR however.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Nov 13, 2024

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/terraform

Inputs:

  • PR number: 4216
  • Commit SHA: 3176788b2327e9aee54f894ca889d710559a212e

Checks will be approved automatically on success.

@eng-dev-ecosystem-bot
Copy link
Copy Markdown
Collaborator

eng-dev-ecosystem-bot commented Nov 13, 2024

Test Details: go/deco-tests/11815755779

@shreyas-goenka shreyas-goenka added this pull request to the merge queue Nov 13, 2024
Merged via the queue into main with commit 8f68baa Nov 13, 2024
@shreyas-goenka shreyas-goenka deleted the remove/job-cluster-validation branch November 13, 2024 11:55
@shreyas-goenka shreyas-goenka mentioned this pull request Nov 13, 2024
Closed
@shreyas-goenka shreyas-goenka mentioned this pull request Nov 13, 2024
Merged
@tanmay-db
Copy link
Copy Markdown
Contributor

tanmay-db commented Nov 14, 2024

@shreyas-goenka Please update this as well:

terraform-provider-databricks/clusters/resource_cluster.go

Lines 133 to 135 in e3b2561

// This method is a duplicate of Validate() in clusters/clusters_api.go that uses Go SDK.
// Long term, Validate() in clusters_api.go will be removed once all the resources using clusters are migrated to Go SDK.
func Validate(cluster any) error {

@tanmay-db Can you confirm if this aligns with the direction you're taking for these resources?

Hi @pietern, yes this would be implemented a bit differently in plugin framework so it should be good. Eventually we will remove the cluster resource in SDKv2 so there would only be one place where validation would happen.

github-merge-queue Bot pushed a commit that referenced this pull request Nov 14, 2024
@shreyas-goenka
[Fix] Remove single-node validation from interactive clusters (#4222)
6e7ca4c 
## Changes
There were reports on databricks/cli#1546
about customers being unable to use cluster policies for single-node job
clusters because Terraform was performing this validation.
#4216
removed this validation for job clusters. This PR removes this
validation for interactive clusters as well to keep them consistent and
unblock the use of policy from interactive clusters.

Also, the clusters team is improving the API interface to simplify
creating single-node clusters, significantly reducing the chances of
users misconfiguring them.

## Tests
Unit test and manually. Clusters are now successfully created and
updated with `num_workers=0`
hectorcast-db added a commit that referenced this pull request Nov 20, 2024
@hectorcast-db
[Release] Release v1.59.0
c075486 
### New Features and Improvements

 * Add `databricks_mws_network_connectivity_config` and `databricks_mws_network_connectivity_configs` data source ([#3665](#3665)).
 * Add support partitions in policy data sources ([#4181](#4181)).
 * Added `databricks_registered_model_versions` data source ([#4100](#4100)).
 * Update databricks_permissions resource to support vector-search-endpoints ([#4209](#4209)).
 * add `databricks_serving_endpoints` data source ([#4226](#4226)).

### Bug Fixes

 * Add validation for `run_as_mode` in `databricks_query` ([#4233](#4233)).
 * Correct handling of updates for empty comments and `force_destroy` in UC catalog, schema, registered models and volumes ([#4244](#4244)).
 * Fix deletion of dashboard if it was trashed out of band ([#4235](#4235)).
 * Fix waiting for `databricks_vector_search_index` readiness ([#4243](#4243)).
 * Remove single-node validation from interactive clusters ([#4222](#4222)).
 * Remove single-node validation from jobs clusters ([#4216](#4216)).
 * Use cluster list API to determine pinned cluster status ([#4203](#4203)).
 * fix issue cased by setting pause_status in update monitor  ([#4242](#4242)).

### Documentation

 * Clarify workspace provider config ([#4208](#4208)).
 * Update "Databricks Workspace Creator" permissions on gcp-workspace.md ([#4201](#4201)).
 * Update `grants.md` references ([#4246](#4246)).
 * Update description of `group_id` in `databricks_mws_ncc_private_endpoint_rule` ([#4238](#4238)).
 * remove subnet sharing limitation in AWS ([#4239](#4239)).

### Internal Changes

 * Bump Go SDK to latest and generate TF structs ([#4249](#4249)).
 * Mark TestUcAccModelServingProvisionedThroughput as flaky. to be rever… ([#4232](#4232)).
 * Rename resources directory to products in pluginframework ([#4139](#4139)).
 * Revert "mark TestUcAccModelServingProvisionedThroughput as flaky. to … ([#4240](#4240)).
 * Set user agent in some resources implemented in plugin framework ([#4187](#4187)).
 * make `ApplyAndExpectData` work with nested set ([#4237](#4237)).

### Dependency Updates

 * Bump dependencies for Plugin Framework and SDK v2 ([#4215](#4215)).
 * Bump github.com/hashicorp/hcl/v2 from 2.22.0 to 2.23.0 ([#4236](#4236)).
 * Bump github.com/hashicorp/terraform-plugin-testing from 1.10.0 to 1.11.0 ([#4247](#4247)).

### Exporter

 * Add `List` operation for `users` service ([#4204](#4204)).
 * Fix interactive selection of services ([#4245](#4245)).
@hectorcast-db hectorcast-db mentioned this pull request Nov 20, 2024
Merged
hectorcast-db added a commit that referenced this pull request Nov 20, 2024
@hectorcast-db
[Release] Release v1.59.0
81b4e7c 
 * Add `databricks_mws_network_connectivity_config` and `databricks_mws_network_connectivity_configs` data source ([#3665](#3665)).
 * Add support partitions in policy data sources ([#4181](#4181)).
 * Added `databricks_registered_model_versions` data source ([#4100](#4100)).
 * Update databricks_permissions resource to support vector-search-endpoints ([#4209](#4209)).
 * add `databricks_serving_endpoints` data source ([#4226](#4226)).

 * Add validation for `run_as_mode` in `databricks_query` ([#4233](#4233)).
 * Correct handling of updates for empty comments and `force_destroy` in UC catalog, schema, registered models and volumes ([#4244](#4244)).
 * Fix deletion of dashboard if it was trashed out of band ([#4235](#4235)).
 * Fix waiting for `databricks_vector_search_index` readiness ([#4243](#4243)).
 * Remove single-node validation from interactive clusters ([#4222](#4222)).
 * Remove single-node validation from jobs clusters ([#4216](#4216)).
 * Use cluster list API to determine pinned cluster status ([#4203](#4203)).
 * fix issue cased by setting pause_status in update monitor  ([#4242](#4242)).

 * Clarify workspace provider config ([#4208](#4208)).
 * Update "Databricks Workspace Creator" permissions on gcp-workspace.md ([#4201](#4201)).
 * Update `grants.md` references ([#4246](#4246)).
 * Update description of `group_id` in `databricks_mws_ncc_private_endpoint_rule` ([#4238](#4238)).
 * remove subnet sharing limitation in AWS ([#4239](#4239)).

 * Bump Go SDK to latest and generate TF structs ([#4249](#4249)).
 * Mark TestUcAccModelServingProvisionedThroughput as flaky. to be rever… ([#4232](#4232)).
 * Rename resources directory to products in pluginframework ([#4139](#4139)).
 * Revert "mark TestUcAccModelServingProvisionedThroughput as flaky. to … ([#4240](#4240)).
 * Set user agent in some resources implemented in plugin framework ([#4187](#4187)).
 * make `ApplyAndExpectData` work with nested set ([#4237](#4237)).

 * Bump dependencies for Plugin Framework and SDK v2 ([#4215](#4215)).
 * Bump github.com/hashicorp/hcl/v2 from 2.22.0 to 2.23.0 ([#4236](#4236)).
 * Bump github.com/hashicorp/terraform-plugin-testing from 1.10.0 to 1.11.0 ([#4247](#4247)).

 * Add `List` operation for `users` service ([#4204](#4204)).
 * Fix interactive selection of services ([#4245](#4245)).
github-merge-queue Bot pushed a commit that referenced this pull request Nov 20, 2024
@hectorcast-db
[Release] Release v1.59.0 (#4252)
68ba26d 
### New Features and Improvements

* Add `databricks_mws_network_connectivity_config` and
`databricks_mws_network_connectivity_configs` data source
([#3665](#3665)).
* Add support partitions in policy data sources
([#4181](#4181)).
* Added `databricks_registered_model_versions` data source
([#4100](#4100)).
* Update databricks_permissions resource to support
vector-search-endpoints
([#4209](#4209)).
* add `databricks_serving_endpoints` data source
([#4226](#4226)).


### Bug Fixes

* Add validation for `run_as_mode` in `databricks_query`
([#4233](#4233)).
* Correct handling of updates for empty comments and `force_destroy` in
UC catalog, schema, registered models and volumes
([#4244](#4244)).
* Fix deletion of dashboard if it was trashed out of band
([#4235](#4235)).
* Fix waiting for `databricks_vector_search_index` readiness
([#4243](#4243)).
* Remove single-node validation from interactive clusters
([#4222](#4222)).
* Remove single-node validation from jobs clusters
([#4216](#4216)).
* Use cluster list API to determine pinned cluster status
([#4203](#4203)).
* fix issue cased by setting pause_status in update monitor
([#4242](#4242)).


### Documentation

* Clarify workspace provider config
([#4208](#4208)).
* Update "Databricks Workspace Creator" permissions on gcp-workspace.md
([#4201](#4201)).
* Update `grants.md` references
([#4246](#4246)).
* Update description of `group_id` in
`databricks_mws_ncc_private_endpoint_rule`
([#4238](#4238)).
* remove subnet sharing limitation in AWS
([#4239](#4239)).


### Internal Changes

* Bump Go SDK to latest and generate TF structs
([#4249](#4249)).
* Mark TestUcAccModelServingProvisionedThroughput as flaky. to be rever…
([#4232](#4232)).
* Rename resources directory to products in pluginframework
([#4139](#4139)).
* Revert "mark TestUcAccModelServingProvisionedThroughput as flaky. to …
([#4240](#4240)).
* Set user agent in some resources implemented in plugin framework
([#4187](#4187)).
* make `ApplyAndExpectData` work with nested set
([#4237](#4237)).


### Dependency Updates

* Bump dependencies for Plugin Framework and SDK v2
([#4215](#4215)).
* Bump github.com/hashicorp/hcl/v2 from 2.22.0 to 2.23.0
([#4236](#4236)).
* Bump github.com/hashicorp/terraform-plugin-testing from 1.10.0 to
1.11.0
([#4247](#4247)).


### Exporter

* Add `List` operation for `users` service
([#4204](#4204)).
* Fix interactive selection of services
([#4245](#4245)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alexott alexott alexott approved these changes

@tanmay-db tanmay-db Awaiting requested review from tanmay-db

@renaudhartert-db renaudhartert-db Awaiting requested review from renaudhartert-db

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Cannot create a cluster with num_workers as 0

5 participants

@shreyas-goenka @alexott @pietern @eng-dev-ecosystem-bot @tanmay-db