Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE]: infer linted values that resolve to dbutils.widgets.get() #1880

Closed
ericvergnaud opened this issue Jun 11, 2024 · 0 comments · Fixed by #1891
Closed

[FEATURE]: infer linted values that resolve to dbutils.widgets.get() #1880

ericvergnaud opened this issue Jun 11, 2024 · 0 comments · Fixed by #1891
Assignees
Labels
migrate/jobs Step 5 - Upgrading Jobs for External Tables migrate/python Pull requests that update Python code

Comments

@ericvergnaud
Copy link
Contributor

ericvergnaud commented Jun 11, 2024

dbutils.widgets.get(X) can retrieve values supplied from Job's named parameters. These values may contain random data, including storage location prefixes and table names:

selected_table = dbutils.widgets.get("selected_table")
query = f"select * from some_schema.{selected_table}"
spark.sql(query").collect()

As of de4fa7f we fetch named parameters into CurrentSessionState:

@dataclass
class CurrentSessionState:
"""
A data class that represents the current state of a session.
This class can be used to track various aspects of a session, such as the current schema.
Attributes:
catalog (str): The current schema of the session. If not provided, it defaults to 'DEFAULT_CATALOG'.
schema (str): The current schema of the session. If not provided, it defaults to 'DEFAULT_SCHEMA'.
"""
schema: str = DEFAULT_SCHEMA
catalog: str = DEFAULT_CATALOG
spark_conf: dict[str, str] | None = None
named_parameters: dict[str, str] | None = None
data_security_mode: compute.DataSecurityMode | None = None

In scope:

  • Resolve every dbutils.widgets.get("XXXXXXXX") into CurrentSessionState#named_parameters.get("XXXXXXXX")
  • [stretch-goal] dbutils.widgets.getAll() to resolve into CurrentSessionState#named_parameters value

Out of scope:

  • parsing of any other methods of dbutils.widgets than get(str) -> str and getAll() -> dict[str,str].
@ericvergnaud ericvergnaud added enhancement New feature or request needs-triage labels Jun 11, 2024
@ericvergnaud ericvergnaud added this to UCX Jun 11, 2024
@github-project-automation github-project-automation bot moved this to Triage in UCX Jun 11, 2024
@ericvergnaud ericvergnaud self-assigned this Jun 11, 2024
@ericvergnaud ericvergnaud changed the title [FEATURE]: do not raise advices when linting values that resolve to dbutils.widgets.get() [FEATURE]: infer linted values that resolve to dbutils.widgets.get() Jun 11, 2024
@nfx nfx added migrate/jobs Step 5 - Upgrading Jobs for External Tables migrate/python Pull requests that update Python code and removed enhancement New feature or request needs-triage labels Jun 11, 2024
@nfx nfx closed this as completed in #1891 Jun 13, 2024
@github-project-automation github-project-automation bot moved this from Triage to Archive in UCX Jun 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
migrate/jobs Step 5 - Upgrading Jobs for External Tables migrate/python Pull requests that update Python code
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

2 participants