Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE]: Simplify problem collection during building of dependency graph #1559

Closed
1 task done
Tracked by #1202
ericvergnaud opened this issue Apr 25, 2024 · 0 comments · Fixed by #1613
Closed
1 task done
Tracked by #1202

[FEATURE]: Simplify problem collection during building of dependency graph #1559

ericvergnaud opened this issue Apr 25, 2024 · 0 comments · Fixed by #1613
Assignees
@nfx @ericvergnaud
Labels
tech debt chores and design flaws

Comments

@ericvergnaud
Copy link
Contributor

ericvergnaud commented Apr 25, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Problem statement

Collecting problems during building of dependency graph is a complex task that involves many methods that may not have access to all the required data and/or endpoints.
As a result our current implementation ends up passing a dependency_collector callback to many methods, which is inelegant and brittle.

Proposed Solution

Simplify problem collection

Additional Context

sub-ticket of #1202
@ericvergnaud ericvergnaud added enhancement New feature or request needs-triage labels Apr 25, 2024
@ericvergnaud ericvergnaud added this to UCX Apr 25, 2024
@github-project-automation github-project-automation bot moved this to Triage in UCX Apr 25, 2024
This was referenced Apr 25, 2024
Merged
Closed
nfx added a commit that referenced this issue May 1, 2024
@ericvergnaud @nfx
Use stack of dependency resolvers (#1560)
da71371 
## Changes
Implement resolvers as a stack
Implements an initial version of SysPathProvider
Eliminates horrible hacks introduced in previous PRs 

### Linked issues
#1202
Resolves #1499
Resolves #1421


### Functionality 

- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs ucx ...`
- [ ] added a new workflow
- [ ] modified existing workflow: `...`
- [ ] added a new table
- [ ] modified existing table: `...`

### Tests

- [ ] manually tested
- [x] added unit tests
- [ ] added integration tests
- [ ] verified on staging environment (screenshot attached)

@nfx this PR does not address the tactical `problem_collector`
parameter, I have created a specific issue for that
#1559

---------

Co-authored-by: Serge Smertin <[email protected]>
@nfx nfx moved this from Triage to Quarter Backlog in UCX May 2, 2024
@nfx nfx added tech debt chores and design flaws and removed enhancement New feature or request needs-triage labels May 2, 2024
@nfx nfx self-assigned this May 2, 2024
@nfx nfx mentioned this issue May 3, 2024
Merged
@nfx nfx moved this from Quarter Backlog to Active Backlog in UCX May 6, 2024
@ericvergnaud ericvergnaud mentioned this issue May 6, 2024
Closed
11 tasks
@nfx nfx closed this as completed in #1613 May 7, 2024
nfx added a commit that referenced this issue May 7, 2024
@nfx
Added baseline for workflow linter (#1613)
1ae345a 
```mermaid
flowchart TD
    job -->|has many| job_task
    job_task -.-> notebook_task
    job_task -.-> wheel_task 

    job -.-> git_source

    job_task -.->|execute on| interactive_cluster
    interactive_cluster -.-> library

    job_task -.-> library
    library -.-> wheel_on_dbfs
    library -.-> wheel_on_wsfs
    library -.-> wheel_on_volumes
    library -.-> egg_on_dbfs
    library -.-> egg_on_wsfs
    library -.-> pypi
    wheel_task -.-> wheel_on_dbfs
    wheel_task -.-> wheel_on_wsfs

    wheel_on_dbfs -.-> python_file
    wheel_on_wsfs -.-> python_file
    egg_on_dbfs -.-> python_file
    egg_on_wsfs -.-> python_file
    pypi -.-> python_file
    wsfs_file -.-> python_file
    python_file -.->|import| python_file
    notebook_task -.-> notebook
    notebook -.->|import| python_file
    notebook -.->|can run| notebook

    job_task -.-> dependency_graph
    python_file --> dependency_graph
    notebook --> dependency_graph

    git_source -.-> python_file
    git_source -.-> notebook
    lint_local_code_cli --> dependency_graph

    workflow_linter --> dependency_graph
    workflow_linter -.-> job_problems
    dependency_graph -.-> job_problems
    job_problems -.->|viz| redash_dashboard
```

This PR adds baseline for linting workflows

Related to:
- #1542 
- #1541
- #1540
- #1539
- #1382
- #1204
- #1203
- #1085

closes #1559
closes #1468
closes #1286
@github-project-automation github-project-automation bot moved this from Active Backlog to Archive in UCX May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nfx nfx

@ericvergnaud ericvergnaud

Labels
tech debt chores and design flaws
Projects
UCX
Archived in project
Milestone
No milestone
2 participants
@nfx @ericvergnaud