Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Lateral Column Alias Reference Configurable #539

Closed
maoxingda opened this issue Jan 9, 2024 · 1 comment · Fixed by #540
Closed

Make Lateral Column Alias Reference Configurable #539

maoxingda opened this issue Jan 9, 2024 · 1 comment · Fixed by #540
Labels
bug Something isn't working

Comments

@maoxingda
Copy link
Contributor

SQL

insert into public.tgt_tbl1
(
    name,
    email
)
select
    st1.name,
    st1.name || st1.email || '@gmail.com' as email
from
    public.src_tbl1 as st1

To Reproduce

Note here we refer to SQL provided in prior step as stored in a file named test.sql

from sqllineage.runner import LineageRunner

with open("test.sql") as f:
    sql = f.read()

lr = LineageRunner(sql, dialect="redshift")

lr.print_column_lineage()

Actual behavior

public.tgt_tbl1.email <- public.src_tbl1.email

Expected behavior

public.tgt_tbl1.name <- public.src_tbl1.name
public.tgt_tbl1.name <- public.src_tbl1.email
public.tgt_tbl1.email <- public.src_tbl1.email

Python version (available via python --version)

  • 3.10

SQLLineage version (available via sqllineage --version):

  • 1.5.0

Additional context

@maoxingda maoxingda added the bug Something isn't working label Jan 9, 2024
maoxingda added a commit to maoxingda/sqllineage that referenced this issue Jan 9, 2024
maoxingda added a commit to maoxingda/sqllineage that referenced this issue Jan 9, 2024
maoxingda added a commit to maoxingda/sqllineage that referenced this issue Jan 9, 2024
maoxingda added a commit to maoxingda/sqllineage that referenced this issue Jan 9, 2024
@reata
Copy link
Owner

reata commented Jan 9, 2024

This is the side effect of #507.

It looks like "Lateral Column Alias Reference" is highly dependent on metadata. I'm thinking that maybe we should disable this logic by default and only trigger it if a true metadata_provider is passed in.

maoxingda added a commit to maoxingda/sqllineage that referenced this issue Jan 9, 2024
maoxingda added a commit to maoxingda/sqllineage that referenced this issue Jan 10, 2024
maoxingda added a commit to maoxingda/sqllineage that referenced this issue Jan 10, 2024
maoxingda added a commit to maoxingda/sqllineage that referenced this issue Jan 13, 2024
maoxingda added a commit to maoxingda/sqllineage that referenced this issue Jan 13, 2024
maoxingda added a commit to maoxingda/sqllineage that referenced this issue Jan 13, 2024
maoxingda added a commit to maoxingda/sqllineage that referenced this issue Jan 13, 2024
maoxingda added a commit to maoxingda/sqllineage that referenced this issue Jan 13, 2024
maoxingda added a commit to maoxingda/sqllineage that referenced this issue Jan 13, 2024
maoxingda added a commit to maoxingda/sqllineage that referenced this issue Jan 14, 2024
maoxingda added a commit to maoxingda/sqllineage that referenced this issue Jan 14, 2024
reata pushed a commit to maoxingda/sqllineage that referenced this issue Jan 16, 2024
reata pushed a commit to maoxingda/sqllineage that referenced this issue Jan 21, 2024
reata pushed a commit to maoxingda/sqllineage that referenced this issue Jan 27, 2024
@reata reata changed the title Misidentify column name as alias Make Lateral Column Alias Reference Configurable Jan 27, 2024
reata added a commit that referenced this issue Jan 29, 2024
* fix: misidentify-column-name-as-alias (#539)

* add LATERAL_COLUMN_ALIAS_REFERENCE in SQLLineageConfig

* adjust import order

* add test_column_top_level_enable_lateral_ref_with_metadata_from_nested_subquery

* unknown

* refactor: rebase master and convert LATERAL_COLUMN_ALIAS_REFERENCE to bool type

* refactor: use as few condition as possible: SQLLineageConfig.LATERAL_COLUMN_ALIAS_REFERENCE

* refactor: rebase master and resolve conflict

* refactor: move logic from to_source_columns to end_of_query_cleanup

* refactor: rebase master and fix black format

* docs: LATERAL_COLUMN_ALIAS_REFERENCE how-to guide

* docs: starting version for each config

---------

Co-authored-by: reata <[email protected]>
delphisharp pushed a commit to delphisharp/sqllineage that referenced this issue Feb 26, 2024
* fix: misidentify-column-name-as-alias (reata#539)

* add LATERAL_COLUMN_ALIAS_REFERENCE in SQLLineageConfig

* adjust import order

* add test_column_top_level_enable_lateral_ref_with_metadata_from_nested_subquery

* unknown

* refactor: rebase master and convert LATERAL_COLUMN_ALIAS_REFERENCE to bool type

* refactor: use as few condition as possible: SQLLineageConfig.LATERAL_COLUMN_ALIAS_REFERENCE

* refactor: rebase master and resolve conflict

* refactor: move logic from to_source_columns to end_of_query_cleanup

* refactor: rebase master and fix black format

* docs: LATERAL_COLUMN_ALIAS_REFERENCE how-to guide

* docs: starting version for each config

---------

Co-authored-by: reata <[email protected]>
reata added a commit to delphisharp/sqllineage that referenced this issue Mar 10, 2024
* fix: misidentify-column-name-as-alias (reata#539)

* add LATERAL_COLUMN_ALIAS_REFERENCE in SQLLineageConfig

* adjust import order

* add test_column_top_level_enable_lateral_ref_with_metadata_from_nested_subquery

* unknown

* refactor: rebase master and convert LATERAL_COLUMN_ALIAS_REFERENCE to bool type

* refactor: use as few condition as possible: SQLLineageConfig.LATERAL_COLUMN_ALIAS_REFERENCE

* refactor: rebase master and resolve conflict

* refactor: move logic from to_source_columns to end_of_query_cleanup

* refactor: rebase master and fix black format

* docs: LATERAL_COLUMN_ALIAS_REFERENCE how-to guide

* docs: starting version for each config

---------

Co-authored-by: reata <[email protected]>
reata added a commit that referenced this issue Apr 7, 2024
* fix: Set param from config #545
redo rebase and clean submit

* fix: misidentify column name as lateral alias (#540)

* fix: misidentify-column-name-as-alias (#539)

* add LATERAL_COLUMN_ALIAS_REFERENCE in SQLLineageConfig

* adjust import order

* add test_column_top_level_enable_lateral_ref_with_metadata_from_nested_subquery

* unknown

* refactor: rebase master and convert LATERAL_COLUMN_ALIAS_REFERENCE to bool type

* refactor: use as few condition as possible: SQLLineageConfig.LATERAL_COLUMN_ALIAS_REFERENCE

* refactor: rebase master and resolve conflict

* refactor: move logic from to_source_columns to end_of_query_cleanup

* refactor: rebase master and fix black format

* docs: LATERAL_COLUMN_ALIAS_REFERENCE how-to guide

* docs: starting version for each config

---------

Co-authored-by: reata <[email protected]>

* feat:SQLLineageConfig supports set value and thread safety

* fix: Fix mypy error

* fix: Fix pytest cov

* fix: Fix the scenario of direct assignment without using with. Add the test of multi-process scenario.

* fix: add  SQLLineageConfigLoader set function

* feat: disable setattr for SQLLineageConfig

* feat: make SQLLineageConfig context manager non-reentrant

* feat: disable set unknown config

* feat: access config in parallel

* chore: disable A005 for module name builtin conflict

* refactor: classmethod to staticmethod

---------

Co-authored-by: liuzhou <[email protected]>
Co-authored-by: maoxd <[email protected]>
Co-authored-by: reata <[email protected]>
delphisharp pushed a commit to delphisharp/sqllineage that referenced this issue Apr 15, 2024
* fix: misidentify-column-name-as-alias (reata#539)

* add LATERAL_COLUMN_ALIAS_REFERENCE in SQLLineageConfig

* adjust import order

* add test_column_top_level_enable_lateral_ref_with_metadata_from_nested_subquery

* unknown

* refactor: rebase master and convert LATERAL_COLUMN_ALIAS_REFERENCE to bool type

* refactor: use as few condition as possible: SQLLineageConfig.LATERAL_COLUMN_ALIAS_REFERENCE

* refactor: rebase master and resolve conflict

* refactor: move logic from to_source_columns to end_of_query_cleanup

* refactor: rebase master and fix black format

* docs: LATERAL_COLUMN_ALIAS_REFERENCE how-to guide

* docs: starting version for each config

---------

Co-authored-by: reata <[email protected]>
delphisharp added a commit to delphisharp/sqllineage that referenced this issue Apr 15, 2024
* fix: Set param from config reata#545
redo rebase and clean submit

* fix: misidentify column name as lateral alias (reata#540)

* fix: misidentify-column-name-as-alias (reata#539)

* add LATERAL_COLUMN_ALIAS_REFERENCE in SQLLineageConfig

* adjust import order

* add test_column_top_level_enable_lateral_ref_with_metadata_from_nested_subquery

* unknown

* refactor: rebase master and convert LATERAL_COLUMN_ALIAS_REFERENCE to bool type

* refactor: use as few condition as possible: SQLLineageConfig.LATERAL_COLUMN_ALIAS_REFERENCE

* refactor: rebase master and resolve conflict

* refactor: move logic from to_source_columns to end_of_query_cleanup

* refactor: rebase master and fix black format

* docs: LATERAL_COLUMN_ALIAS_REFERENCE how-to guide

* docs: starting version for each config

---------

Co-authored-by: reata <[email protected]>

* feat:SQLLineageConfig supports set value and thread safety

* fix: Fix mypy error

* fix: Fix pytest cov

* fix: Fix the scenario of direct assignment without using with. Add the test of multi-process scenario.

* fix: add  SQLLineageConfigLoader set function

* feat: disable setattr for SQLLineageConfig

* feat: make SQLLineageConfig context manager non-reentrant

* feat: disable set unknown config

* feat: access config in parallel

* chore: disable A005 for module name builtin conflict

* refactor: classmethod to staticmethod

---------

Co-authored-by: liuzhou <[email protected]>
Co-authored-by: maoxd <[email protected]>
Co-authored-by: reata <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants