fix: misidentify column name as lateral alias (#539)

[maoxingda]

fix #539

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (a93b894) 99.50% compared to head (df79cf0) 99.50%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #540   +/-   ##
=======================================
  Coverage   99.50%   99.50%           
=======================================
  Files          41       41           
  Lines        2208     2223   +15     
=======================================
+ Hits         2197     2212   +15     
  Misses         11       11           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[maoxingda]

    insert into public.tgt_tbl1
    (
        id,
        id_original
    )
    select
        a || b || c || id as id,
        id as id_original        -- # noqa: E501 TODO: I need the metadata information for the table public.src_tbl1 to identify whether the column reference 'id' in this context is from the table public.src_tbl1 or from an alias reference, currently being used as an alias reference. Note: This decision may significantly deviate from the actual scenario.
    from
        public.src_tbl1

I need the help of a pro.😜😜😜

[reata]

See my comment on #539 or email. Maybe we should take a step back and rethink on the approach.

[maoxingda]

select
    a || b || c as c,
    c as d            -- Metadata for a subquery is needed in this context to confirm whether the reference to 'c' is from the subquery or an alias reference.
from
    (
        select
            1 as a,
            2 as b,
            3 as c
    )

The current parsing method has no knowledge of the subquery in this context. Because at this point, the subquery has not yet begun to be parsed.

[maoxingda]

https://github.com/maoxingda/sqllineage/blob/ba0f63097ce06d20bb31e564c25af5e0e8ebfb99/sqllineage/core/parser/sqlfluff/extractors/cte.py#L67

https://github.com/maoxingda/sqllineage/blob/ba0f63097ce06d20bb31e564c25af5e0e8ebfb99/sqllineage/core/parser/sqlfluff/extractors/select.py#L77

https://github.com/maoxingda/sqllineage/blob/ba0f63097ce06d20bb31e564c25af5e0e8ebfb99/sqllineage/core/parser/sqlfluff/extractors/update.py#L76

Move the parsing of the subquery ahead of the for loop, so that the metadata information of the subquery is available. This way, can we determine whether the column references in the SELECT clause come from the subquery or lateral alias references?

[reata]

Given lateral column alias reference is not universally supported, let's not make it default behavior.

[maoxingda]

After #552 merge, I will refactor the configuration of LATERAL_COLUMN_ALIAS_REFERENCE

[maoxingda]

already refactor done, please re-review, thanks. ✅

[maoxingda]

https://www.databricks.com/blog/introducing-support-lateral-column-alias

I don’t know much about databricks but it seems that it also supports lateral column alias (LCA) reference.

[maoxingda]

https://sqlkover.com/cool-stuff-in-snowflake-part-4-aliasing-all-the-things/

It seems that snowflake also supports it, but unfortunately I don’t have the environment to verify it.

[maoxingda]

https://forums.oracle.com/ords/apexds/post/how-to-reference-column-alias-names-in-other-columns-of-a-q-7737

It seems that snowflake also supports it, but unfortunately I don’t find the official document.

[reata]

Popularity is one of the considerations. The major reason I'd like to move this feature as configurable is that it won't function without metadata. And the assumption is that by default sqllineage only does static code analysis and metadata is not present.

[maoxingda]

Agree. This feature is added because it is very pleasant to use.😜😜😜

[reata]

The overall code structure now looks good. I might still want to tweak a thing or two and more importantly add some documents on how to use this config. But you don't need to worry about this PR any more.

Target getting this merged by end of this week. I'll handle it if there's any conflict from master with other PR merged.

[maoxingda]

👌

[reata]

Postpone merging to next week.

Right now to_source_columns does part of the column to table/subquery resolution work. It would be better that we can handle this universally in _build_digraph method of class SQLLineageHolder.

I'm investigating whether we can move all logic to end_of_query_cleanup and not modify to_source_columns.