Record column lineage details

[Praveen2112]

Overrides #7354

[kokosing]

Do you have test for CREATE TABLE LIKE where we should see that columns have nice source columns set.

[Praveen2112]

In case of LIKE we don't capture the data right ? We just reuse the name and type.

[skrzypo987]

Skimmed, looks ok

[Praveen2112]

@kokosing , @skrzypo987 AC

[Praveen2112]

@kasiafi Thanks for the review. AC

[Praveen2112]

@kokosing Added tests

[kasiafi]

A couple of comments.

I went through the code. However, I could not make sure if all sites are covered which should expose source columns. The same about the test coverage.

[Praveen2112]

I could not make sure if all sites are covered which should expose source columns. The same about the test coverage.

I guess we could cover all the sites in the incremental way. It is kind of experimental for now. WDYT ?

[Praveen2112]

@kasiafi AC

[kasiafi]

Looks good to me, provided that a more consistent column reporting will be added as a follow-up.

[Praveen2112]

provided that a more consistent column reporting will be added as a follow-up.

Sure thing

[rsaw4]

@Praveen2112 how to view the lineage? what is sample output for a nested create view, ie viewA which selects from viewB which selects from viewC

[wangqinghuan]

Same question with @rsaw4. How to view the lineage details? @Praveen2112

[Praveen2112]

The table level lineage details can be accessed via

QueryCompletedEvent -> QueryMetadata -> List<TableInfo>

what is sample output for a nested create view, ie viewA which selects from viewB which selects from viewC

For this view we get an List of TableInfo for viewB, viewC and tables which are required for viewC but only for viewB we could have TableInfo#directlyReferenced set to true while for viewC and its dependent tables - that flag would be unset (false).

For column level lineage detail we would get it from

QueryCompletedEvent -> QueryOutputMetadata -> OutputColumnMetadata for each column -> List<ColumnDetail>

ColumnDetail will provide us the necessary information about the input columns.

[amalakar]

@Praveen2112 we are trying to leverage this lineage and found that when an expression is being used, source columns are not being captured. For a query like:

create table amalakar.new_query_log_1 as
with queries as
(
select * from 
hive.default.event_presto_query_logged p2
where ds='2021-12-08' and hr=3
)
SELECT
  p1.occurred_at as occurred_at,
  substr(p2.query_id, 1, 10) as new_query_id
  FROM queries p1 
  inner join queries p2
ON p1.query_id=p2.query_id

limit 10

Lineage I am seeing is after light transformation of the lineage we get via the QueryIOMetadata:

{
  "hive.amalakar.new_query_log_1.new_query_id": [],
  "hive.amalakar.new_query_log_1.occurred_at": [
    {
      "columnName": "hive.default.event_presto_query_logged.occurred_at"
    }
  ]
}

The transformation code looks like:

  public Optional<String> getColumnLineage(QueryIOMetadata ioMetadata) {
    Map<String, List<UpstreamColumn>> lineage = new HashMap<>();

    if (ioMetadata.getOutput().isPresent()) {
      QueryOutputMetadata outputMetadata = ioMetadata.getOutput().get();
      if (outputMetadata.getColumns().isPresent()) {
        List<OutputColumnMetadata> outputColumns = outputMetadata.getColumns().get();
        for (OutputColumnMetadata outputColumn : outputColumns) {
          List<UpstreamColumn> upstreamColumns =
              outputColumn.getSourceColumns().stream()
                  .map(col -> new UpstreamColumn(getQualifiedColumnName(col)))
                  .collect(Collectors.toList());
          String outputColumnName =
              String.format(
                  "%s.%s", getQualifiedTableName(outputMetadata), outputColumn.getColumnName());
          lineage.put(outputColumnName, upstreamColumns);
        }
      } 
...
  }

[amalakar]

Created: #10272