[ODM] Creating final view with labels

[SophieYu41]

This is the change for label join step 2 - generating final view with features.

• Label table is deduped based on the join key to avoid join duplication

• Label table only contains keys + labels to avoid dup columns on the left

• label columns in final view would have a prefix "label" for easy identification

• Final view generation will run in same "label-join" job but will not be a materialized table

• Generate final view which joins features with labels

• Generate another "latest label" view to show the latest available labels given a ds

• Add underlying table to view properties for source tracking purposes

Next improvement[DONE]:
Use viewProperties to store metadata of underlying feature & label table.

Rebased on #370

End to end tested on real user data. Views generated as expected.

zipline_test_test_label_join_v3 // feature backfill table
 zipline_test_test_label_join_v3_labeled // final joined view
 zipline_test_test_label_join_v3_labeled_latest // latest label view
 zipline_test_test_label_join_v3_labels // label table

@hzding621 @yunfeng-hao @nikhilsimha

[hzding621]

LGTM w/ a few minor suggestions!

Love the unit tests, great job!

[hzding621]

consider other things to test:

• label_ds is unique per ds.
• if no label_ds existed for a ds from the label_table, we still keep the feature data but leave label_ds as null

[SophieYu41]

we still keep the feature data but leave label_ds as null

Based on how we compute the label table & latest view, think this case would only exist in label final labeled_view but not the latest_view since we computed partition info based on label_table, and label_table would always have label_ds even the label columns are null (Here is the fix we had to make sure null label_ds would not happen)

[hzding621]

since we computed partition info based on label_table

label_table may not have all feature_ds in it. for example, if label_offset is set to 90 and 30 for start and end respectively, and we only run label job for the most recent ds, but the feature table can contain many years of data, then for the majority of feature_ds, they don't have corresponding label yet. but in the latest_view we would want to keep them. this is basically the ELSE TRUE case

[SophieYu41]

Ah right. Just to clarify if no label_ds existing, it would be (0 rows) showing up in query.
----+----------+-------------+------------------------------------
(0 rows)

[hzding621]

Label join is a Left outer join between features and labels. Therefore, when labels are not present (because they were never computed for those feature DS), we should just keep all the rows from features and leave the label columns as NULL. So it shouldn't be 0 rows.

[SophieYu41]

Ah right. I was mislead by a bad example the ds actually does not exist on the left and result and end up with 0 rows.

[hzding621]

we can combine this entire logic into labelJoinConf.rowIdentifier.

there is also a place in unit test where you used labelJoinConf.rowIdentifier to compute the expected df but technically it's better to use the combined logic.

[SophieYu41]

Agree. Updated.

Side question for rowId - The whole label table dedup logic and join logic is depending on this row id. The rowId must be accurate so the label join is doing right thing otherwise could be pretty messy. If we are depending user to provide this info, do we have validation somewhere making sure users passed in the correct list?
e.g. what should happen if user's rowId != rowIdentifier we computed here?

[pengyu-hou]

great unit testings!

[SophieYu41]

Merging this change since it's been opening a while, will address comments in a follow-up change if any. cc @nikhilsimha