[CUJ]: Milestones for table migration - Azure

[qziyuan]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

The development of the table migration feature will be structured into several phases. Please review the plan outlined below, which includes three milestones.
Parent issues: #333, #670

Proposed Solution

Milestone 0: single migrate_tables workflow

flowchart TD
    subgraph CLI
    create_table_mapping --> create_catalogs_schemas
    create_uber_principal
    principal_prefix_access --> migrate_credentials
    migrate_credentials --> migrate_locations
    migrate_locations --> create_catalogs_schemas
    end
    
    create_uber_principal --> mt_workflow
    create_table_mapping --> mt_workflow
    create_catalogs_schemas --> mt_workflow
    
    subgraph mt_workflow[workflow: migrate_tables, with description claiming above dependecies]
    migrate_tables_task[migrate_tables]
    end

Loading

Milestone 1: single migrate_tables workflow with database tracking the progress

flowchart TD
    subgraph CLI
    create_table_mapping --> create_catalogs_schemas
    create_uber_principal
    principal_prefix_access --> migrate_credentials
    migrate_credentials --> migrate_locations
    migrate_locations --> create_catalogs_schemas
    end
    
    create_uber_principal --> mt_workflow
    create_table_mapping --> mt_workflow
    create_catalogs_schemas --> mt_workflow
    
    subgraph mt_workflow[workflow: migrate_tables, with description claiming above dependecies]
    migrate_tables_task[migrate_tables]
    end

    migrate_tables_task --> tm_results_table
    subgraph inventory_database
    tm_results_table[table_migration_progress]
    end

    tm_results_table --> tm_results_chart
    subgraph dashboard
    tm_results_chart[table_migration_charts]
    end

Loading

Milestone 2: migrate_tables CLI and dependencies workflows

flowchart TD
    subgraph CLI
    create_table_mapping --> create_catalogs_schemas
    principal_prefix_access --> migrate_credentials
    migrate_credentials --> migrate_locations
    migrate_locations --> create_catalogs_schemas
    create_uber_principal --> mt_cli[migrate_tables]
    create_catalogs_schemas --> mt_cli
    create_table_mapping --> mt_cli
    end

    subgraph principal_prefix_access_wf[workflow: principal_prefix_access]
    principal_prefix_access_task
    end
    principal_prefix_access_wf --1. user review azure_storage_account_info.csv
    2. latest_job_status--> mc_check_dependencies

    subgraph table_mapping_wf[workflow: create_table_mapping]
    create_table_mapping_task
    end
    table_mapping_wf --1. user review mapping.csv
    2. latest_job_status--> mc_check_dependencies


    subgraph mt_dependencies_wf[workflow: migrate_dependencies]
    mc_check_dependencies[check_dependencies] --> migrate_credentials_task
    migrate_credentials_task --> migrate_locations_task
    migrate_locations_task --> create_catalogs_schemas_task
    end
    mt_dependencies_wf --latest_job_status-->mt_check_dependencies
    
    subgraph mt_workflow[workflow: migrate_tables]
    mt_check_dependencies[check_dependencies] --> migrate_tables_task
    migrate_tables_task[migrate_tables]
    end
    migrate_tables_task --> tm_results_table

    subgraph inventory_database
    tm_results_table[table_migration_results]
    end

    tm_results_table --> tm_results_chart
    subgraph dashboard
    tm_results_chart[table_migration_charts]
    end

Loading

Milestone 3: More migration result tables and dashboards

flowchart TD
    subgraph CLI
    create_table_mapping --> create_catalogs_schemas
    principal_prefix_access --> migrate_credentials
    migrate_credentials --> migrate_locations
    migrate_locations --> create_catalogs_schemas
    create_uber_principal --> mt_cli[migrate_tables]
    create_catalogs_schemas --> mt_cli
    create_table_mapping --> mt_cli
    end

    subgraph principal_prefix_access_wf[workflow: principal_prefix_access]
    principal_prefix_access_task
    end
    principal_prefix_access_wf --1. user review azure_storage_account_info.csv
    2. latest_job_status--> mc_check_dependencies

    subgraph table_mapping_wf[workflow: create_table_mapping]
    create_table_mapping_task
    end
    table_mapping_wf --1. user review mapping.csv
    2. latest_job_status--> mc_check_dependencies

    subgraph mt_dependencies_wf[workflow: migrate_dependencies]
    mc_check_dependencies[check_dependencies] --> migrate_credentials_task
    migrate_credentials_task --> migrate_locations_task
    migrate_locations_task --> create_catalogs_schemas_task
    end
    mt_dependencies_wf --latest_job_status-->mt_check_dependencies
    migrate_credentials_task --> mc_results_table
    migrate_locations_task --> ml_results_table
    create_catalogs_schemas_task --> ccs_results_table
    
    subgraph mt_workflow[workflow: migrate_tables]
    mt_check_dependencies[check_dependencies] --> migrate_tables_task
    migrate_tables_task[migrate_tables]
    end
    migrate_tables_task --> tm_results_table

    subgraph inventory_database
    tm_results_table[table_migration_results]
    mc_results_table[credential_migration_results]
    ml_results_table[location_migration_results]
    ccs_results_table[catalog_schema_creation_results]
    end
    inventory_database --> dashboard
    
    subgraph dashboard
    tm_results_chart[table_migration_charts]
    mc_result_chart[credential_migration_chart]
    ml_results_chart[location_migration_chart]
    ccs_results_chart[catalog_schema_creation_chart]
    end

Loading

Additional Context

No response

[qziyuan]

This milestone thought is no longer aligned with our implementation. Close it.