Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add distributed merge into metrics #13594

Merged
merged 15 commits into from
Nov 6, 2023
Merged

feat: add distributed merge into metrics #13594

merged 15 commits into from
Nov 6, 2023

Conversation

JackTan25
Copy link
Contributor

@JackTan25 JackTan25 commented Nov 6, 2023 •
edited
Loading

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Summary about this PR

  1. we need to add more metrics for distributed merge into .
  2. fix distributed bug, we need to accumulate row number in every node.
  • Closes #issue

This change is Reviewable

@vercel Vercel
Copy link

vercel bot commented Nov 6, 2023 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
databend ⬜️ Ignored (Inspect) Visit Preview Nov 6, 2023 9:29am

@JackTan25 JackTan25 added the ci-cloud Build docker image for cloud test label Nov 6, 2023
@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Nov 6, 2023
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2023

Docker Image for PR

  • tag: pr-13594-c2ee3a4

note: this image tag is only available for internal use,
please check the internal doc for more details.

@JackTan25 JackTan25 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Nov 6, 2023
@JackTan25 JackTan25 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Nov 6, 2023
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2023

Docker Image for PR

  • tag: pr-13594-b7c576a

note: this image tag is only available for internal use,
please check the internal doc for more details.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2023

Docker Image for PR

  • tag: pr-13594-6e4c3ff

note: this image tag is only available for internal use,
please check the internal doc for more details.

@JackTan25 JackTan25 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Nov 6, 2023
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2023

Docker Image for PR

  • tag: pr-13594-437c490

note: this image tag is only available for internal use,
please check the internal doc for more details.

@JackTan25 JackTan25 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Nov 6, 2023
@JackTan25 JackTan25 force-pushed the add_distributed_metrics branch from 4b94c15 to 4b41d4f Compare November 6, 2023 09:28
@JackTan25 JackTan25 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Nov 6, 2023
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2023

Docker Image for PR

  • tag: pr-13594-6e488f6

note: this image tag is only available for internal use,
please check the internal doc for more details.

@JackTan25 JackTan25 requested a review from dantengsky November 6, 2023 09:45
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2023

Docker Image for PR

  • tag: pr-13594-2d04c94

note: this image tag is only available for internal use,
please check the internal doc for more details.

@JackTan25 JackTan25 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Nov 6, 2023
@JackTan25 JackTan25 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Nov 6, 2023
@JackTan25 JackTan25 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Nov 6, 2023
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2023

Docker Image for PR

  • tag: pr-13594-33023d0

note: this image tag is only available for internal use,
please check the internal doc for more details.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2023

Docker Image for PR

  • tag: pr-13594-aff8f05

note: this image tag is only available for internal use,
please check the internal doc for more details.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 6, 2023

Docker Image for PR

  • tag: pr-13594-90f2acd

note: this image tag is only available for internal use,
please check the internal doc for more details.

@JackTan25 JackTan25 marked this pull request as ready for review November 6, 2023 11:49
@SkyFan2002
Copy link
Member

Tested in cluster mode for 3000 times using: https://github.com/JackTan25/test-scripts/tree/test_delete

[2023-11-06T13:57:50Z INFO  test_replace_recluster] ==========================
[2023-11-06T13:57:50Z INFO  test_replace_recluster] ====verify table state====
[2023-11-06T13:57:50Z INFO  test_replace_recluster] ==========================
[2023-11-06T13:57:50Z INFO  test_replace_recluster]                            
[2023-11-06T13:57:50Z INFO  test_replace_recluster]                            
[2023-11-06T13:57:50Z INFO  test_replace_recluster] number of successfully executed merge-into statements : 3000
[2023-11-06T13:57:50Z INFO  test_replace_recluster]                            
[2023-11-06T13:57:50Z INFO  test_replace_recluster]                            
[2023-11-06T13:57:50Z INFO  test_replace_recluster] CHECK: value of successfully executed merge-into statements
[2023-11-06T13:57:50Z INFO  test_replace_recluster] CHECK: value of successfully executed merge-into statements: client 3000000, server 0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] CHECK: full table scanning
[2023-11-06T13:57:50Z INFO  test_replace_recluster] ===========================
[2023-11-06T13:57:50Z INFO  test_replace_recluster] ======     PASSED      ====
[2023-11-06T13:57:50Z INFO  test_replace_recluster] ===========================
[2023-11-06T13:57:50Z INFO  test_replace_recluster]                            
[2023-11-06T13:57:50Z INFO  test_replace_recluster]                            
[2023-11-06T13:57:50Z INFO  test_replace_recluster] ========METRICS============
[2023-11-06T13:57:50Z INFO  test_replace_recluster] fuse_remote_io_read_bytes_after_merged_total : 300183849.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] fuse_remote_io_seeks_after_merged_total : 3000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds : [{"less_than":10.0,"count":3000.0},{"less_than":50.0,"count":3000.0},{"less_than":100.0,"count":3000.0},{"less_than":250.0,"count":3000.0},{"less_than":500.0,"count":3000.0},{"less_than":1000.0,"count":3000.0},{"less_than":2500.0,"count":3000.0},{"less_than":5000.0,"count":3000.0},{"less_than":10000.0,"count":3000.0},{"less_than":20000.0,"count":3000.0},{"less_than":30000.0,"count":3000.0},{"less_than":60000.0,"count":3000.0},{"less_than":300000.0,"count":3000.0},{"less_than":600000.0,"count":3000.0},{"less_than":1800000.0,"count":3000.0},{"less_than":null,"count":3000.0}]
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_count : 3000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_sum : 6.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_append_blocks_counter_total : 3000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_append_blocks_rows_counter_total : 3000000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds : [{"less_than":10.0,"count":6000.0},{"less_than":50.0,"count":6000.0},{"less_than":100.0,"count":6000.0},{"less_than":250.0,"count":6000.0},{"less_than":500.0,"count":6000.0},{"less_than":1000.0,"count":6000.0},{"less_than":2500.0,"count":6000.0},{"less_than":5000.0,"count":6000.0},{"less_than":10000.0,"count":6000.0},{"less_than":20000.0,"count":6000.0},{"less_than":30000.0,"count":6000.0},{"less_than":60000.0,"count":6000.0},{"less_than":300000.0,"count":6000.0},{"less_than":600000.0,"count":6000.0},{"less_than":1800000.0,"count":6000.0},{"less_than":null,"count":6000.0}]
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds : [{"less_than":10.0,"count":6000.0},{"less_than":50.0,"count":6000.0},{"less_than":100.0,"count":6000.0},{"less_than":250.0,"count":6000.0},{"less_than":500.0,"count":6000.0},{"less_than":1000.0,"count":6000.0},{"less_than":2500.0,"count":6000.0},{"less_than":5000.0,"count":6000.0},{"less_than":10000.0,"count":6000.0},{"less_than":20000.0,"count":6000.0},{"less_than":30000.0,"count":6000.0},{"less_than":60000.0,"count":6000.0},{"less_than":300000.0,"count":6000.0},{"less_than":600000.0,"count":6000.0},{"less_than":1800000.0,"count":6000.0},{"less_than":null,"count":6000.0}]
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds : [{"less_than":10.0,"count":5999.0},{"less_than":50.0,"count":6000.0},{"less_than":100.0,"count":6000.0},{"less_than":250.0,"count":6000.0},{"less_than":500.0,"count":6000.0},{"less_than":1000.0,"count":6000.0},{"less_than":2500.0,"count":6000.0},{"less_than":5000.0,"count":6000.0},{"less_than":10000.0,"count":6000.0},{"less_than":20000.0,"count":6000.0},{"less_than":30000.0,"count":6000.0},{"less_than":60000.0,"count":6000.0},{"less_than":300000.0,"count":6000.0},{"less_than":600000.0,"count":6000.0},{"less_than":1800000.0,"count":6000.0},{"less_than":null,"count":6000.0}]
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_count : 6000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_count : 6000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_count : 6000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_sum : 0.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_sum : 0.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_sum : 10611.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_counter_total : 3000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_rows_counter_total : 3000000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_distributed_apply_row_number_total : 3000000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_distributed_deduplicate_row_number_total : 15000000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_distributed_empty_row_number_total : 3000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_distributed_generate_row_numbers_total : 6000000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_distributed_hashtable_empty_block_total : 3000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_distributed_hashtable_fetch_row_number_total : 3000000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_distributed_hashtable_push_null_block_rows_total : 3000000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_distributed_hashtable_push_null_block_total : 3000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_distributed_init_unique_number_total : 3009000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_distributed_new_set_len_total : 6006000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_matched_rows_total : 3000000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds : [{"less_than":10.0,"count":3000.0},{"less_than":50.0,"count":3000.0},{"less_than":100.0,"count":3000.0},{"less_than":250.0,"count":3000.0},{"less_than":500.0,"count":3000.0},{"less_than":1000.0,"count":3000.0},{"less_than":2500.0,"count":3000.0},{"less_than":5000.0,"count":3000.0},{"less_than":10000.0,"count":3000.0},{"less_than":20000.0,"count":3000.0},{"less_than":30000.0,"count":3000.0},{"less_than":60000.0,"count":3000.0},{"less_than":300000.0,"count":3000.0},{"less_than":600000.0,"count":3000.0},{"less_than":1800000.0,"count":3000.0},{"less_than":null,"count":3000.0}]
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds_count : 3000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds_sum : 0.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_split_milliseconds : [{"less_than":10.0,"count":6000.0},{"less_than":50.0,"count":6000.0},{"less_than":100.0,"count":6000.0},{"less_than":250.0,"count":6000.0},{"less_than":500.0,"count":6000.0},{"less_than":1000.0,"count":6000.0},{"less_than":2500.0,"count":6000.0},{"less_than":5000.0,"count":6000.0},{"less_than":10000.0,"count":6000.0},{"less_than":20000.0,"count":6000.0},{"less_than":30000.0,"count":6000.0},{"less_than":60000.0,"count":6000.0},{"less_than":300000.0,"count":6000.0},{"less_than":600000.0,"count":6000.0},{"less_than":1800000.0,"count":6000.0},{"less_than":null,"count":6000.0}]
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_split_milliseconds : [{"less_than":10.0,"count":6000.0},{"less_than":50.0,"count":6000.0},{"less_than":100.0,"count":6000.0},{"less_than":250.0,"count":6000.0},{"less_than":500.0,"count":6000.0},{"less_than":1000.0,"count":6000.0},{"less_than":2500.0,"count":6000.0},{"less_than":5000.0,"count":6000.0},{"less_than":10000.0,"count":6000.0},{"less_than":20000.0,"count":6000.0},{"less_than":30000.0,"count":6000.0},{"less_than":60000.0,"count":6000.0},{"less_than":300000.0,"count":6000.0},{"less_than":600000.0,"count":6000.0},{"less_than":1800000.0,"count":6000.0},{"less_than":null,"count":6000.0}]
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_split_milliseconds : [{"less_than":10.0,"count":6000.0},{"less_than":50.0,"count":6000.0},{"less_than":100.0,"count":6000.0},{"less_than":250.0,"count":6000.0},{"less_than":500.0,"count":6000.0},{"less_than":1000.0,"count":6000.0},{"less_than":2500.0,"count":6000.0},{"less_than":5000.0,"count":6000.0},{"less_than":10000.0,"count":6000.0},{"less_than":20000.0,"count":6000.0},{"less_than":30000.0,"count":6000.0},{"less_than":60000.0,"count":6000.0},{"less_than":300000.0,"count":6000.0},{"less_than":600000.0,"count":6000.0},{"less_than":1800000.0,"count":6000.0},{"less_than":null,"count":6000.0}]
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_count : 6000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_count : 6000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_count : 6000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_sum : 0.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_sum : 0.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_sum : 0.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_unmatched_rows_total : 3000000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_unmatched_rows_total : 6000000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] query_merge_into_unmatched_rows_total : 6000000.0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] ===========================
[2023-11-06T13:57:50Z INFO  test_replace_recluster]                            
[2023-11-06T13:57:50Z INFO  test_replace_recluster]                            
[2023-11-06T13:57:50Z INFO  test_replace_recluster] ======CLUSTERING INFO======
[2023-11-06T13:57:50Z INFO  test_replace_recluster] cluster_key : (to_yyyymmdd(insert_time), id)
[2023-11-06T13:57:50Z INFO  test_replace_recluster] block_count: 0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] constant_block_count: 0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] unclustered_block_count: 0
[2023-11-06T13:57:50Z INFO  test_replace_recluster] average_overlaps: NaN
[2023-11-06T13:57:50Z INFO  test_replace_recluster] average_depth: NaN
[2023-11-06T13:57:50Z INFO  test_replace_recluster] block_depth_histogram: {}
[2023-11-06T13:57:50Z INFO  test_replace_recluster] ===========================

@dantengsky
Copy link
Member

dantengsky commented Nov 6, 2023 •
edited
Loading

Tested in cluster mode for 3000 times using
https://github.com/datafuse-extras/test-scripts/tree/test-merge-into


[2023-11-06T14:46:02Z INFO  test_replace_recluster] ==========================
[2023-11-06T14:46:02Z INFO  test_replace_recluster] ====verify table state====
[2023-11-06T14:46:02Z INFO  test_replace_recluster] ==========================
[2023-11-06T14:46:02Z INFO  test_replace_recluster]
[2023-11-06T14:46:02Z INFO  test_replace_recluster]
[2023-11-06T14:46:02Z INFO  test_replace_recluster] number of successfully executed merge-into statements : 3000
[2023-11-06T14:46:02Z INFO  test_replace_recluster]
[2023-11-06T14:46:02Z INFO  test_replace_recluster]
[2023-11-06T14:46:02Z INFO  test_replace_recluster] CHECK: value of successfully executed merge-into statements
[2023-11-06T14:46:02Z INFO  test_replace_recluster] CHECK: value of successfully executed merge-into statements: client 3000000, server 3000000
[2023-11-06T14:46:02Z INFO  test_replace_recluster] CHECK: distinct ids: client 3000, server 3000
[2023-11-06T14:46:02Z INFO  test_replace_recluster] CHECK: value of correlated column
[2023-11-06T14:46:02Z INFO  test_replace_recluster] CHECK: full table scanning
[2023-11-06T14:46:02Z INFO  test_replace_recluster] ===========================
[2023-11-06T14:46:02Z INFO  test_replace_recluster] ======     PASSED      ====
[2023-11-06T14:46:02Z INFO  test_replace_recluster] ===========================
[2023-11-06T14:46:02Z INFO  test_replace_recluster]
[2023-11-06T14:46:02Z INFO  test_replace_recluster]
[2023-11-06T14:46:02Z INFO  test_replace_recluster] ========METRICS============
[2023-11-06T14:46:03Z INFO  test_replace_recluster] fuse_commit_mutation_unresolvable_conflict_total : 428.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] fuse_remote_io_read_bytes_after_merged_total : 27604022581.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] fuse_remote_io_read_bytes_after_merged_total : 6505972568.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] fuse_remote_io_read_bytes_after_merged_total : 10314424028.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] fuse_remote_io_seeks_after_merged_total : 25454.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] fuse_remote_io_seeks_after_merged_total : 6297.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] fuse_remote_io_seeks_after_merged_total : 10582.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds : [{"less_than":10.0,"count":4264.0},{"less_than":50.0,"count":4264.0},{"less_than":100.0,"count":4264.0},{"less_than":250.0,"count":4264.0},{"less_than":500.0,"count":4264.0},{"less_than":1000.0,"count":4264.0},{"less_than":2500.0,"count":4264.0},{"less_than":5000.0,"count":4264.0},{"less_than":10000.0,"count":4264.0},{"less_than":20000.0,"count":4264.0},{"less_than":30000.0,"count":4264.0},{"less_than":60000.0,"count":4264.0},{"less_than":300000.0,"count":4264.0},{"less_than":600000.0,"count":4264.0},{"less_than":1800000.0,"count":4264.0},{"less_than":null,"count":4264.0}]
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds : [{"less_than":10.0,"count":1069.0},{"less_than":50.0,"count":1069.0},{"less_than":100.0,"count":1069.0},{"less_than":250.0,"count":1069.0},{"less_than":500.0,"count":1069.0},{"less_than":1000.0,"count":1069.0},{"less_than":2500.0,"count":1069.0},{"less_than":5000.0,"count":1069.0},{"less_than":10000.0,"count":1069.0},{"less_than":20000.0,"count":1069.0},{"less_than":30000.0,"count":1069.0},{"less_than":60000.0,"count":1069.0},{"less_than":300000.0,"count":1069.0},{"less_than":600000.0,"count":1069.0},{"less_than":1800000.0,"count":1069.0},{"less_than":null,"count":1069.0}]
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds : [{"less_than":10.0,"count":2018.0},{"less_than":50.0,"count":2018.0},{"less_than":100.0,"count":2018.0},{"less_than":250.0,"count":2018.0},{"less_than":500.0,"count":2018.0},{"less_than":1000.0,"count":2018.0},{"less_than":2500.0,"count":2018.0},{"less_than":5000.0,"count":2018.0},{"less_than":10000.0,"count":2018.0},{"less_than":20000.0,"count":2018.0},{"less_than":30000.0,"count":2018.0},{"less_than":60000.0,"count":2018.0},{"less_than":300000.0,"count":2018.0},{"less_than":600000.0,"count":2018.0},{"less_than":1800000.0,"count":2018.0},{"less_than":null,"count":2018.0}]
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_count : 4264.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_count : 1069.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_count : 2018.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_sum : 0.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_sum : 0.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_sum : 0.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_append_blocks_counter_total : 4264.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_append_blocks_counter_total : 1069.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_append_blocks_counter_total : 5018.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_append_blocks_rows_counter_total : 433697.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_append_blocks_rows_counter_total : 211176.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_append_blocks_rows_counter_total : 3639127.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds : [{"less_than":10.0,"count":127.0},{"less_than":50.0,"count":128.0},{"less_than":100.0,"count":128.0},{"less_than":250.0,"count":128.0},{"less_than":500.0,"count":128.0},{"less_than":1000.0,"count":128.0},{"less_than":2500.0,"count":331.0},{"less_than":5000.0,"count":426.0},{"less_than":10000.0,"count":428.0},{"less_than":20000.0,"count":428.0},{"less_than":30000.0,"count":428.0},{"less_than":60000.0,"count":428.0},{"less_than":300000.0,"count":428.0},{"less_than":600000.0,"count":428.0},{"less_than":1800000.0,"count":428.0},{"less_than":null,"count":428.0}]
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds : [{"less_than":10.0,"count":291.0},{"less_than":50.0,"count":292.0},{"less_than":100.0,"count":293.0},{"less_than":250.0,"count":302.0},{"less_than":500.0,"count":305.0},{"less_than":1000.0,"count":319.0},{"less_than":2500.0,"count":413.0},{"less_than":5000.0,"count":428.0},{"less_than":10000.0,"count":428.0},{"less_than":20000.0,"count":428.0},{"less_than":30000.0,"count":428.0},{"less_than":60000.0,"count":428.0},{"less_than":300000.0,"count":428.0},{"less_than":600000.0,"count":428.0},{"less_than":1800000.0,"count":428.0},{"less_than":null,"count":428.0}]
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds : [{"less_than":10.0,"count":3155.0},{"less_than":50.0,"count":3158.0},{"less_than":100.0,"count":3159.0},{"less_than":250.0,"count":3160.0},{"less_than":500.0,"count":3169.0},{"less_than":1000.0,"count":3177.0},{"less_than":2500.0,"count":3360.0},{"less_than":5000.0,"count":3427.0},{"less_than":10000.0,"count":3428.0},{"less_than":20000.0,"count":3428.0},{"less_than":30000.0,"count":3428.0},{"less_than":60000.0,"count":3428.0},{"less_than":300000.0,"count":3428.0},{"less_than":600000.0,"count":3428.0},{"less_than":1800000.0,"count":3428.0},{"less_than":null,"count":3428.0}]
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_count : 428.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_count : 428.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_count : 3428.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_sum : 714161.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_sum : 235422.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_sum : 583077.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_counter_total : 20.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_counter_total : 22.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_counter_total : 386.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_rows_counter_total : 20000.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_rows_counter_total : 22000.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_rows_counter_total : 386000.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_distributed_deduplicate_row_number_total : 2568000.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_distributed_empty_row_number_total : 72.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_distributed_generate_row_numbers_total : 433697.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_distributed_generate_row_numbers_total : 211176.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_distributed_generate_row_numbers_total : 639127.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_distributed_hashtable_empty_block_total : 428.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_distributed_init_unique_number_total : 1122321.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_distributed_new_set_len_total : 758032.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds : [{"less_than":10.0,"count":4264.0},{"less_than":50.0,"count":4264.0},{"less_than":100.0,"count":4264.0},{"less_than":250.0,"count":4264.0},{"less_than":500.0,"count":4264.0},{"less_than":1000.0,"count":4264.0},{"less_than":2500.0,"count":4264.0},{"less_than":5000.0,"count":4264.0},{"less_than":10000.0,"count":4264.0},{"less_than":20000.0,"count":4264.0},{"less_than":30000.0,"count":4264.0},{"less_than":60000.0,"count":4264.0},{"less_than":300000.0,"count":4264.0},{"less_than":600000.0,"count":4264.0},{"less_than":1800000.0,"count":4264.0},{"less_than":null,"count":4264.0}]
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds : [{"less_than":10.0,"count":1069.0},{"less_than":50.0,"count":1069.0},{"less_than":100.0,"count":1069.0},{"less_than":250.0,"count":1069.0},{"less_than":500.0,"count":1069.0},{"less_than":1000.0,"count":1069.0},{"less_than":2500.0,"count":1069.0},{"less_than":5000.0,"count":1069.0},{"less_than":10000.0,"count":1069.0},{"less_than":20000.0,"count":1069.0},{"less_than":30000.0,"count":1069.0},{"less_than":60000.0,"count":1069.0},{"less_than":300000.0,"count":1069.0},{"less_than":600000.0,"count":1069.0},{"less_than":1800000.0,"count":1069.0},{"less_than":null,"count":1069.0}]
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds : [{"less_than":10.0,"count":2018.0},{"less_than":50.0,"count":2018.0},{"less_than":100.0,"count":2018.0},{"less_than":250.0,"count":2018.0},{"less_than":500.0,"count":2018.0},{"less_than":1000.0,"count":2018.0},{"less_than":2500.0,"count":2018.0},{"less_than":5000.0,"count":2018.0},{"less_than":10000.0,"count":2018.0},{"less_than":20000.0,"count":2018.0},{"less_than":30000.0,"count":2018.0},{"less_than":60000.0,"count":2018.0},{"less_than":300000.0,"count":2018.0},{"less_than":600000.0,"count":2018.0},{"less_than":1800000.0,"count":2018.0},{"less_than":null,"count":2018.0}]
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_count : 4264.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_count : 1069.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_count : 2018.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_sum : 190.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_sum : 343.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_sum : 1605.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_matched_rows_total : 433697.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_matched_rows_total : 211176.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_matched_rows_total : 639127.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds : [{"less_than":10.0,"count":3000.0},{"less_than":50.0,"count":3000.0},{"less_than":100.0,"count":3000.0},{"less_than":250.0,"count":3000.0},{"less_than":500.0,"count":3000.0},{"less_than":1000.0,"count":3000.0},{"less_than":2500.0,"count":3000.0},{"less_than":5000.0,"count":3000.0},{"less_than":10000.0,"count":3000.0},{"less_than":20000.0,"count":3000.0},{"less_than":30000.0,"count":3000.0},{"less_than":60000.0,"count":3000.0},{"less_than":300000.0,"count":3000.0},{"less_than":600000.0,"count":3000.0},{"less_than":1800000.0,"count":3000.0},{"less_than":null,"count":3000.0}]
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds_count : 3000.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds_sum : 0.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_replace_blocks_counter_total : 708.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_replace_blocks_counter_total : 197.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_replace_blocks_counter_total : 282.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_replace_blocks_rows_counter_total : 272591712.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_replace_blocks_rows_counter_total : 64335091.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_replace_blocks_rows_counter_total : 101027197.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_split_milliseconds : [{"less_than":10.0,"count":4680.0},{"less_than":50.0,"count":4680.0},{"less_than":100.0,"count":4680.0},{"less_than":250.0,"count":4680.0},{"less_than":500.0,"count":4680.0},{"less_than":1000.0,"count":4680.0},{"less_than":2500.0,"count":4680.0},{"less_than":5000.0,"count":4680.0},{"less_than":10000.0,"count":4680.0},{"less_than":20000.0,"count":4680.0},{"less_than":30000.0,"count":4680.0},{"less_than":60000.0,"count":4680.0},{"less_than":300000.0,"count":4680.0},{"less_than":600000.0,"count":4680.0},{"less_than":1800000.0,"count":4680.0},{"less_than":null,"count":4680.0}]
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_split_milliseconds : [{"less_than":10.0,"count":1490.0},{"less_than":50.0,"count":1490.0},{"less_than":100.0,"count":1490.0},{"less_than":250.0,"count":1490.0},{"less_than":500.0,"count":1490.0},{"less_than":1000.0,"count":1490.0},{"less_than":2500.0,"count":1490.0},{"less_than":5000.0,"count":1490.0},{"less_than":10000.0,"count":1490.0},{"less_than":20000.0,"count":1490.0},{"less_than":30000.0,"count":1490.0},{"less_than":60000.0,"count":1490.0},{"less_than":300000.0,"count":1490.0},{"less_than":600000.0,"count":1490.0},{"less_than":1800000.0,"count":1490.0},{"less_than":null,"count":1490.0}]
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_split_milliseconds : [{"less_than":10.0,"count":5393.0},{"less_than":50.0,"count":5393.0},{"less_than":100.0,"count":5393.0},{"less_than":250.0,"count":5393.0},{"less_than":500.0,"count":5393.0},{"less_than":1000.0,"count":5393.0},{"less_than":2500.0,"count":5393.0},{"less_than":5000.0,"count":5393.0},{"less_than":10000.0,"count":5393.0},{"less_than":20000.0,"count":5393.0},{"less_than":30000.0,"count":5393.0},{"less_than":60000.0,"count":5393.0},{"less_than":300000.0,"count":5393.0},{"less_than":600000.0,"count":5393.0},{"less_than":1800000.0,"count":5393.0},{"less_than":null,"count":5393.0}]
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_count : 4680.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_count : 1490.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_count : 5393.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_sum : 0.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_sum : 2.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_sum : 0.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_unmatched_rows_total : 3644873.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_unmatched_rows_total : 850303.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] query_merge_into_unmatched_rows_total : 1072824.0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] ===========================
[2023-11-06T14:46:03Z INFO  test_replace_recluster]
[2023-11-06T14:46:03Z INFO  test_replace_recluster]
[2023-11-06T14:46:03Z INFO  test_replace_recluster] ======CLUSTERING INFO======
[2023-11-06T14:46:03Z INFO  test_replace_recluster] cluster_key : (to_yyyymmdd(insert_time), id)
[2023-11-06T14:46:03Z INFO  test_replace_recluster] block_count: 11
[2023-11-06T14:46:03Z INFO  test_replace_recluster] constant_block_count: 0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] unclustered_block_count: 0
[2023-11-06T14:46:03Z INFO  test_replace_recluster] average_overlaps: 8.1818
[2023-11-06T14:46:03Z INFO  test_replace_recluster] average_depth: 7
[2023-11-06T14:46:03Z INFO  test_replace_recluster] block_depth_histogram: {"00007":11}
[2023-11-06T14:46:03Z INFO  test_replace_recluster] ===========================

@SkyFan2002
Copy link
Member

Tested in cluster mode for 3000 times using https://github.com/JackTan25/test-scripts/tree/distributed_test

[2023-11-06T16:20:43Z INFO  test_replace_recluster] ==========================
[2023-11-06T16:20:43Z INFO  test_replace_recluster] ====verify table state====
[2023-11-06T16:20:43Z INFO  test_replace_recluster] ==========================
[2023-11-06T16:20:43Z INFO  test_replace_recluster]                            
[2023-11-06T16:20:43Z INFO  test_replace_recluster]                            
[2023-11-06T16:20:43Z INFO  test_replace_recluster] number of successfully executed merge-into statements : 3000
[2023-11-06T16:20:43Z INFO  test_replace_recluster]                            
[2023-11-06T16:20:43Z INFO  test_replace_recluster]                            
[2023-11-06T16:20:43Z INFO  test_replace_recluster] CHECK: value of successfully executed merge-into statements
[2023-11-06T16:20:43Z INFO  test_replace_recluster] CHECK: value of successfully executed merge-into statements: client 3000000, server 3000000
[2023-11-06T16:20:43Z INFO  test_replace_recluster] CHECK: distinct ids: client 3000, server 3000
[2023-11-06T16:20:43Z INFO  test_replace_recluster] CHECK: value of correlated column
[2023-11-06T16:20:43Z INFO  test_replace_recluster] CHECK: full table scanning
[2023-11-06T16:20:44Z INFO  test_replace_recluster] ===========================
[2023-11-06T16:20:44Z INFO  test_replace_recluster] ======     PASSED      ====
[2023-11-06T16:20:44Z INFO  test_replace_recluster] ===========================
[2023-11-06T16:20:44Z INFO  test_replace_recluster]                            
[2023-11-06T16:20:44Z INFO  test_replace_recluster]                            
[2023-11-06T16:20:44Z INFO  test_replace_recluster] ========METRICS============
[2023-11-06T16:20:44Z INFO  test_replace_recluster] fuse_commit_mutation_unresolvable_conflict_total : 428.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] fuse_remote_io_read_bytes_after_merged_total : 4771122194.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] fuse_remote_io_read_bytes_after_merged_total : 14792406067.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] fuse_remote_io_read_bytes_after_merged_total : 29590158248.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] fuse_remote_io_seeks_after_merged_total : 4355.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] fuse_remote_io_seeks_after_merged_total : 13209.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] fuse_remote_io_seeks_after_merged_total : 28475.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds : [{"less_than":10.0,"count":787.0},{"less_than":50.0,"count":787.0},{"less_than":100.0,"count":787.0},{"less_than":250.0,"count":787.0},{"less_than":500.0,"count":787.0},{"less_than":1000.0,"count":787.0},{"less_than":2500.0,"count":787.0},{"less_than":5000.0,"count":787.0},{"less_than":10000.0,"count":787.0},{"less_than":20000.0,"count":787.0},{"less_than":30000.0,"count":787.0},{"less_than":60000.0,"count":787.0},{"less_than":300000.0,"count":787.0},{"less_than":600000.0,"count":787.0},{"less_than":1800000.0,"count":787.0},{"less_than":null,"count":787.0}]
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds : [{"less_than":10.0,"count":2798.0},{"less_than":50.0,"count":2798.0},{"less_than":100.0,"count":2798.0},{"less_than":250.0,"count":2798.0},{"less_than":500.0,"count":2798.0},{"less_than":1000.0,"count":2798.0},{"less_than":2500.0,"count":2798.0},{"less_than":5000.0,"count":2798.0},{"less_than":10000.0,"count":2798.0},{"less_than":20000.0,"count":2798.0},{"less_than":30000.0,"count":2798.0},{"less_than":60000.0,"count":2798.0},{"less_than":300000.0,"count":2798.0},{"less_than":600000.0,"count":2798.0},{"less_than":1800000.0,"count":2798.0},{"less_than":null,"count":2798.0}]
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds : [{"less_than":10.0,"count":4679.0},{"less_than":50.0,"count":4679.0},{"less_than":100.0,"count":4679.0},{"less_than":250.0,"count":4679.0},{"less_than":500.0,"count":4679.0},{"less_than":1000.0,"count":4679.0},{"less_than":2500.0,"count":4679.0},{"less_than":5000.0,"count":4679.0},{"less_than":10000.0,"count":4679.0},{"less_than":20000.0,"count":4679.0},{"less_than":30000.0,"count":4679.0},{"less_than":60000.0,"count":4679.0},{"less_than":300000.0,"count":4679.0},{"less_than":600000.0,"count":4679.0},{"less_than":1800000.0,"count":4679.0},{"less_than":null,"count":4679.0}]
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_count : 787.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_count : 2798.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_count : 4679.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_sum : 0.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_sum : 14.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_accumulate_milliseconds_sum : 0.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_append_blocks_counter_total : 787.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_append_blocks_counter_total : 2798.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_append_blocks_counter_total : 7679.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_append_blocks_rows_counter_total : 138101.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_append_blocks_rows_counter_total : 769193.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_append_blocks_rows_counter_total : 3376706.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds : [{"less_than":10.0,"count":3295.0},{"less_than":50.0,"count":3296.0},{"less_than":100.0,"count":3298.0},{"less_than":250.0,"count":3303.0},{"less_than":500.0,"count":3310.0},{"less_than":1000.0,"count":3318.0},{"less_than":2500.0,"count":3428.0},{"less_than":5000.0,"count":3428.0},{"less_than":10000.0,"count":3428.0},{"less_than":20000.0,"count":3428.0},{"less_than":30000.0,"count":3428.0},{"less_than":60000.0,"count":3428.0},{"less_than":300000.0,"count":3428.0},{"less_than":600000.0,"count":3428.0},{"less_than":1800000.0,"count":3428.0},{"less_than":null,"count":3428.0}]
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds : [{"less_than":10.0,"count":3062.0},{"less_than":50.0,"count":3063.0},{"less_than":100.0,"count":3063.0},{"less_than":250.0,"count":3063.0},{"less_than":500.0,"count":3063.0},{"less_than":1000.0,"count":3076.0},{"less_than":2500.0,"count":3428.0},{"less_than":5000.0,"count":3428.0},{"less_than":10000.0,"count":3428.0},{"less_than":20000.0,"count":3428.0},{"less_than":30000.0,"count":3428.0},{"less_than":60000.0,"count":3428.0},{"less_than":300000.0,"count":3428.0},{"less_than":600000.0,"count":3428.0},{"less_than":1800000.0,"count":3428.0},{"less_than":null,"count":3428.0}]
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds : [{"less_than":10.0,"count":3156.0},{"less_than":50.0,"count":3157.0},{"less_than":100.0,"count":3158.0},{"less_than":250.0,"count":3159.0},{"less_than":500.0,"count":3163.0},{"less_than":1000.0,"count":3163.0},{"less_than":2500.0,"count":3428.0},{"less_than":5000.0,"count":3428.0},{"less_than":10000.0,"count":3428.0},{"less_than":20000.0,"count":3428.0},{"less_than":30000.0,"count":3428.0},{"less_than":60000.0,"count":3428.0},{"less_than":300000.0,"count":3428.0},{"less_than":600000.0,"count":3428.0},{"less_than":1800000.0,"count":3428.0},{"less_than":null,"count":3428.0}]
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_count : 3428.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_count : 3428.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_count : 3428.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_sum : 176681.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_sum : 555599.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_apply_milliseconds_sum : 396298.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_counter_total : 2.0
[2023-11-06T16:20:44Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_counter_total : 426.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_counter_total : 0.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_rows_counter_total : 2000.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_rows_counter_total : 426000.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_deleted_blocks_rows_counter_total : 0.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_distributed_apply_row_number_total : 3000000.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_distributed_deduplicate_row_number_total : 11568000.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_distributed_empty_row_number_total : 102.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_distributed_generate_row_numbers_total : 138101.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_distributed_generate_row_numbers_total : 769193.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_distributed_generate_row_numbers_total : 3376706.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_distributed_hashtable_empty_block_total : 428.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_distributed_hashtable_fetch_row_number_total : 3000000.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_distributed_hashtable_push_null_block_rows_total : 3000000.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_distributed_hashtable_push_null_block_total : 3000.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_distributed_init_unique_number_total : 4132388.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_distributed_new_set_len_total : 6780232.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds : [{"less_than":10.0,"count":787.0},{"less_than":50.0,"count":787.0},{"less_than":100.0,"count":787.0},{"less_than":250.0,"count":787.0},{"less_than":500.0,"count":787.0},{"less_than":1000.0,"count":787.0},{"less_than":2500.0,"count":787.0},{"less_than":5000.0,"count":787.0},{"less_than":10000.0,"count":787.0},{"less_than":20000.0,"count":787.0},{"less_than":30000.0,"count":787.0},{"less_than":60000.0,"count":787.0},{"less_than":300000.0,"count":787.0},{"less_than":600000.0,"count":787.0},{"less_than":1800000.0,"count":787.0},{"less_than":null,"count":787.0}]
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds : [{"less_than":10.0,"count":2798.0},{"less_than":50.0,"count":2798.0},{"less_than":100.0,"count":2798.0},{"less_than":250.0,"count":2798.0},{"less_than":500.0,"count":2798.0},{"less_than":1000.0,"count":2798.0},{"less_than":2500.0,"count":2798.0},{"less_than":5000.0,"count":2798.0},{"less_than":10000.0,"count":2798.0},{"less_than":20000.0,"count":2798.0},{"less_than":30000.0,"count":2798.0},{"less_than":60000.0,"count":2798.0},{"less_than":300000.0,"count":2798.0},{"less_than":600000.0,"count":2798.0},{"less_than":1800000.0,"count":2798.0},{"less_than":null,"count":2798.0}]
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds : [{"less_than":10.0,"count":4679.0},{"less_than":50.0,"count":4679.0},{"less_than":100.0,"count":4679.0},{"less_than":250.0,"count":4679.0},{"less_than":500.0,"count":4679.0},{"less_than":1000.0,"count":4679.0},{"less_than":2500.0,"count":4679.0},{"less_than":5000.0,"count":4679.0},{"less_than":10000.0,"count":4679.0},{"less_than":20000.0,"count":4679.0},{"less_than":30000.0,"count":4679.0},{"less_than":60000.0,"count":4679.0},{"less_than":300000.0,"count":4679.0},{"less_than":600000.0,"count":4679.0},{"less_than":1800000.0,"count":4679.0},{"less_than":null,"count":4679.0}]
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_count : 787.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_count : 2798.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_count : 4679.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_sum : 180.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_sum : 1304.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_matched_operation_milliseconds_sum : 54.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_matched_rows_total : 138101.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_matched_rows_total : 769193.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_matched_rows_total : 376706.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds : [{"less_than":10.0,"count":3000.0},{"less_than":50.0,"count":3000.0},{"less_than":100.0,"count":3000.0},{"less_than":250.0,"count":3000.0},{"less_than":500.0,"count":3000.0},{"less_than":1000.0,"count":3000.0},{"less_than":2500.0,"count":3000.0},{"less_than":5000.0,"count":3000.0},{"less_than":10000.0,"count":3000.0},{"less_than":20000.0,"count":3000.0},{"less_than":30000.0,"count":3000.0},{"less_than":60000.0,"count":3000.0},{"less_than":300000.0,"count":3000.0},{"less_than":600000.0,"count":3000.0},{"less_than":1800000.0,"count":3000.0},{"less_than":null,"count":3000.0}]
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds_count : 3000.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_not_matched_operation_milliseconds_sum : 0.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_replace_blocks_counter_total : 138.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_replace_blocks_counter_total : 365.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_replace_blocks_counter_total : 785.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_replace_blocks_rows_counter_total : 47135000.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_replace_blocks_rows_counter_total : 145765102.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_replace_blocks_rows_counter_total : 291479898.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_split_milliseconds : [{"less_than":10.0,"count":4215.0},{"less_than":50.0,"count":4215.0},{"less_than":100.0,"count":4215.0},{"less_than":250.0,"count":4215.0},{"less_than":500.0,"count":4215.0},{"less_than":1000.0,"count":4215.0},{"less_than":2500.0,"count":4215.0},{"less_than":5000.0,"count":4215.0},{"less_than":10000.0,"count":4215.0},{"less_than":20000.0,"count":4215.0},{"less_than":30000.0,"count":4215.0},{"less_than":60000.0,"count":4215.0},{"less_than":300000.0,"count":4215.0},{"less_than":600000.0,"count":4215.0},{"less_than":1800000.0,"count":4215.0},{"less_than":null,"count":4215.0}]
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_split_milliseconds : [{"less_than":10.0,"count":6124.0},{"less_than":50.0,"count":6124.0},{"less_than":100.0,"count":6124.0},{"less_than":250.0,"count":6124.0},{"less_than":500.0,"count":6124.0},{"less_than":1000.0,"count":6124.0},{"less_than":2500.0,"count":6124.0},{"less_than":5000.0,"count":6124.0},{"less_than":10000.0,"count":6124.0},{"less_than":20000.0,"count":6124.0},{"less_than":30000.0,"count":6124.0},{"less_than":60000.0,"count":6124.0},{"less_than":300000.0,"count":6124.0},{"less_than":600000.0,"count":6124.0},{"less_than":1800000.0,"count":6124.0},{"less_than":null,"count":6124.0}]
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_split_milliseconds : [{"less_than":10.0,"count":8107.0},{"less_than":50.0,"count":8107.0},{"less_than":100.0,"count":8107.0},{"less_than":250.0,"count":8107.0},{"less_than":500.0,"count":8107.0},{"less_than":1000.0,"count":8107.0},{"less_than":2500.0,"count":8107.0},{"less_than":5000.0,"count":8107.0},{"less_than":10000.0,"count":8107.0},{"less_than":20000.0,"count":8107.0},{"less_than":30000.0,"count":8107.0},{"less_than":60000.0,"count":8107.0},{"less_than":300000.0,"count":8107.0},{"less_than":600000.0,"count":8107.0},{"less_than":1800000.0,"count":8107.0},{"less_than":null,"count":8107.0}]
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_count : 4215.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_count : 6124.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_count : 8107.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_sum : 0.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_sum : 0.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_split_milliseconds_sum : 0.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_unmatched_rows_total : 3907294.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_unmatched_rows_total : 4145899.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] query_merge_into_unmatched_rows_total : 3514807.0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] ===========================
[2023-11-06T16:20:58Z INFO  test_replace_recluster]                            
[2023-11-06T16:20:58Z INFO  test_replace_recluster]                            
[2023-11-06T16:20:58Z INFO  test_replace_recluster] ======CLUSTERING INFO======
[2023-11-06T16:20:58Z INFO  test_replace_recluster] cluster_key : (to_yyyymmdd(insert_time), id)
[2023-11-06T16:20:58Z INFO  test_replace_recluster] block_count: 12
[2023-11-06T16:20:58Z INFO  test_replace_recluster] constant_block_count: 0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] unclustered_block_count: 0
[2023-11-06T16:20:58Z INFO  test_replace_recluster] average_overlaps: 8.5
[2023-11-06T16:20:58Z INFO  test_replace_recluster] average_depth: 7
[2023-11-06T16:20:58Z INFO  test_replace_recluster] block_depth_histogram: {"00007":12}
[2023-11-06T16:20:58Z INFO  test_replace_recluster] ===========================

@dantengsky dantengsky added this pull request to the merge queue Nov 6, 2023
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Nov 6, 2023
@BohuTANG BohuTANG merged commit 10f44fe into databendlabs:main Nov 6, 2023
61 checks passed
andylokandy pushed a commit to andylokandy/databend that referenced this pull request Nov 27, 2023
@JackTan25 @andylokandy
feat: add distributed merge into metrics (databendlabs#13594)
14610f9 
* fix conflict

* add more metrics

* add more metrics

* add more metrics

* add assert

* accumulate row_number

* fix check

* add pipe

* macthed all data

* macthed all data

* macthed all data
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dantengsky dantengsky dantengsky approved these changes

Assignees
No one assigned
Labels
ci-cloud Build docker image for cloud test pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@JackTan25 @SkyFan2002 @dantengsky @BohuTANG