statistics: init LastAnalyzeVersion with snapshot timestamp #54465

hi-rustin · 2024-07-05T03:41:12Z

What problem does this PR solve?

Issue Number: ref #53567

Problem Summary:

It only occurs in the following situations:

The table has already been analyzed, but because the predicate columns feature is turned on, and it doesn't have any columns or indexes analyzed, it only analyzes _row_id and refreshes stats_meta, in which case the snapshot is not zero.
LastAnalyzeVersion is 0 because this are no histogram(buckets) records for this table at all.

So in this case, the auto-analyze will try to analyze it again and again.

If you find an actual case and attempt to compare the snapshot from stats_meta with the version from the histogram, you will always find that this equation is true.

+------------------+--------+------------+-----+------------------+
|version           |table_id|modify_count|count|snapshot          |
+------------------+--------+------------+-----+------------------+
|450928595513638913|104     |0           |3000 |450928595500531720|
+------------------+--------+------------+-----+------------------+

+--------+--------+-------+--------------+----------+------------+------------+------------------+---------+---------+----+---------------------+--------------------------------------------------+
|table_id|is_index|hist_id|distinct_count|null_count|tot_col_size|modify_count|version           |cm_sketch|stats_ver|flag|correlation          |last_analyze_pos                                  |
+--------+--------+-------+--------------+----------+------------+------------+------------------+---------+---------+----+---------------------+--------------------------------------------------+
|104     |0       |1      |3000          |0         |24000       |0           |450928595513638913|null     |2        |1   |1                    |com.intellij.database.extractors.TextInfo@efb43b49|
|104     |0       |2      |504           |0         |24000       |0           |450928595513638913|null     |2        |1   |0.026502627833625315 |com.intellij.database.extractors.TextInfo@ee1d8593|
|104     |0       |3      |3000          |0         |363000      |0           |450928595513638913|null     |2        |1   |-0.007597387066376341|com.intellij.database.extractors.TextInfo@53641c4d|
|104     |0       |4      |3000          |0         |180000      |0           |450928595513638913|null     |2        |1   |-0.014492787832531981|com.intellij.database.extractors.TextInfo@f16e0d31|
|104     |1       |1      |504           |0         |24000       |0           |450928595513638913|null     |2        |1   |0                    |0x0380000000000008D1                              |
+--------+--------+-------+--------------+----------+------------+------------+------------------+---------+---------+----+---------------------+--------------------------------------------------+

450928595500531720 - 450928595513638913 = -13,107,193

So we can safely use the snapshot to initialize the LastAnalyzeVersion.

What changed and how does it work?

Initialized LastAnalyzeVersion with snapshot timestamp.
Added a test case to cover it.

Check List

Tests

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

codecov · 2024-07-05T04:08:31Z

Codecov Report

Attention: Patch coverage is 87.50000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 55.5985%. Comparing base (9044acb) to head (1ca210d).
Report is 6 commits behind head on master.

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #54465         +/-   ##
=================================================
- Coverage   72.8339%   55.5985%   -17.2355%     
=================================================
  Files          1551       1672        +121     
  Lines        436700     609230     +172530     
=================================================
+ Hits         318066     338723      +20657     
- Misses        99093     247472     +148379     
- Partials      19541      23035       +3494

Flag	Coverage Δ
integration	`36.0274% <87.5000%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.9656% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`52.5009% <ø> (+6.6536%)`	⬆️

hi-rustin · 2024-07-05T04:59:24Z

Tested locally:
Before:

Start the TiDB cluster:

mysql> select tidb_version();
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version()                                                                                                                                                                                                                                                      |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v8.2.0-alpha-469-gc983546c52
Edition: Community
Git Commit Hash: c983546c525f7278946ceb67540d8d74448d3ef0
Git Branch: HEAD
UTC Build Time: 2024-07-01 06:02:24
GoVersion: go1.21.10
Race Enabled: false
Check Table Before Drop: false
Store: tikv |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Enable the predicate columns feature:

mysql> set global tidb_analyze_column_options='PREDICATE';
Query OK, 0 rows affected (0.01 sec)
mysql> select @@tidb_analyze_column_options;
+-------------------------------+
| @@tidb_analyze_column_options |
+-------------------------------+
| PREDICATE                     |
+-------------------------------+
1 row in set (0.00 sec)

Create a table and insert 1000 rows:

CREATE TABLE test.random_data (
    id INT,
    value VARCHAR(100)
);

INSERT INTO test.random_data (value)
SELECT
    CONCAT('RandomValue', FLOOR(RAND() * 10000))
FROM
    (SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9 UNION SELECT 10) t1,
    (SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9 UNION SELECT 10) t2,
    (SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9 UNION SELECT 10) t3;

Check the stats_meta:

mysql> select * from mysql.stats_meta;
+------------------+--------+------------+-----+------------------+
|version           |table_id|modify_count|count|snapshot          |
+------------------+--------+------------+-----+------------------+
|450928411885961235|104     |0           |1000 |450928411885961225|
+------------------+--------+------------+-----+------------------+

Check logs:

[2024/07/05 12:56:01.866 +08:00] [INFO] [analyze.go:748] ["analyze table `test`.`random_data` has finished"] [partition=] ["job info"="auto analyze table columns  with 256 buckets, 100 topn, 1 samplerate"] ["start time"=2024/07/05 12:56:01.860 +08:00] ["end time"=2024/07/05 12:56:01.864 +08:00] [cost=4.716083ms] ["sample rate reason"="use min(1, 110000/1000) as the sample-rate=1"]
[2024/07/05 12:56:04.878 +08:00] [INFO] [refresher.go:101] ["Auto analyze triggered"] [category=stats] [job="NonPartitionedTableAnalysisJob:\n\tAnalyzeType: analyzeTable\n\tIndexes: \n\tSchema: test\n\tTable: random_data\n\tTableID: 104\n\tTableStatsVer: 2\n\tChangePercentage: 1.000000\n\tTableSize: 2000.00\n\tLastAnalysisDuration: 30m0s\n\tWeight: 1.463794\n"]
[2024/07/05 12:56:04.892 +08:00] [INFO] [save.go:200] ["incrementally update modifyCount"] [category=stats] [tableID=104] [curModifyCnt=0] [results.BaseModifyCnt=0] [modifyCount=0]
[2024/07/05 12:56:04.892 +08:00] [INFO] [save.go:222] ["directly update count"] [category=stats] [tableID=104] [results.Count=1000] [count=1000]
[2024/07/05 12:56:04.895 +08:00] [INFO] [analyze.go:748] ["analyze table `test`.`random_data` has finished"] [partition=] ["job info"="auto analyze table columns  with 256 buckets, 100 topn, 1 samplerate"] ["start time"=2024/07/05 12:56:04.889 +08:00] ["end time"=2024/07/05 12:56:04.894 +08:00] [cost=4.835291ms] ["sample rate reason"="use min(1, 110000/1000) as the sample-rate=1"]
[2024/07/05 12:56:07.854 +08:00] [INFO] [refresher.go:101] ["Auto analyze triggered"] [category=stats] [job="NonPartitionedTableAnalysisJob:\n\tAnalyzeType: analyzeTable\n\tIndexes: \n\tSchema: test\n\tTable: random_data\n\tTableID: 104\n\tTableStatsVer: 2\n\tChangePercentage: 1.000000\n\tTableSize: 2000.00\n\tLastAnalysisDuration: 30m0s\n\tWeight: 1.463794\n"]
[2024/07/05 12:56:07.861 +08:00] [INFO] [save.go:200] ["incrementally update modifyCount"] [category=stats] [tableID=104] [curModifyCnt=0] [results.BaseModifyCnt=0] [modifyCount=0]
[2024/07/05 12:56:07.861 +08:00] [INFO] [save.go:222] ["directly update count"] [category=stats] [tableID=104] [results.Count=1000] [count=1000]

After:

Start the TiDB cluster:

mysql> select tidb_version();
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version()                                                                                                                                                                                                                                                       |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v8.2.0-alpha-527-gba34df6c91
Edition: Community
Git Commit Hash: ba34df6c91c46d82ce3e07b79823ffc0d0480513
Git Branch: master
UTC Build Time: 2024-07-05 03:51:07
GoVersion: go1.22.2
Race Enabled: false
Check Table Before Drop: false
Store: tikv |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Enable the predicate columns feature:

mysql> set global tidb_analyze_column_options='PREDICATE';
Query OK, 0 rows affected (0.01 sec)
mysql> select @@tidb_analyze_column_options;
+-------------------------------+
| @@tidb_analyze_column_options |
+-------------------------------+
| PREDICATE                     |
+-------------------------------+
1 row in set (0.00 sec)

Create a table and insert 1000 rows:

CREATE TABLE test.random_data (
    id INT,
    value VARCHAR(100)
);

INSERT INTO test.random_data (value)
SELECT
    CONCAT('RandomValue', FLOOR(RAND() * 10000))
FROM
    (SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9 UNION SELECT 10) t1,
    (SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9 UNION SELECT 10) t2,
    (SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9 UNION SELECT 10) t3;

Check the stats_meta:

+------------------+--------+------------+-----+------------------+
|version           |table_id|modify_count|count|snapshot          |
+------------------+--------+------------+-----+------------------+
|450928506810662939|104     |0           |1000 |450928506810662922|
+------------------+--------+------------+-----+------------------+

Check logs:

[2024/07/05 13:02:21.960 +08:00] [INFO] [analyze.go:748] ["analyze table `test`.`random_data` has finished"] [partition=] ["job info"="auto analyze table with 256 buckets, 100 topn, 1 samplerate"] ["start time"=2024/07/05 13:02:21.947 +08:00] ["end time"=2024/07/05 13:02:21.958 +08:00] [cost=10.369458ms] ["sample rate reason"="use min(1, 110000/1000) as the sample-rate=1"]

hi-rustin

🔢 Self-check (PR reviewed by myself and ready for feedback.)

pkg/statistics/handle/cache/statscache.go

misoperation

hi-rustin · 2024-07-12T09:01:57Z

I also tested the show statements:

mysql> analyze table t;
Query OK, 0 rows affected, 2 warnings (0.01 sec)

mysql> show stats_meta;
+---------+------------+----------------+---------------------+--------------+-----------+---------------------+
| Db_name | Table_name | Partition_name | Update_time         | Modify_count | Row_count | Last_analyze_time   |
+---------+------------+----------------+---------------------+--------------+-----------+---------------------+
| test    | t          |                | 2024-07-12 16:59:33 |            0 |         3 | 2024-07-12 16:59:32 |
+---------+------------+----------------+---------------------+--------------+-----------+---------------------+
1 row in set (0.00 sec)

mysql> SHOW STATS_HEALTHY;
+---------+------------+----------------+---------+
| Db_name | Table_name | Partition_name | Healthy |
+---------+------------+----------------+---------+
| test    | t          |                |     100 |
+---------+------------+----------------+---------+
1 row in set (0.00 sec)

mysql> SHOW STATS_HISTOGRAMS;
Empty set (0.01 sec)

hi-rustin

🔢 Self-check (PR reviewed by myself and ready for feedback.)

ti-chi-bot · 2024-07-15T17:06:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: time-and-fate

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/statistics/OWNERS~~ [time-and-fate]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

time-and-fate · 2024-07-15T17:06:12Z

/hold

ti-chi-bot · 2024-07-15T17:06:12Z

[LGTM Timeline notifier]

Timeline:

2024-07-08 16:50:06.755100887 +0000 UTC m=+289903.990335000: ☑️ agreed by winoros.
2024-07-15 17:06:11.755624925 +0000 UTC m=+287193.746566395: ☑️ agreed by time-and-fate.

hi-rustin · 2024-07-16T07:12:08Z

/unhold

hawkingrei · 2024-07-16T08:05:38Z

/retest

ti-chi-bot bot added release-note-none sig/planner SIG: Planner size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 5, 2024

hi-rustin requested review from winoros and time-and-fate July 5, 2024 06:20

hi-rustin commented Jul 5, 2024

View reviewed changes

winoros previously approved these changes Jul 8, 2024

View reviewed changes

pkg/statistics/handle/cache/statscache.go Show resolved Hide resolved

ti-chi-bot bot added approved needs-1-more-lgtm labels Jul 8, 2024

ti-chi-bot bot removed the approved label Jul 8, 2024

ti-chi-bot bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 15, 2024

hi-rustin added 4 commits July 15, 2024 20:19

statistics: init LastAnalyzeVersion with snapshot timestamp

2dbd254

refactor: better code

7c1ae95

fix: typo

e8dc0a7

fix: better comment

daf23ff

hi-rustin force-pushed the rustin-patch-snapshot branch from 36c4200 to daf23ff Compare July 15, 2024 12:20

ti-chi-bot bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 15, 2024

hi-rustin added 3 commits July 15, 2024 20:24

test: rename and use the all columns option

e816860

refactor: better comment

f14fb46

refactor: better comment

1ca210d

hi-rustin commented Jul 15, 2024

View reviewed changes

time-and-fate approved these changes Jul 15, 2024

View reviewed changes

ti-chi-bot bot added lgtm approved and removed needs-1-more-lgtm labels Jul 15, 2024

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 15, 2024

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 16, 2024

ti-chi-bot bot merged commit 1bf27af into pingcap:master Jul 16, 2024
22 of 23 checks passed

hi-rustin deleted the rustin-patch-snapshot branch July 16, 2024 08:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

statistics: init LastAnalyzeVersion with snapshot timestamp #54465

statistics: init LastAnalyzeVersion with snapshot timestamp #54465

hi-rustin commented Jul 5, 2024 •

edited

Loading

codecov bot commented Jul 5, 2024 •

edited

Loading

hi-rustin commented Jul 5, 2024 •

edited

Loading

hi-rustin left a comment

hi-rustin commented Jul 12, 2024

hi-rustin left a comment

ti-chi-bot bot commented Jul 15, 2024

time-and-fate commented Jul 15, 2024

ti-chi-bot bot commented Jul 15, 2024

hi-rustin commented Jul 16, 2024

hawkingrei commented Jul 16, 2024

statistics: init LastAnalyzeVersion with snapshot timestamp #54465

statistics: init LastAnalyzeVersion with snapshot timestamp #54465

Conversation

hi-rustin commented Jul 5, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

codecov bot commented Jul 5, 2024 • edited Loading

Codecov Report

hi-rustin commented Jul 5, 2024 • edited Loading

hi-rustin left a comment

Choose a reason for hiding this comment

hi-rustin commented Jul 12, 2024

hi-rustin left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Jul 15, 2024

time-and-fate commented Jul 15, 2024

ti-chi-bot bot commented Jul 15, 2024

[LGTM Timeline notifier]

hi-rustin commented Jul 16, 2024

hawkingrei commented Jul 16, 2024

hi-rustin commented Jul 5, 2024 •

edited

Loading

codecov bot commented Jul 5, 2024 •

edited

Loading

hi-rustin commented Jul 5, 2024 •

edited

Loading