invalid memory address or nil pointer dereference in expression.RemoveDupExprs

[ycybfhb]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

First execute the following valid.sql
valid.txt
Then a crash occurs when executing the error.sql below
error.txt

2. What did you expect to see? (Required)

Expect no crashes

3. What did you see instead (Required)

runtime error: invalid memory address or nil pointer dereference

tidb.log:

[2024/05/27 05:00:20.859 +00:00] [ERROR] [conn.go:1013] ["connection running loop panic"] [conn=1904214046] [session_alias=] [err="runtime error: invalid memory address or nil pointer dereference"] [stack="github.com/pingcap/tidb/pkg/server.(*clientConn).Run.func1
	/workspace/source/tidb/pkg/server/conn.go:1016
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:914
github.com/pingcap/tidb/pkg/executor.(*Compiler).Compile.func1
	/workspace/source/tidb/pkg/executor/compiler.go:57
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:914
runtime.panicmem
	/usr/local/go/src/runtime/panic.go:261
runtime.sigpanic
	/usr/local/go/src/runtime/signal_unix.go:861
github.com/pingcap/tidb/pkg/expression.RemoveDupExprs
	/workspace/source/tidb/pkg/expression/util.go:1379
github.com/pingcap/tidb/pkg/expression.(*propConstSolver).solve
	/workspace/source/tidb/pkg/expression/constant_propagation.go:354
github.com/pingcap/tidb/pkg/expression.(*propConstSolver).PropagateConstant
	/workspace/source/tidb/pkg/expression/constant_propagation.go:711
github.com/pingcap/tidb/pkg/expression.PropagateConstant
	/workspace/source/tidb/pkg/expression/constant_propagation.go:361
github.com/pingcap/tidb/pkg/planner/core.(*DataSource).PredicatePushDown
	/workspace/source/tidb/pkg/planner/core/rule_predicate_push_down.go:139
github.com/pingcap/tidb/pkg/planner/core.(*LogicalSelection).PredicatePushDown
	/workspace/source/tidb/pkg/planner/core/rule_predicate_push_down.go:111
github.com/pingcap/tidb/pkg/planner/core.(*baseLogicalPlan).PredicatePushDown
	/workspace/source/tidb/pkg/planner/core/rule_predicate_push_down.go:84
github.com/pingcap/tidb/pkg/planner/core.(*LogicalProjection).PredicatePushDown
	/workspace/source/tidb/pkg/planner/core/rule_predicate_push_down.go:425
github.com/pingcap/tidb/pkg/planner/core.(*LogicalUnionAll).PredicatePushDown
	/workspace/source/tidb/pkg/planner/core/rule_predicate_push_down.go:434
github.com/pingcap/tidb/pkg/planner/core.(*baseLogicalPlan).PredicatePushDown
	/workspace/source/tidb/pkg/planner/core/rule_predicate_push_down.go:84
github.com/pingcap/tidb/pkg/planner/core.(*LogicalAggregation).PredicatePushDown
	/workspace/source/tidb/pkg/planner/core/rule_predicate_push_down.go:555
github.com/pingcap/tidb/pkg/planner/core.(*LogicalJoin).PredicatePushDown
	/workspace/source/tidb/pkg/planner/core/rule_predicate_push_down.go:242
github.com/pingcap/tidb/pkg/planner/core.(*baseLogicalPlan).PredicatePushDown
	/workspace/source/tidb/pkg/planner/core/rule_predicate_push_down.go:84
github.com/pingcap/tidb/pkg/planner/core.(*LogicalProjection).PredicatePushDown
	/workspace/source/tidb/pkg/planner/core/rule_predicate_push_down.go:425
github.com/pingcap/tidb/pkg/planner/core.(*baseLogicalPlan).PredicatePushDown
	/workspace/source/tidb/pkg/planner/core/rule_predicate_push_down.go:84
github.com/pingcap/tidb/pkg/planner/core.(*LogicalWindow).PredicatePushDown
	/workspace/source/tidb/pkg/planner/core/rule_predicate_push_down.go:731
github.com/pingcap/tidb/pkg/planner/core.(*baseLogicalPlan).PredicatePushDown
	/workspace/source/tidb/pkg/planner/core/rule_predicate_push_down.go:84
github.com/pingcap/tidb/pkg/planner/core.(*LogicalProjection).PredicatePushDown
	/workspace/source/tidb/pkg/planner/core/rule_predicate_push_down.go:425
github.com/pingcap/tidb/pkg/planner/core.(*LogicalSelection).PredicatePushDown
	/workspace/source/tidb/pkg/planner/core/rule_predicate_push_down.go:111
github.com/pingcap/tidb/pkg/planner/core.(*ppdSolver).optimize
	/workspace/source/tidb/pkg/planner/core/rule_predicate_push_down.go:49
github.com/pingcap/tidb/pkg/planner/core.logicalOptimize
	/workspace/source/tidb/pkg/planner/core/optimizer.go:1005
github.com/pingcap/tidb/pkg/planner/core.doOptimize
	/workspace/source/tidb/pkg/planner/core/optimizer.go:289
github.com/pingcap/tidb/pkg/planner/core.(*LogicalCTE).DeriveStats
	/workspace/source/tidb/pkg/planner/core/stats.go:1036
github.com/pingcap/tidb/pkg/planner/core.(*baseLogicalPlan).RecursiveDeriveStats
	/workspace/source/tidb/pkg/planner/core/stats.go:144
github.com/pingcap/tidb/pkg/planner/core.(*baseLogicalPlan).RecursiveDeriveStats
	/workspace/source/tidb/pkg/planner/core/stats.go:137
github.com/pingcap/tidb/pkg/planner/core.(*baseSingleGroupJoinOrderSolver).generateJoinOrderNode
	/workspace/source/tidb/pkg/planner/core/rule_join_reorder.go:477
github.com/pingcap/tidb/pkg/planner/core.(*joinReorderGreedySolver).solve
	/workspace/source/tidb/pkg/planner/core/rule_join_reorder_greedy.go:48
github.com/pingcap/tidb/pkg/planner/core.(*joinReOrderSolver).optimizeRecursive
	/workspace/source/tidb/pkg/planner/core/rule_join_reorder.go:297
github.com/pingcap/tidb/pkg/planner/core.(*joinReOrderSolver).optimizeRecursive
	/workspace/source/tidb/pkg/planner/core/rule_join_reorder.go:335
github.com/pingcap/tidb/pkg/planner/core.(*joinReOrderSolver).optimizeRecursive
	/workspace/source/tidb/pkg/planner/core/rule_join_reorder.go:335
github.com/pingcap/tidb/pkg/planner/core.(*joinReOrderSolver).optimizeRecursive
	/workspace/source/tidb/pkg/planner/core/rule_join_reorder.go:335
github.com/pingcap/tidb/pkg/planner/core.(*joinReOrderSolver).optimizeRecursive
	/workspace/source/tidb/pkg/planner/core/rule_join_reorder.go:335
github.com/pingcap/tidb/pkg/planner/core.(*joinReOrderSolver).optimizeRecursive
	/workspace/source/tidb/pkg/planner/core/rule_join_reorder.go:335
github.com/pingcap/tidb/pkg/planner/core.(*joinReOrderSolver).optimize
	/workspace/source/tidb/pkg/planner/core/rule_join_reorder.go:231
github.com/pingcap/tidb/pkg/planner/core.logicalOptimize
	/workspace/source/tidb/pkg/planner/core/optimizer.go:1005
github.com/pingcap/tidb/pkg/planner/core.doOptimize
	/workspace/source/tidb/pkg/planner/core/optimizer.go:289
github.com/pingcap/tidb/pkg/planner/core.DoOptimize
	/workspace/source/tidb/pkg/planner/core/optimizer.go:348
github.com/pingcap/tidb/pkg/planner.optimize
	/workspace/source/tidb/pkg/planner/optimize.go:503
github.com/pingcap/tidb/pkg/planner.Optimize
	/workspace/source/tidb/pkg/planner/optimize.go:334
github.com/pingcap/tidb/pkg/executor.(*Compiler).Compile
	/workspace/source/tidb/pkg/executor/compiler.go:99
github.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt
	/workspace/source/tidb/pkg/session/session.go:2094
github.com/pingcap/tidb/pkg/server.(*TiDBContext).ExecuteStmt
	/workspace/source/tidb/pkg/server/driver_tidb.go:294
github.com/pingcap/tidb/pkg/server.(*clientConn).handleStmt
	/workspace/source/tidb/pkg/server/conn.go:2021
github.com/pingcap/tidb/pkg/server.(*clientConn).handleQuery
	/workspace/source/tidb/pkg/server/conn.go:1774
github.com/pingcap/tidb/pkg/server.(*clientConn).dispatch
	/workspace/source/tidb/pkg/server/conn.go:1348
github.com/pingcap/tidb/pkg/server.(*clientConn).Run
	/workspace/source/tidb/pkg/server/conn.go:1114
github.com/pingcap/tidb/pkg/server.(*Server).onConn
	/workspace/source/tidb/pkg/server/server.go:739"]

4. What is your TiDB version? (Required)

+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version()                                                                                                                                                                                                                                                   |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v8.2.0-alpha-216-gfe5858b
Edition: Community
Git Commit Hash: fe5858b00cd63808ac414c6e102a353778b0aaa7
Git Branch: HEAD
UTC Build Time: 2024-05-23 01:44:42
GoVersion: go1.21.10
Race Enabled: false
Check Table Before Drop: false
Store: tikv |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

We are the BASS team from the School of Cyber Science and Technology at Beihang University. Our main focus is on system software security, operating systems, and program analysis research, as well as the development of automated program testing frameworks for detecting software defects. Using our self-developed database vulnerability testing tool, we have identified the above-mentioned vulnerabilities in TiDB that may lead to database crashes.

[Rustin170506]

Do you know what SQL caused the panic? Could you please provide a minimum set of steps?

[ycybfhb]

The provided valid.sql file is used to set up the database. Now we have simplified the database table creation statements in
simplified_init.txt and the error-generating statements in simplified_error.txt. First, execute simplified_init.sql to create the tables, and then execute simplified_error.sql to reproduce the error.

[YangKeao]

It may have a similar root cause with #53580, and I have tried the fix #53716 can also fix this issue. PTAL @hi-rustin