Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SR-3289] FE schedule algorithm for backend disk balance #5

Closed
wants to merge 2 commits into from
Closed

[SR-3289] FE schedule algorithm for backend disk balance #5

wants to merge 2 commits into from

Conversation

wyb
Copy link
Contributor

@wyb wyb commented Sep 5, 2021

  1. On the basis of the original cluster tablet balance, add backend disk balance schedule algorithm.
  2. Implement backend local migration schedule process.
  3. Improve the storage medium migration process.
  4. Add storage medium, tablet num in BackendProcNode.

gengjun-git
gengjun-git reviewed Sep 6, 2021
View reviewed changes
fe/fe-core/src/main/java/com/starrocks/clone/DiskAndTabletLoadReBalancer.java Outdated
* cluster balance args: beStats, isLocalBalance is false.
* backend balance args: pathStats, beId, beNum, isLocalBalance is true.
*/
private void balanceDisk(String clusterName, TStorageMedium medium, double avgUsedPercent,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use two functions to distinguish between cluster disk balance and local disk balance, too many if (!isLocalBalance) makes the reviewers confused.

gengjun-git
gengjun-git reviewed Sep 6, 2021
View reviewed changes
fe/fe-core/src/main/java/com/starrocks/clone/DiskAndTabletLoadReBalancer.java Outdated
* for every backend:
* 1. for every partition, calculate the distribution skew, skew is (max tablet number on path) - (min tablet number on path)
* 2. sort partition by skew in desc order
* 3. for every partition, sort be by tablet number
Copy link
Contributor

@gengjun-git gengjun-git Sep 6, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. for every partition, sort disk by tablet number

gengjun-git
gengjun-git reviewed Sep 6, 2021
View reviewed changes
fe/fe-core/src/main/java/com/starrocks/clone/DiskAndTabletLoadReBalancer.java Outdated
@@ -236,73 +374,221 @@ private boolean isDiskBalanced(ClusterLoadStatistic clusterStat, TStorageMedium

// sort be by disk used percent in asc order
beStats.sort(new BackendLoadStatistic.BeStatComparatorForUsedPercent(medium));
LOG.debug("get backend stats for cluster disk balance. be stats: {}", beStats);

balanceDisk(clusterName, medium, avgUsedPercent, alternativeTablets, false, beStats, null, -1, 0);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For cluster disk balance, avgUsedPercent = (avg(be1) + avg(be2) + ... + avg(ben)) / n,
But for local disk balance, avgUsedPercent = (used(disk1) + used(disk2) + ... + used(diskn)) / (total(disk1) + total (disk2) + ... + total(diskn)).
Is it better to use the same algorithm?

@CLAassistant
Copy link

CLAassistant commented Sep 8, 2021 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@wyb wyb closed this Sep 9, 2021
@wangruin wangruin mentioned this pull request Nov 10, 2021
Closed
This was referenced Nov 11, 2021
Closed
Closed
@wanpengfei-git wanpengfei-git mentioned this pull request Nov 29, 2021
Closed
This was referenced Dec 13, 2021
Closed
Closed
@wanpengfei-git wanpengfei-git mentioned this pull request Jan 12, 2022
Closed
@Astralidea Astralidea mentioned this pull request Feb 21, 2022
Merged
4 tasks
@Seaven Seaven mentioned this pull request Mar 18, 2022
Closed
@Linkerist Linkerist mentioned this pull request May 12, 2022
Closed
@colorfulu colorfulu mentioned this pull request May 14, 2022
Closed
@tiannan-sr tiannan-sr mentioned this pull request Jun 7, 2022
Closed
chaoyli pushed a commit that referenced this pull request Aug 3, 2022
@Linkerist
[BugFix] Vectical rowset writer final flush need writer finalize foot…
5fa910d 
…er for partial columns. (#9075)

Once the following conditions are met, a data load will cause the final merge in the rowset writer.
1. partial update
2. large data load in one batch to cause final merge
3. the number of partial columns large than config::vertical_compaction_max_columns_per_group [default = 5]

final merge will do vertical write under some circumstances, will clear partial_rowset_footers array at first, need to create new partial_rowset_footers for newly merged segments, old code of horizontal rowset writer did this, but the newly added VerticalRowsetWriter didn't, which will cause the partial_rowset_footers is empty. null pointer error will occur in rowset load like this:
```
#0  0x0000000001f8f7a0 in starrocks::BetaRowset::do_load() ()
#1  0x0000000001f8790c in starrocks::Rowset::load() ()
#2  0x0000000001f8e1d6 in starrocks::BetaRowset::get_segment_iterators2(starrocks::vectorized::Schema const&, starrocks::KVStore*, long, starrocks::OlapReaderStatistics*) ()
#3  0x0000000001c40c68 in starrocks::RowsetUpdateState::_do_load(starrocks::Tablet*, starrocks::Rowset*) ()
#4  0x0000000001c41775 in void std::call_once<starrocks::RowsetUpdateState::load(starrocks::Tablet*, starrocks::Rowset*)::{lambda()#1}>(std::once_flag&, starrocks::RowsetUpdateState::load(starrocks::Tablet*, starrocks::Rowset*)::{lambda()#1}&&)::{lambda()#2}::_FUN() ()
#5  0x00007fd681f9f1cb in __pthread_once_slow () from /lib64/libpthread.so.0
#6  0x0000000001c3d9b3 in starrocks::RowsetUpdateState::load(starrocks::Tablet*, starrocks::Rowset*) ()
#7  0x0000000001ad7203 in starrocks::UpdateManager::on_rowset_finished(starrocks::Tablet*, starrocks::Rowset*) ()
#8  0x00000000032ff75b in starrocks::vectorized::DeltaWriter::commit() ()
#9  0x00000000032f03c1 in starrocks::vectorized::AsyncDeltaWriter::_execute(void*, bthread::TaskIterator<starrocks::vectorized::AsyncDeltaWriter::Task>&) ()
#10 0x00000000040581ec in bthread::ExecutionQueueBase::_execute(bthread::TaskNode*, bool, int*) ()
#11 0x0000000004058fb8 in bthread::ExecutionQueueBase::_execute_tasks(void*) ()
#12 0x00000000021257e9 in starrocks::ThreadPool::dispatch_thread() ()
#13 0x000000000212139a in starrocks::Thread::supervise_thread(void*) ()
#14 0x00007fd681fa0e65 in start_thread () from /lib64/libpthread.so.0
#15 0x00007fd6815bb88d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#16 0x0000000000000000 in ?? ()
```

```
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000001f8f7a0 in google::protobuf::internal::RepeatedPtrFieldBase::Get<google::protobuf::RepeatedPtrField<starrocks::FooterPointerPB>::TypeHandler> (index=0, this=<optimized out>) at /root/starrocks/be/../gensrc/build/gen_cpp/olap_file.pb.h:4467
4467	/root/starrocks/be/../gensrc/build/gen_cpp/olap_file.pb.h: No such file or directory.
[Current thread is 1 (LWP 51456)]
(gdb) bt
#0  0x0000000001f8f7a0 in google::protobuf::internal::RepeatedPtrFieldBase::Get<google::protobuf::RepeatedPtrField<starrocks::FooterPointerPB>::TypeHandler> (index=0, this=<optimized out>) at /root/starrocks/be/../gensrc/build/gen_cpp/olap_file.pb.h:4467
#1  google::protobuf::RepeatedPtrField<starrocks::FooterPointerPB>::Get (index=0, this=<optimized out>) at /var/local/thirdparty/installed/include/google/protobuf/repeated_field.h:2173
#2  starrocks::RowsetTxnMetaPB::_internal_partial_rowset_footers (index=0, this=<optimized out>) at /root/starrocks/be/../gensrc/build/gen_cpp/olap_file.pb.h:4463
#3  starrocks::RowsetTxnMetaPB::partial_rowset_footers (index=0, this=<optimized out>) at /root/starrocks/be/../gensrc/build/gen_cpp/olap_file.pb.h:4467
#4  starrocks::RowsetMeta::partial_rowset_footer (segment_id=0, this=0x11ccc910) at /root/starrocks/be/src/storage/rowset/rowset_meta.h:174
#5  starrocks::BetaRowset::do_load (this=0xfd993f0) at /root/starrocks/be/src/storage/rowset/beta_rowset.cpp:81
#6  0x0000000001f8790c in starrocks::Rowset::load (this=this@entry=0xfd993f0) at /root/starrocks/be/src/storage/rowset/rowset.cpp:50
#7  0x0000000001f8e1d6 in starrocks::BetaRowset::get_segment_iterators2 (this=this@entry=0xfd993f0, schema=..., meta=meta@entry=0x0, version=version@entry=0, stats=stats@entry=0x7fd6619bc070) at /root/starrocks/be/src/storage/rowset/beta_rowset.cpp:358
#8  0x0000000001c40c68 in starrocks::RowsetUpdateState::_do_load (this=0x10576d50, tablet=0xf130380, rowset=0xfd993f0) at /root/starrocks/be/src/storage/rowset_update_state.cpp:81
#9  0x0000000001c41775 in operator() (__closure=0x7fd6619bc2b0) at /root/starrocks/be/src/storage/rowset_update_state.cpp:38
#10 std::__invoke_impl<void, starrocks::RowsetUpdateState::load(starrocks::Tablet*, starrocks::Rowset*)::<lambda()> > (__f=...) at /usr/include/c++/10.3.0/bits/invoke.h:60
#11 std::__invoke<starrocks::RowsetUpdateState::load(starrocks::Tablet*, starrocks::Rowset*)::<lambda()> > (__fn=...) at /usr/include/c++/10.3.0/bits/invoke.h:95
#12 operator() (this=<optimized out>) at /usr/include/c++/10.3.0/mutex:717
#13 operator() (this=0x0) at /usr/include/c++/10.3.0/mutex:722
#14 _FUN () at /usr/include/c++/10.3.0/mutex:722
#15 0x00007fd681f9f1cb in __pthread_once_slow () from /lib64/libpthread.so.0
#16 0x0000000001c3d9b3 in __gthread_once (__func=<optimized out>, __once=0x10576d50) at /usr/include/c++/10.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:700
#17 std::call_once<starrocks::RowsetUpdateState::load(starrocks::Tablet*, starrocks::Rowset*)::<lambda()> > (__f=..., __once=...) at /usr/include/c++/10.3.0/mutex:729
#18 starrocks::RowsetUpdateState::load (this=this@entry=0x10576d50, tablet=tablet@entry=0xf130380, rowset=rowset@entry=0xfd993f0) at /root/starrocks/be/src/storage/rowset_update_state.cpp:36
#19 0x0000000001ad7203 in starrocks::UpdateManager::on_rowset_finished (this=0x94a3a40, tablet=0xf130380, rowset=0xfd993f0) at /root/starrocks/be/src/storage/update_manager.cpp:260
#20 0x00000000032ff75b in starrocks::vectorized::DeltaWriter::commit (this=this@entry=0xfc5f790) at /root/starrocks/be/src/storage/storage_engine.h:164
#21 0x00000000032f03c1 in starrocks::vectorized::AsyncDeltaWriter::_execute (meta=0xfc5f790, iter=...) at /root/starrocks/be/src/storage/async_delta_writer.cpp:38
#22 0x00000000040581ec in bthread::ExecutionQueueBase::_execute (this=0xabbe600, head=<optimized out>, high_priority=<optimized out>, niterated=0x0) at /var/local/thirdparty/src/incubator-brpc-0.9.7/src/bthread/execution_queue.cpp:272
#23 0x0000000004058fb8 in bthread::ExecutionQueueBase::_execute_tasks (arg=<optimized out>) at /var/local/thirdparty/src/incubator-brpc-0.9.7/src/bthread/execution_queue.cpp:151
#24 0x00000000021257e9 in std::function<void ()>::operator()() const (this=<optimized out>) at /usr/include/c++/10.3.0/bits/std_function.h:248
#25 starrocks::FunctionRunnable::run (this=<optimized out>) at /root/starrocks/be/src/util/threadpool.cpp:44
#26 starrocks::ThreadPool::dispatch_thread (this=0xac36540) at /root/starrocks/be/src/util/threadpool.cpp:513
#27 0x000000000212139a in std::function<void ()>::operator()() const (this=0x9e02c58) at /usr/include/c++/10.3.0/bits/std_function.h:248
#28 starrocks::Thread::supervise_thread (arg=0x9e02c40) at /root/starrocks/be/src/util/thread.cpp:326
#29 0x00007fd681fa0e65 in start_thread () from /lib64/libpthread.so.0
#30 0x00007fd6815bb88d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#31 0x0000000000000000 in ?? ()
```
@meegoo meegoo mentioned this pull request Aug 18, 2022
Closed
@dulong41 dulong41 mentioned this pull request Nov 10, 2022
Closed
This was referenced Nov 18, 2022
Closed
Closed
Closed
This was referenced Nov 29, 2022
Closed
Closed
@dulong41 dulong41 mentioned this pull request Dec 6, 2022
Closed
@yongbingwang yongbingwang mentioned this pull request Feb 22, 2023
Closed
imay pushed a commit that referenced this pull request Jul 10, 2023
@trueeyu
[BugFix] Fix the mem leak of build in predicate for es external table (…
8918a95 
…#26631) (#26805)

VExtLiteral is not deleted.

```
==76869==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x9793a37 in operator new(unsigned long) ../../.././libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x129dd44e in operator()<int const> /root/starrocks/be/src/exec/es/es_predicate.cpp:416
    #2 0x129dd817 in build_inpred_values<(starrocks::LogicalType)5, starrocks::EsPredicate::_build_in_predicate(const starrocks::Expr*, bool*)::<lambda(auto:29&)> > /root/starrocks/be/src/exec/es/es_predicate.cpp:366
    #3 0x129d7824 in starrocks::EsPredicate::_build_in_predicate(starrocks::Expr const*, bool*) /root/starrocks/be/src/exec/es/es_predicate.cpp:416
    #4 0x129d2f2d in starrocks::EsPredicate::_vec_build_disjuncts_list(starrocks::Expr const*) /root/starrocks/be/src/exec/es/es_predicate.cpp:185
    #5 0x129d2a53 in starrocks::EsPredicate::build_disjuncts_list() /root/starrocks/be/src/exec/es/es_predicate.cpp:148
    #6 0x128ab50d in starrocks::connector::ESDataSource::_build_conjuncts() /root/starrocks/be/src/connector/es_connector.cpp:123
    #7 0x128aa7c5 in starrocks::connector::ESDataSource::open(starrocks::RuntimeState*) /root/starrocks/be/src/connector/es_connector.cpp:89
    #8 0x9e33847 in starrocks::pipeline::ConnectorChunkSource::_open_data_source(starrocks::RuntimeState*) /root/starrocks/be/src/exec/pipeline/scan/connector_scan_operator.cpp:504
    #9 0x9e33ed0 in starrocks::pipeline::ConnectorChunkSource::_read_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/pipeline/scan/connector_scan_operator.cpp:533
    #10 0xabc8b65 in starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking(starrocks::RuntimeState*, unsigned long, starrocks::workgroup::WorkGroup const*) /root/starrocks/be/src/exec/pipeline/scan/chunk_source.cpp:67
    #11 0x9e168c7 in operator() /root/starrocks/be/src/exec/pipeline/scan/scan_operator.cpp:396
    #12 0x9e1c157 in __invoke_impl<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    #13 0x9e1c005 in __invoke_r<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #14 0x9e1be7a in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #15 0x992a271 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #16 0x9e519fa in starrocks::workgroup::ScanExecutor::worker_thread() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:67
    #17 0x9e5121f in operator() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:31
    #18 0x9e52c5d in __invoke_impl<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    #19 0x9e52925 in __invoke_r<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #20 0x9e5249a in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #21 0x992a271 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #22 0x1181f23d in starrocks::FunctionRunnable::run() /root/starrocks/be/src/util/threadpool.cpp:58
    #23 0x1181bff2 in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:553
    #24 0x11837c4b in void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:73
    #25 0x118375a4 in std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.
3.0/bits/invoke.h:95
    #26 0x1183699b in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /opt/gcc/usr/include/c++/10.3.0/functional:416
    #27 0x118352fd in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::operator()<, void>() /opt/gcc/usr/include/c++/10.3.0/functional:499
    #28 0x11832361 in void std::__invoke_impl<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    #29 0x1182fcc5 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110

```

Signed-off-by: trueeyu <[email protected]>
imay pushed a commit that referenced this pull request Jul 10, 2023
@trueeyu
[BugFix] Fix the mem leak of build in predicate for es external table (…
bbfae11 
…#26631) (#26817)

VExtLiteral is not deleted.

```
==76869==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x9793a37 in operator new(unsigned long) ../../.././libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x129dd44e in operator()<int const> /root/starrocks/be/src/exec/es/es_predicate.cpp:416
    #2 0x129dd817 in build_inpred_values<(starrocks::LogicalType)5, starrocks::EsPredicate::_build_in_predicate(const starrocks::Expr*, bool*)::<lambda(auto:29&)> > /root/starrocks/be/src/exec/es/es_predicate.cpp:366
    #3 0x129d7824 in starrocks::EsPredicate::_build_in_predicate(starrocks::Expr const*, bool*) /root/starrocks/be/src/exec/es/es_predicate.cpp:416
    #4 0x129d2f2d in starrocks::EsPredicate::_vec_build_disjuncts_list(starrocks::Expr const*) /root/starrocks/be/src/exec/es/es_predicate.cpp:185
    #5 0x129d2a53 in starrocks::EsPredicate::build_disjuncts_list() /root/starrocks/be/src/exec/es/es_predicate.cpp:148
    #6 0x128ab50d in starrocks::connector::ESDataSource::_build_conjuncts() /root/starrocks/be/src/connector/es_connector.cpp:123
    #7 0x128aa7c5 in starrocks::connector::ESDataSource::open(starrocks::RuntimeState*) /root/starrocks/be/src/connector/es_connector.cpp:89
    #8 0x9e33847 in starrocks::pipeline::ConnectorChunkSource::_open_data_source(starrocks::RuntimeState*) /root/starrocks/be/src/exec/pipeline/scan/connector_scan_operator.cpp:504
    #9 0x9e33ed0 in starrocks::pipeline::ConnectorChunkSource::_read_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/pipeline/scan/connector_scan_operator.cpp:533
    #10 0xabc8b65 in starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking(starrocks::RuntimeState*, unsigned long, starrocks::workgroup::WorkGroup const*) /root/starrocks/be/src/exec/pipeline/scan/chunk_source.cpp:67
    #11 0x9e168c7 in operator() /root/starrocks/be/src/exec/pipeline/scan/scan_operator.cpp:396
    #12 0x9e1c157 in __invoke_impl<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    #13 0x9e1c005 in __invoke_r<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #14 0x9e1be7a in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #15 0x992a271 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #16 0x9e519fa in starrocks::workgroup::ScanExecutor::worker_thread() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:67
    #17 0x9e5121f in operator() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:31
    #18 0x9e52c5d in __invoke_impl<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    #19 0x9e52925 in __invoke_r<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #20 0x9e5249a in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #21 0x992a271 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #22 0x1181f23d in starrocks::FunctionRunnable::run() /root/starrocks/be/src/util/threadpool.cpp:58
    #23 0x1181bff2 in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:553
    #24 0x11837c4b in void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:73
    #25 0x118375a4 in std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.
3.0/bits/invoke.h:95
    #26 0x1183699b in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /opt/gcc/usr/include/c++/10.3.0/functional:416
    #27 0x118352fd in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::operator()<, void>() /opt/gcc/usr/include/c++/10.3.0/functional:499
    #28 0x11832361 in void std::__invoke_impl<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    #29 0x1182fcc5 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110

```

Signed-off-by: trueeyu <[email protected]>
imay pushed a commit that referenced this pull request Jul 10, 2023
@trueeyu
[BugFix] Fix the mem leak of build in predicate for es external table (…
d6cbfd7 
…#26631) (#26814)

VExtLiteral is not deleted.

```
==76869==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x9793a37 in operator new(unsigned long) ../../.././libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x129dd44e in operator()<int const> /root/starrocks/be/src/exec/es/es_predicate.cpp:416
    #2 0x129dd817 in build_inpred_values<(starrocks::LogicalType)5, starrocks::EsPredicate::_build_in_predicate(const starrocks::Expr*, bool*)::<lambda(auto:29&)> > /root/starrocks/be/src/exec/es/es_predicate.cpp:366
    #3 0x129d7824 in starrocks::EsPredicate::_build_in_predicate(starrocks::Expr const*, bool*) /root/starrocks/be/src/exec/es/es_predicate.cpp:416
    #4 0x129d2f2d in starrocks::EsPredicate::_vec_build_disjuncts_list(starrocks::Expr const*) /root/starrocks/be/src/exec/es/es_predicate.cpp:185
    #5 0x129d2a53 in starrocks::EsPredicate::build_disjuncts_list() /root/starrocks/be/src/exec/es/es_predicate.cpp:148
    #6 0x128ab50d in starrocks::connector::ESDataSource::_build_conjuncts() /root/starrocks/be/src/connector/es_connector.cpp:123
    #7 0x128aa7c5 in starrocks::connector::ESDataSource::open(starrocks::RuntimeState*) /root/starrocks/be/src/connector/es_connector.cpp:89
    #8 0x9e33847 in starrocks::pipeline::ConnectorChunkSource::_open_data_source(starrocks::RuntimeState*) /root/starrocks/be/src/exec/pipeline/scan/connector_scan_operator.cpp:504
    #9 0x9e33ed0 in starrocks::pipeline::ConnectorChunkSource::_read_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/pipeline/scan/connector_scan_operator.cpp:533
    #10 0xabc8b65 in starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking(starrocks::RuntimeState*, unsigned long, starrocks::workgroup::WorkGroup const*) /root/starrocks/be/src/exec/pipeline/scan/chunk_source.cpp:67
    #11 0x9e168c7 in operator() /root/starrocks/be/src/exec/pipeline/scan/scan_operator.cpp:396
    #12 0x9e1c157 in __invoke_impl<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    #13 0x9e1c005 in __invoke_r<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #14 0x9e1be7a in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #15 0x992a271 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #16 0x9e519fa in starrocks::workgroup::ScanExecutor::worker_thread() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:67
    #17 0x9e5121f in operator() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:31
    #18 0x9e52c5d in __invoke_impl<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    #19 0x9e52925 in __invoke_r<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #20 0x9e5249a in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #21 0x992a271 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #22 0x1181f23d in starrocks::FunctionRunnable::run() /root/starrocks/be/src/util/threadpool.cpp:58
    #23 0x1181bff2 in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:553
    #24 0x11837c4b in void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:73
    #25 0x118375a4 in std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.
3.0/bits/invoke.h:95
    #26 0x1183699b in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /opt/gcc/usr/include/c++/10.3.0/functional:416
    #27 0x118352fd in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::operator()<, void>() /opt/gcc/usr/include/c++/10.3.0/functional:499
    #28 0x11832361 in void std::__invoke_impl<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    #29 0x1182fcc5 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110

```

Signed-off-by: trueeyu <[email protected]>
@sonarcloud
Copy link

sonarcloud bot commented Jul 12, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

imay pushed a commit that referenced this pull request Jul 27, 2023
@trueeyu
[BugFix] Fix the bug of RuntimeState use_after_free (#28055)
9635e6e 
Fixes [#3000](StarRocks/StarRocksTest#3000)

```
==2390==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7f3c65102a14 at pc 0x00000a0c5def bp 0x7f3c65101c40 sp 0x7f3c65101c38
READ of size 4 at 0x7f3c65102a14 thread T197 (load_rpc)
    #0 0xa0c5dee in starrocks::RuntimeState::chunk_size() const /root/starrocks/be/src/runtime/runtime_state.h:123
    #1 0xd0a9cdb in starrocks::ORCScanner::open() /root/starrocks/be/src/exec/orc_scanner.cpp:84
    #2 0x12b6dcff in starrocks::PInternalServiceImplBase<doris::PBackendService>::_get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*) /root/starrocks/be/src/service/internal_service.cpp:818
    #3 0x12b7a02f in starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}::operator()() const /root/starrocks/be/src/service/internal_service.cpp:757
    #4 0x12b89b89 in void std::__invoke_impl<void, starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}&>(std::__invoke_other, starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    #5 0x12b87158 in std::enable_if<is_invocable_r_v<void, starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}&>, std::enable_if>::type std::__invoke_r<void, starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}&>(void&&, (starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}&)...) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #6 0x12b83377 in std::_Function_handler<void (), starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #7 0x99f88bd in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #8 0xa436e73 in starrocks::FunctionRunnable::run() (/home/disk1/sr/be/lib/starrocks_be+0xa436e73)
    #9 0xa4339c8 in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:553
    #10 0xa450765 in void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:73
    #11 0xa450250 in std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:95
    #12 0xa44f199 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /opt/gcc/usr/include/c++/10.3.0/functional:416
    #13 0xa44e107 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::operator()<, void>() /opt/gcc/usr/include/c++/10.3.0/functional:499
    #14 0xa44bc61 in void std::__invoke_impl<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) (/home/disk1/sr/be/lib/starrocks_be+0xa44bc61)
    #15 0xa4485cf in std::enable_if<is_invocable_r_v<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #16 0xa4448ba in std::_Function_handler<void (), std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #17 0x99f88bd in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #18 0xa41b326 in starrocks::Thread::supervise_thread(void*) /root/starrocks/be/src/util/thread.cpp:364
    #19 0x7f3cc51b5ea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
    #20 0x7f3cc47d0b0c in clone (/lib64/libc.so.6+0xfeb0c)
```

Signed-off-by: trueeyu <[email protected]>
mergify bot pushed a commit that referenced this pull request Jul 27, 2023
@trueeyu @mergify
[BugFix] Fix the bug of RuntimeState use_after_free (#28055)
53624fd 
Fixes [#3000](StarRocks/StarRocksTest#3000)

```
==2390==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7f3c65102a14 at pc 0x00000a0c5def bp 0x7f3c65101c40 sp 0x7f3c65101c38
READ of size 4 at 0x7f3c65102a14 thread T197 (load_rpc)
    #0 0xa0c5dee in starrocks::RuntimeState::chunk_size() const /root/starrocks/be/src/runtime/runtime_state.h:123
    #1 0xd0a9cdb in starrocks::ORCScanner::open() /root/starrocks/be/src/exec/orc_scanner.cpp:84
    #2 0x12b6dcff in starrocks::PInternalServiceImplBase<doris::PBackendService>::_get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*) /root/starrocks/be/src/service/internal_service.cpp:818
    #3 0x12b7a02f in starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}::operator()() const /root/starrocks/be/src/service/internal_service.cpp:757
    #4 0x12b89b89 in void std::__invoke_impl<void, starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}&>(std::__invoke_other, starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    #5 0x12b87158 in std::enable_if<is_invocable_r_v<void, starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}&>, std::enable_if>::type std::__invoke_r<void, starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}&>(void&&, (starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}&)...) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #6 0x12b83377 in std::_Function_handler<void (), starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #7 0x99f88bd in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #8 0xa436e73 in starrocks::FunctionRunnable::run() (/home/disk1/sr/be/lib/starrocks_be+0xa436e73)
    #9 0xa4339c8 in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:553
    #10 0xa450765 in void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:73
    #11 0xa450250 in std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:95
    #12 0xa44f199 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /opt/gcc/usr/include/c++/10.3.0/functional:416
    #13 0xa44e107 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::operator()<, void>() /opt/gcc/usr/include/c++/10.3.0/functional:499
    #14 0xa44bc61 in void std::__invoke_impl<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) (/home/disk1/sr/be/lib/starrocks_be+0xa44bc61)
    #15 0xa4485cf in std::enable_if<is_invocable_r_v<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #16 0xa4448ba in std::_Function_handler<void (), std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #17 0x99f88bd in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #18 0xa41b326 in starrocks::Thread::supervise_thread(void*) /root/starrocks/be/src/util/thread.cpp:364
    #19 0x7f3cc51b5ea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
    #20 0x7f3cc47d0b0c in clone (/lib64/libc.so.6+0xfeb0c)
```

Signed-off-by: trueeyu <[email protected]>
(cherry picked from commit 9635e6e)
wanpengfei-git pushed a commit that referenced this pull request Jul 27, 2023
@trueeyu @wanpengfei-git
[BugFix] Fix the bug of RuntimeState use_after_free (#28055)
f31b4c9 
Fixes [#3000](StarRocks/StarRocksTest#3000)

```
==2390==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7f3c65102a14 at pc 0x00000a0c5def bp 0x7f3c65101c40 sp 0x7f3c65101c38
READ of size 4 at 0x7f3c65102a14 thread T197 (load_rpc)
    #0 0xa0c5dee in starrocks::RuntimeState::chunk_size() const /root/starrocks/be/src/runtime/runtime_state.h:123
    #1 0xd0a9cdb in starrocks::ORCScanner::open() /root/starrocks/be/src/exec/orc_scanner.cpp:84
    #2 0x12b6dcff in starrocks::PInternalServiceImplBase<doris::PBackendService>::_get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*) /root/starrocks/be/src/service/internal_service.cpp:818
    #3 0x12b7a02f in starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}::operator()() const /root/starrocks/be/src/service/internal_service.cpp:757
    #4 0x12b89b89 in void std::__invoke_impl<void, starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}&>(std::__invoke_other, starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    #5 0x12b87158 in std::enable_if<is_invocable_r_v<void, starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}&>, std::enable_if>::type std::__invoke_r<void, starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}&>(void&&, (starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}&)...) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #6 0x12b83377 in std::_Function_handler<void (), starrocks::PInternalServiceImplBase<doris::PBackendService>::get_file_schema(google::protobuf::RpcController*, starrocks::PGetFileSchemaRequest const*, starrocks::PGetFileSchemaResult*, google::protobuf::Closure*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #7 0x99f88bd in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #8 0xa436e73 in starrocks::FunctionRunnable::run() (/home/disk1/sr/be/lib/starrocks_be+0xa436e73)
    #9 0xa4339c8 in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:553
    #10 0xa450765 in void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:73
    #11 0xa450250 in std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:95
    #12 0xa44f199 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /opt/gcc/usr/include/c++/10.3.0/functional:416
    #13 0xa44e107 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::operator()<, void>() /opt/gcc/usr/include/c++/10.3.0/functional:499
    #14 0xa44bc61 in void std::__invoke_impl<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) (/home/disk1/sr/be/lib/starrocks_be+0xa44bc61)
    #15 0xa4485cf in std::enable_if<is_invocable_r_v<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #16 0xa4448ba in std::_Function_handler<void (), std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #17 0x99f88bd in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #18 0xa41b326 in starrocks::Thread::supervise_thread(void*) /root/starrocks/be/src/util/thread.cpp:364
    #19 0x7f3cc51b5ea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
    #20 0x7f3cc47d0b0c in clone (/lib64/libc.so.6+0xfeb0c)
```

Signed-off-by: trueeyu <[email protected]>
(cherry picked from commit 9635e6e)
imay pushed a commit that referenced this pull request Aug 9, 2023
@trueeyu
[BugFix] Fix use-after-free of SchemaLoadTrackingLogsScanner (#28929)
ffe721a 
Fixes [#issue](StarRocks/StarRocksTest#3087)

ss.str() is one local var, already released before execute
`fill_column_with_slot`.

```
crash log:
==2761==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000ca1280 at pc 0x000009bdfe27 bp 0x7f566078ad80 sp 0x7f566078ad78
READ of size 1 at 0x60b000ca1280 thread T243 (pip_wg_scan_io)
    #0 0x9bdfe26 in decltype (::new ((void*)(0)) unsigned char((declval<char&>)())) std::construct_at<unsigned char, char&>(unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/stl_construct.h:97
    #1 0x9bdfeaa in void std::allocator_traits<std::allocator<unsigned char> >::construct<unsigned char, char&>(std::allocator<unsigned char>&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:514
    #2 0x9bdf441 in void starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >::construct<unsigned char, char&>(unsigned char*, char&) /root/starrocks/be/src/util/raw_container.h:69
    #3 0x9bdca44 in std::enable_if<std::__and_<std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::__construct_helper<unsigned char, char&>::type>::value, void>::type std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_S_construct<unsigned char, char&>(starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:247
    #4 0x9bd9f73 in decltype (_S_construct({parm#1}, {parm#2}, (forward<char&>)({parm#3}))) std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::construct<unsigned char, char&>(starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:360
    #5 0x9bd4921 in unsigned char* std::__uninitialized_copy_a<char*, unsigned char*, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >(char*, char*, unsigned char*, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&) /opt/gcc/usr/include/c++/10.3.0/bits/stl_uninitialized.h:311
    #6 0x9bc9ad8 in void std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_M_range_insert<char*>(__gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*, std::forward_iterator_tag) /opt/gcc/usr/include/c++/10.3.0/bits/vector.tcc:778
    #7 0x9bbac6e in void std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_M_insert_dispatch<char*>(__gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*, std::__false_type) /opt/gcc/usr/include/c++/10.3.0/bits/stl_vector.h:1665
    #8 0x9ba647c in __gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > > std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::insert<char*, void>(__gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*) /opt/gcc/usr/include/c++/10.3.0/bits/stl_vector.h:1383
    #9 0x9b8358f in starrocks::BinaryColumnBase<unsigned int>::append(starrocks::Slice const&) /root/starrocks/be/src/column/binary_column.h:167
    #10 0xd24e6d1 in void starrocks::fill_data_column_with_slot<(starrocks::LogicalType)17>(starrocks::Column*, void*) /root/starrocks/be/src/exec/schema_scanner/schema_helper.h:116
    #11 0xd24e23d in void starrocks::fill_column_with_slot<(starrocks::LogicalType)17>(starrocks::Column*, void*) /root/starrocks/be/src/exec/schema_scanner/schema_helper.h:127
    #12 0xd2808d3 in starrocks::SchemaLoadTrackingLogsScanner::fill_chunk(std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/schema_scanner/schema_load_tracking_logs_scanner.cpp:112
    #13 0xd281dca in starrocks::SchemaLoadTrackingLogsScanner::get_next(std::shared_ptr<starrocks::Chunk>*, bool*) /root/starrocks/be/src/exec/schema_scanner/schema_load_tracking_logs_scanner.cpp:160
    #14 0xd3f5dd5 in starrocks::pipeline::OlapSchemaChunkSource::_read_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/pipeline/scan/olap_schema_chunk_source.cpp:132
    #15 0xd3d0b33 in starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking(starrocks::RuntimeState*, unsigned long, starrocks::workgroup::WorkGroup const*) /root/starrocks/be/src/exec/pipeline/scan/chunk_source.cpp:67
    #16 0xc3deac7 in operator() /root/starrocks/be/src/exec/pipeline/scan/scan_operator.cpp:400
    #17 0xc3e4345 in __invoke_impl<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    #18 0xc3e41f3 in __invoke_r<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #19 0xc3e4068 in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #20 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #21 0xc916608 in starrocks::workgroup::ScanExecutor::worker_thread() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:70
    #22 0xc915e2d in operator() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:34
    #23 0xc917de3 in __invoke_impl<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    #24 0xc9179b2 in __invoke_r<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #25 0xc91734f in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #26 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #27 0xa533efd in starrocks::FunctionRunnable::run() /root/starrocks/be/src/util/threadpool.cpp:58
    #28 0xa5309bc in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:553
    #29 0xa54d7ef in void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:73
    #30 0xa54d2da in std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:95
    #31 0xa54c223 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /opt/gcc/usr/include/c++/10.3.0/functional:416
    #32 0xa54b191 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::operator()<, void>() /opt/gcc/usr/include/c++/10.3.0/functional:499
    #33 0xa548ceb in void std::__invoke_impl<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) (/home/disk1/sr/be/lib/starrocks_be+0xa548ceb)
    #34 0xa545659 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #35 0xa541944 in std::_Function_handler<void (), std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #36 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #37 0xa518286 in starrocks::Thread::supervise_thread(void*) /root/starrocks/be/src/util/thread.cpp:364
    #38 0x7f56ec0b0ea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
    #39 0x7f56eb6cbb0c in clone (/lib64/libc.so.6+0xfeb0c)
```

Signed-off-by: trueeyu <[email protected]>
mergify bot pushed a commit that referenced this pull request Aug 9, 2023
@trueeyu @mergify
[BugFix] Fix use-after-free of SchemaLoadTrackingLogsScanner (#28929)
286ac40 
Fixes [#issue](StarRocks/StarRocksTest#3087)

ss.str() is one local var, already released before execute
`fill_column_with_slot`.

```
crash log:
==2761==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000ca1280 at pc 0x000009bdfe27 bp 0x7f566078ad80 sp 0x7f566078ad78
READ of size 1 at 0x60b000ca1280 thread T243 (pip_wg_scan_io)
    #0 0x9bdfe26 in decltype (::new ((void*)(0)) unsigned char((declval<char&>)())) std::construct_at<unsigned char, char&>(unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/stl_construct.h:97
    #1 0x9bdfeaa in void std::allocator_traits<std::allocator<unsigned char> >::construct<unsigned char, char&>(std::allocator<unsigned char>&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:514
    #2 0x9bdf441 in void starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >::construct<unsigned char, char&>(unsigned char*, char&) /root/starrocks/be/src/util/raw_container.h:69
    #3 0x9bdca44 in std::enable_if<std::__and_<std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::__construct_helper<unsigned char, char&>::type>::value, void>::type std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_S_construct<unsigned char, char&>(starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:247
    #4 0x9bd9f73 in decltype (_S_construct({parm#1}, {parm#2}, (forward<char&>)({parm#3}))) std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::construct<unsigned char, char&>(starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:360
    #5 0x9bd4921 in unsigned char* std::__uninitialized_copy_a<char*, unsigned char*, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >(char*, char*, unsigned char*, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&) /opt/gcc/usr/include/c++/10.3.0/bits/stl_uninitialized.h:311
    #6 0x9bc9ad8 in void std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_M_range_insert<char*>(__gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*, std::forward_iterator_tag) /opt/gcc/usr/include/c++/10.3.0/bits/vector.tcc:778
    #7 0x9bbac6e in void std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_M_insert_dispatch<char*>(__gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*, std::__false_type) /opt/gcc/usr/include/c++/10.3.0/bits/stl_vector.h:1665
    #8 0x9ba647c in __gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > > std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::insert<char*, void>(__gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*) /opt/gcc/usr/include/c++/10.3.0/bits/stl_vector.h:1383
    #9 0x9b8358f in starrocks::BinaryColumnBase<unsigned int>::append(starrocks::Slice const&) /root/starrocks/be/src/column/binary_column.h:167
    #10 0xd24e6d1 in void starrocks::fill_data_column_with_slot<(starrocks::LogicalType)17>(starrocks::Column*, void*) /root/starrocks/be/src/exec/schema_scanner/schema_helper.h:116
    #11 0xd24e23d in void starrocks::fill_column_with_slot<(starrocks::LogicalType)17>(starrocks::Column*, void*) /root/starrocks/be/src/exec/schema_scanner/schema_helper.h:127
    #12 0xd2808d3 in starrocks::SchemaLoadTrackingLogsScanner::fill_chunk(std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/schema_scanner/schema_load_tracking_logs_scanner.cpp:112
    #13 0xd281dca in starrocks::SchemaLoadTrackingLogsScanner::get_next(std::shared_ptr<starrocks::Chunk>*, bool*) /root/starrocks/be/src/exec/schema_scanner/schema_load_tracking_logs_scanner.cpp:160
    #14 0xd3f5dd5 in starrocks::pipeline::OlapSchemaChunkSource::_read_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/pipeline/scan/olap_schema_chunk_source.cpp:132
    #15 0xd3d0b33 in starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking(starrocks::RuntimeState*, unsigned long, starrocks::workgroup::WorkGroup const*) /root/starrocks/be/src/exec/pipeline/scan/chunk_source.cpp:67
    #16 0xc3deac7 in operator() /root/starrocks/be/src/exec/pipeline/scan/scan_operator.cpp:400
    #17 0xc3e4345 in __invoke_impl<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    #18 0xc3e41f3 in __invoke_r<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #19 0xc3e4068 in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #20 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #21 0xc916608 in starrocks::workgroup::ScanExecutor::worker_thread() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:70
    #22 0xc915e2d in operator() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:34
    #23 0xc917de3 in __invoke_impl<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    #24 0xc9179b2 in __invoke_r<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #25 0xc91734f in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #26 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #27 0xa533efd in starrocks::FunctionRunnable::run() /root/starrocks/be/src/util/threadpool.cpp:58
    #28 0xa5309bc in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:553
    #29 0xa54d7ef in void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:73
    #30 0xa54d2da in std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:95
    #31 0xa54c223 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /opt/gcc/usr/include/c++/10.3.0/functional:416
    #32 0xa54b191 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::operator()<, void>() /opt/gcc/usr/include/c++/10.3.0/functional:499
    #33 0xa548ceb in void std::__invoke_impl<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) (/home/disk1/sr/be/lib/starrocks_be+0xa548ceb)
    #34 0xa545659 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    #35 0xa541944 in std::_Function_handler<void (), std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    #36 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    #37 0xa518286 in starrocks::Thread::supervise_thread(void*) /root/starrocks/be/src/util/thread.cpp:364
    #38 0x7f56ec0b0ea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
    #39 0x7f56eb6cbb0c in clone (/lib64/libc.so.6+0xfeb0c)
```

Signed-off-by: trueeyu <[email protected]>
(cherry picked from commit ffe721a)

# Conflicts:
#	be/src/exec/schema_scanner/schema_load_tracking_logs_scanner.cpp
Seaven pushed a commit to Seaven/starrocks that referenced this pull request Aug 16, 2023
@trueeyu @Seaven
[BugFix] Fix use-after-free of SchemaLoadTrackingLogsScanner (StarRoc…
b6cb58a 
…ks#28929)

Fixes [#issue](StarRocks/StarRocksTest#3087)

ss.str() is one local var, already released before execute
`fill_column_with_slot`.

```
crash log:
==2761==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000ca1280 at pc 0x000009bdfe27 bp 0x7f566078ad80 sp 0x7f566078ad78
READ of size 1 at 0x60b000ca1280 thread T243 (pip_wg_scan_io)
    #0 0x9bdfe26 in decltype (::new ((void*)(0)) unsigned char((declval<char&>)())) std::construct_at<unsigned char, char&>(unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/stl_construct.h:97
    #1 0x9bdfeaa in void std::allocator_traits<std::allocator<unsigned char> >::construct<unsigned char, char&>(std::allocator<unsigned char>&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:514
    #2 0x9bdf441 in void starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >::construct<unsigned char, char&>(unsigned char*, char&) /root/starrocks/be/src/util/raw_container.h:69
    #3 0x9bdca44 in std::enable_if<std::__and_<std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::__construct_helper<unsigned char, char&>::type>::value, void>::type std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_S_construct<unsigned char, char&>(starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:247
    StarRocks#4 0x9bd9f73 in decltype (_S_construct({parm#1}, {parm#2}, (forward<char&>)({parm#3}))) std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::construct<unsigned char, char&>(starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:360
    StarRocks#5 0x9bd4921 in unsigned char* std::__uninitialized_copy_a<char*, unsigned char*, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >(char*, char*, unsigned char*, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&) /opt/gcc/usr/include/c++/10.3.0/bits/stl_uninitialized.h:311
    StarRocks#6 0x9bc9ad8 in void std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_M_range_insert<char*>(__gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*, std::forward_iterator_tag) /opt/gcc/usr/include/c++/10.3.0/bits/vector.tcc:778
    StarRocks#7 0x9bbac6e in void std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_M_insert_dispatch<char*>(__gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*, std::__false_type) /opt/gcc/usr/include/c++/10.3.0/bits/stl_vector.h:1665
    StarRocks#8 0x9ba647c in __gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > > std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::insert<char*, void>(__gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*) /opt/gcc/usr/include/c++/10.3.0/bits/stl_vector.h:1383
    StarRocks#9 0x9b8358f in starrocks::BinaryColumnBase<unsigned int>::append(starrocks::Slice const&) /root/starrocks/be/src/column/binary_column.h:167
    StarRocks#10 0xd24e6d1 in void starrocks::fill_data_column_with_slot<(starrocks::LogicalType)17>(starrocks::Column*, void*) /root/starrocks/be/src/exec/schema_scanner/schema_helper.h:116
    StarRocks#11 0xd24e23d in void starrocks::fill_column_with_slot<(starrocks::LogicalType)17>(starrocks::Column*, void*) /root/starrocks/be/src/exec/schema_scanner/schema_helper.h:127
    StarRocks#12 0xd2808d3 in starrocks::SchemaLoadTrackingLogsScanner::fill_chunk(std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/schema_scanner/schema_load_tracking_logs_scanner.cpp:112
    StarRocks#13 0xd281dca in starrocks::SchemaLoadTrackingLogsScanner::get_next(std::shared_ptr<starrocks::Chunk>*, bool*) /root/starrocks/be/src/exec/schema_scanner/schema_load_tracking_logs_scanner.cpp:160
    StarRocks#14 0xd3f5dd5 in starrocks::pipeline::OlapSchemaChunkSource::_read_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/pipeline/scan/olap_schema_chunk_source.cpp:132
    StarRocks#15 0xd3d0b33 in starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking(starrocks::RuntimeState*, unsigned long, starrocks::workgroup::WorkGroup const*) /root/starrocks/be/src/exec/pipeline/scan/chunk_source.cpp:67
    StarRocks#16 0xc3deac7 in operator() /root/starrocks/be/src/exec/pipeline/scan/scan_operator.cpp:400
    StarRocks#17 0xc3e4345 in __invoke_impl<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    StarRocks#18 0xc3e41f3 in __invoke_r<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#19 0xc3e4068 in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#20 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#21 0xc916608 in starrocks::workgroup::ScanExecutor::worker_thread() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:70
    StarRocks#22 0xc915e2d in operator() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:34
    StarRocks#23 0xc917de3 in __invoke_impl<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    StarRocks#24 0xc9179b2 in __invoke_r<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#25 0xc91734f in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#26 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#27 0xa533efd in starrocks::FunctionRunnable::run() /root/starrocks/be/src/util/threadpool.cpp:58
    StarRocks#28 0xa5309bc in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:553
    StarRocks#29 0xa54d7ef in void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:73
    StarRocks#30 0xa54d2da in std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:95
    StarRocks#31 0xa54c223 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /opt/gcc/usr/include/c++/10.3.0/functional:416
    StarRocks#32 0xa54b191 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::operator()<, void>() /opt/gcc/usr/include/c++/10.3.0/functional:499
    StarRocks#33 0xa548ceb in void std::__invoke_impl<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) (/home/disk1/sr/be/lib/starrocks_be+0xa548ceb)
    StarRocks#34 0xa545659 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#35 0xa541944 in std::_Function_handler<void (), std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#36 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#37 0xa518286 in starrocks::Thread::supervise_thread(void*) /root/starrocks/be/src/util/thread.cpp:364
    StarRocks#38 0x7f56ec0b0ea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
    StarRocks#39 0x7f56eb6cbb0c in clone (/lib64/libc.so.6+0xfeb0c)
```

Signed-off-by: trueeyu <[email protected]>
Signed-off-by: Seaven <[email protected]>
sevev added a commit to sevev/starrocks that referenced this pull request Aug 22, 2023
@sevev
Merge pull request StarRocks#5 from sevev/zq-lsc-main-test
9c22f2f 
merge main branch
mengtingluo pushed a commit to mengtingluo/starrocks that referenced this pull request Aug 22, 2023
@trueeyu @mengtingluo
[BugFix] Fix use-after-free of SchemaLoadTrackingLogsScanner (StarRoc…
1969aba 
…ks#28929)

Fixes [#issue](https://github.com/StarRocks/StarRocksTest/issues/3087)

ss.str() is one local var, already released before execute
`fill_column_with_slot`.

```
crash log:
==2761==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000ca1280 at pc 0x000009bdfe27 bp 0x7f566078ad80 sp 0x7f566078ad78
READ of size 1 at 0x60b000ca1280 thread T243 (pip_wg_scan_io)
    #0 0x9bdfe26 in decltype (::new ((void*)(0)) unsigned char((declval<char&>)())) std::construct_at<unsigned char, char&>(unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/stl_construct.h:97
    StarRocks#1 0x9bdfeaa in void std::allocator_traits<std::allocator<unsigned char> >::construct<unsigned char, char&>(std::allocator<unsigned char>&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:514
    StarRocks#2 0x9bdf441 in void starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >::construct<unsigned char, char&>(unsigned char*, char&) /root/starrocks/be/src/util/raw_container.h:69
    StarRocks#3 0x9bdca44 in std::enable_if<std::__and_<std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::__construct_helper<unsigned char, char&>::type>::value, void>::type std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_S_construct<unsigned char, char&>(starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:247
    StarRocks#4 0x9bd9f73 in decltype (_S_construct({parm#1}, {parm#2}, (forward<char&>)({parm#3}))) std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::construct<unsigned char, char&>(starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:360
    StarRocks#5 0x9bd4921 in unsigned char* std::__uninitialized_copy_a<char*, unsigned char*, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >(char*, char*, unsigned char*, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&) /opt/gcc/usr/include/c++/10.3.0/bits/stl_uninitialized.h:311
    StarRocks#6 0x9bc9ad8 in void std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_M_range_insert<char*>(__gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*, std::forward_iterator_tag) /opt/gcc/usr/include/c++/10.3.0/bits/vector.tcc:778
    StarRocks#7 0x9bbac6e in void std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_M_insert_dispatch<char*>(__gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*, std::__false_type) /opt/gcc/usr/include/c++/10.3.0/bits/stl_vector.h:1665
    StarRocks#8 0x9ba647c in __gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > > std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::insert<char*, void>(__gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*) /opt/gcc/usr/include/c++/10.3.0/bits/stl_vector.h:1383
    StarRocks#9 0x9b8358f in starrocks::BinaryColumnBase<unsigned int>::append(starrocks::Slice const&) /root/starrocks/be/src/column/binary_column.h:167
    StarRocks#10 0xd24e6d1 in void starrocks::fill_data_column_with_slot<(starrocks::LogicalType)17>(starrocks::Column*, void*) /root/starrocks/be/src/exec/schema_scanner/schema_helper.h:116
    StarRocks#11 0xd24e23d in void starrocks::fill_column_with_slot<(starrocks::LogicalType)17>(starrocks::Column*, void*) /root/starrocks/be/src/exec/schema_scanner/schema_helper.h:127
    StarRocks#12 0xd2808d3 in starrocks::SchemaLoadTrackingLogsScanner::fill_chunk(std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/schema_scanner/schema_load_tracking_logs_scanner.cpp:112
    StarRocks#13 0xd281dca in starrocks::SchemaLoadTrackingLogsScanner::get_next(std::shared_ptr<starrocks::Chunk>*, bool*) /root/starrocks/be/src/exec/schema_scanner/schema_load_tracking_logs_scanner.cpp:160
    StarRocks#14 0xd3f5dd5 in starrocks::pipeline::OlapSchemaChunkSource::_read_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/pipeline/scan/olap_schema_chunk_source.cpp:132
    StarRocks#15 0xd3d0b33 in starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking(starrocks::RuntimeState*, unsigned long, starrocks::workgroup::WorkGroup const*) /root/starrocks/be/src/exec/pipeline/scan/chunk_source.cpp:67
    StarRocks#16 0xc3deac7 in operator() /root/starrocks/be/src/exec/pipeline/scan/scan_operator.cpp:400
    StarRocks#17 0xc3e4345 in __invoke_impl<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    StarRocks#18 0xc3e41f3 in __invoke_r<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#19 0xc3e4068 in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#20 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#21 0xc916608 in starrocks::workgroup::ScanExecutor::worker_thread() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:70
    StarRocks#22 0xc915e2d in operator() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:34
    StarRocks#23 0xc917de3 in __invoke_impl<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    StarRocks#24 0xc9179b2 in __invoke_r<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#25 0xc91734f in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#26 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#27 0xa533efd in starrocks::FunctionRunnable::run() /root/starrocks/be/src/util/threadpool.cpp:58
    StarRocks#28 0xa5309bc in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:553
    StarRocks#29 0xa54d7ef in void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:73
    StarRocks#30 0xa54d2da in std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:95
    StarRocks#31 0xa54c223 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /opt/gcc/usr/include/c++/10.3.0/functional:416
    StarRocks#32 0xa54b191 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::operator()<, void>() /opt/gcc/usr/include/c++/10.3.0/functional:499
    StarRocks#33 0xa548ceb in void std::__invoke_impl<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) (/home/disk1/sr/be/lib/starrocks_be+0xa548ceb)
    StarRocks#34 0xa545659 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#35 0xa541944 in std::_Function_handler<void (), std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#36 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#37 0xa518286 in starrocks::Thread::supervise_thread(void*) /root/starrocks/be/src/util/thread.cpp:364
    StarRocks#38 0x7f56ec0b0ea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
    StarRocks#39 0x7f56eb6cbb0c in clone (/lib64/libc.so.6+0xfeb0c)
```

Signed-off-by: trueeyu <[email protected]>
trueeyu added a commit to trueeyu/starrocks that referenced this pull request Aug 22, 2023
@trueeyu
[BugFix] Fix the mem leak of build in predicate for es external table (…
0107091 
…StarRocks#26631) (StarRocks#26805)

VExtLiteral is not deleted.

```
==76869==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x9793a37 in operator new(unsigned long) ../../.././libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x129dd44e in operator()<int const> /root/starrocks/be/src/exec/es/es_predicate.cpp:416
    StarRocks#2 0x129dd817 in build_inpred_values<(starrocks::LogicalType)5, starrocks::EsPredicate::_build_in_predicate(const starrocks::Expr*, bool*)::<lambda(auto:29&)> > /root/starrocks/be/src/exec/es/es_predicate.cpp:366
    StarRocks#3 0x129d7824 in starrocks::EsPredicate::_build_in_predicate(starrocks::Expr const*, bool*) /root/starrocks/be/src/exec/es/es_predicate.cpp:416
    StarRocks#4 0x129d2f2d in starrocks::EsPredicate::_vec_build_disjuncts_list(starrocks::Expr const*) /root/starrocks/be/src/exec/es/es_predicate.cpp:185
    StarRocks#5 0x129d2a53 in starrocks::EsPredicate::build_disjuncts_list() /root/starrocks/be/src/exec/es/es_predicate.cpp:148
    StarRocks#6 0x128ab50d in starrocks::connector::ESDataSource::_build_conjuncts() /root/starrocks/be/src/connector/es_connector.cpp:123
    StarRocks#7 0x128aa7c5 in starrocks::connector::ESDataSource::open(starrocks::RuntimeState*) /root/starrocks/be/src/connector/es_connector.cpp:89
    StarRocks#8 0x9e33847 in starrocks::pipeline::ConnectorChunkSource::_open_data_source(starrocks::RuntimeState*) /root/starrocks/be/src/exec/pipeline/scan/connector_scan_operator.cpp:504
    StarRocks#9 0x9e33ed0 in starrocks::pipeline::ConnectorChunkSource::_read_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/pipeline/scan/connector_scan_operator.cpp:533
    StarRocks#10 0xabc8b65 in starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking(starrocks::RuntimeState*, unsigned long, starrocks::workgroup::WorkGroup const*) /root/starrocks/be/src/exec/pipeline/scan/chunk_source.cpp:67
    StarRocks#11 0x9e168c7 in operator() /root/starrocks/be/src/exec/pipeline/scan/scan_operator.cpp:396
    StarRocks#12 0x9e1c157 in __invoke_impl<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    StarRocks#13 0x9e1c005 in __invoke_r<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#14 0x9e1be7a in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#15 0x992a271 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#16 0x9e519fa in starrocks::workgroup::ScanExecutor::worker_thread() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:67
    StarRocks#17 0x9e5121f in operator() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:31
    StarRocks#18 0x9e52c5d in __invoke_impl<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    StarRocks#19 0x9e52925 in __invoke_r<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#20 0x9e5249a in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#21 0x992a271 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#22 0x1181f23d in starrocks::FunctionRunnable::run() /root/starrocks/be/src/util/threadpool.cpp:58
    StarRocks#23 0x1181bff2 in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:553
    StarRocks#24 0x11837c4b in void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:73
    StarRocks#25 0x118375a4 in std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.
3.0/bits/invoke.h:95
    StarRocks#26 0x1183699b in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /opt/gcc/usr/include/c++/10.3.0/functional:416
    StarRocks#27 0x118352fd in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::operator()<, void>() /opt/gcc/usr/include/c++/10.3.0/functional:499
    StarRocks#28 0x11832361 in void std::__invoke_impl<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    StarRocks#29 0x1182fcc5 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110

```

Signed-off-by: trueeyu <[email protected]>
mengtingluo pushed a commit to mengtingluo/starrocks that referenced this pull request Aug 22, 2023
@trueeyu @mengtingluo
[BugFix] Fix use-after-free of SchemaLoadTrackingLogsScanner (StarRoc…
7250dae 
…ks#28929)

Fixes [#issue](https://github.com/StarRocks/StarRocksTest/issues/3087)

ss.str() is one local var, already released before execute
`fill_column_with_slot`.

```
crash log:
==2761==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000ca1280 at pc 0x000009bdfe27 bp 0x7f566078ad80 sp 0x7f566078ad78
READ of size 1 at 0x60b000ca1280 thread T243 (pip_wg_scan_io)
    #0 0x9bdfe26 in decltype (::new ((void*)(0)) unsigned char((declval<char&>)())) std::construct_at<unsigned char, char&>(unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/stl_construct.h:97
    StarRocks#1 0x9bdfeaa in void std::allocator_traits<std::allocator<unsigned char> >::construct<unsigned char, char&>(std::allocator<unsigned char>&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:514
    StarRocks#2 0x9bdf441 in void starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >::construct<unsigned char, char&>(unsigned char*, char&) /root/starrocks/be/src/util/raw_container.h:69
    StarRocks#3 0x9bdca44 in std::enable_if<std::__and_<std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::__construct_helper<unsigned char, char&>::type>::value, void>::type std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_S_construct<unsigned char, char&>(starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:247
    StarRocks#4 0x9bd9f73 in decltype (_S_construct({parm#1}, {parm#2}, (forward<char&>)({parm#3}))) std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::construct<unsigned char, char&>(starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:360
    StarRocks#5 0x9bd4921 in unsigned char* std::__uninitialized_copy_a<char*, unsigned char*, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >(char*, char*, unsigned char*, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&) /opt/gcc/usr/include/c++/10.3.0/bits/stl_uninitialized.h:311
    StarRocks#6 0x9bc9ad8 in void std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_M_range_insert<char*>(__gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*, std::forward_iterator_tag) /opt/gcc/usr/include/c++/10.3.0/bits/vector.tcc:778
    StarRocks#7 0x9bbac6e in void std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_M_insert_dispatch<char*>(__gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*, std::__false_type) /opt/gcc/usr/include/c++/10.3.0/bits/stl_vector.h:1665
    StarRocks#8 0x9ba647c in __gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > > std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::insert<char*, void>(__gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*) /opt/gcc/usr/include/c++/10.3.0/bits/stl_vector.h:1383
    StarRocks#9 0x9b8358f in starrocks::BinaryColumnBase<unsigned int>::append(starrocks::Slice const&) /root/starrocks/be/src/column/binary_column.h:167
    StarRocks#10 0xd24e6d1 in void starrocks::fill_data_column_with_slot<(starrocks::LogicalType)17>(starrocks::Column*, void*) /root/starrocks/be/src/exec/schema_scanner/schema_helper.h:116
    StarRocks#11 0xd24e23d in void starrocks::fill_column_with_slot<(starrocks::LogicalType)17>(starrocks::Column*, void*) /root/starrocks/be/src/exec/schema_scanner/schema_helper.h:127
    StarRocks#12 0xd2808d3 in starrocks::SchemaLoadTrackingLogsScanner::fill_chunk(std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/schema_scanner/schema_load_tracking_logs_scanner.cpp:112
    StarRocks#13 0xd281dca in starrocks::SchemaLoadTrackingLogsScanner::get_next(std::shared_ptr<starrocks::Chunk>*, bool*) /root/starrocks/be/src/exec/schema_scanner/schema_load_tracking_logs_scanner.cpp:160
    StarRocks#14 0xd3f5dd5 in starrocks::pipeline::OlapSchemaChunkSource::_read_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/pipeline/scan/olap_schema_chunk_source.cpp:132
    StarRocks#15 0xd3d0b33 in starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking(starrocks::RuntimeState*, unsigned long, starrocks::workgroup::WorkGroup const*) /root/starrocks/be/src/exec/pipeline/scan/chunk_source.cpp:67
    StarRocks#16 0xc3deac7 in operator() /root/starrocks/be/src/exec/pipeline/scan/scan_operator.cpp:400
    StarRocks#17 0xc3e4345 in __invoke_impl<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    StarRocks#18 0xc3e41f3 in __invoke_r<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#19 0xc3e4068 in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#20 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#21 0xc916608 in starrocks::workgroup::ScanExecutor::worker_thread() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:70
    StarRocks#22 0xc915e2d in operator() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:34
    StarRocks#23 0xc917de3 in __invoke_impl<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    StarRocks#24 0xc9179b2 in __invoke_r<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#25 0xc91734f in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#26 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#27 0xa533efd in starrocks::FunctionRunnable::run() /root/starrocks/be/src/util/threadpool.cpp:58
    StarRocks#28 0xa5309bc in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:553
    StarRocks#29 0xa54d7ef in void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:73
    StarRocks#30 0xa54d2da in std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:95
    StarRocks#31 0xa54c223 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /opt/gcc/usr/include/c++/10.3.0/functional:416
    StarRocks#32 0xa54b191 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::operator()<, void>() /opt/gcc/usr/include/c++/10.3.0/functional:499
    StarRocks#33 0xa548ceb in void std::__invoke_impl<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) (/home/disk1/sr/be/lib/starrocks_be+0xa548ceb)
    StarRocks#34 0xa545659 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#35 0xa541944 in std::_Function_handler<void (), std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#36 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#37 0xa518286 in starrocks::Thread::supervise_thread(void*) /root/starrocks/be/src/util/thread.cpp:364
    StarRocks#38 0x7f56ec0b0ea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
    StarRocks#39 0x7f56eb6cbb0c in clone (/lib64/libc.so.6+0xfeb0c)
```

Signed-off-by: trueeyu <[email protected]>
Jay-ju pushed a commit to Jay-ju/starrocks that referenced this pull request Sep 7, 2023
@trueeyu @Jay-ju
[BugFix] Fix use-after-free of SchemaLoadTrackingLogsScanner (StarRoc…
a957133 
…ks#28929)

Fixes [#issue](https://github.com/StarRocks/StarRocksTest/issues/3087)

ss.str() is one local var, already released before execute
`fill_column_with_slot`.

```
crash log:
==2761==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000ca1280 at pc 0x000009bdfe27 bp 0x7f566078ad80 sp 0x7f566078ad78
READ of size 1 at 0x60b000ca1280 thread T243 (pip_wg_scan_io)
    #0 0x9bdfe26 in decltype (::new ((void*)(0)) unsigned char((declval<char&>)())) std::construct_at<unsigned char, char&>(unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/stl_construct.h:97
    StarRocks#1 0x9bdfeaa in void std::allocator_traits<std::allocator<unsigned char> >::construct<unsigned char, char&>(std::allocator<unsigned char>&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:514
    StarRocks#2 0x9bdf441 in void starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >::construct<unsigned char, char&>(unsigned char*, char&) /root/starrocks/be/src/util/raw_container.h:69
    StarRocks#3 0x9bdca44 in std::enable_if<std::__and_<std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::__construct_helper<unsigned char, char&>::type>::value, void>::type std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_S_construct<unsigned char, char&>(starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:247
    StarRocks#4 0x9bd9f73 in decltype (_S_construct({parm#1}, {parm#2}, (forward<char&>)({parm#3}))) std::allocator_traits<starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::construct<unsigned char, char&>(starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&, unsigned char*, char&) /opt/gcc/usr/include/c++/10.3.0/bits/alloc_traits.h:360
    StarRocks#5 0x9bd4921 in unsigned char* std::__uninitialized_copy_a<char*, unsigned char*, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >(char*, char*, unsigned char*, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> >&) /opt/gcc/usr/include/c++/10.3.0/bits/stl_uninitialized.h:311
    StarRocks#6 0x9bc9ad8 in void std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_M_range_insert<char*>(__gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*, std::forward_iterator_tag) /opt/gcc/usr/include/c++/10.3.0/bits/vector.tcc:778
    StarRocks#7 0x9bbac6e in void std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::_M_insert_dispatch<char*>(__gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*, std::__false_type) /opt/gcc/usr/include/c++/10.3.0/bits/stl_vector.h:1665
    StarRocks#8 0x9ba647c in __gnu_cxx::__normal_iterator<unsigned char*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > > std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > >::insert<char*, void>(__gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, starrocks::raw::RawAllocator<unsigned char, 16ul, std::allocator<unsigned char> > > >, char*, char*) /opt/gcc/usr/include/c++/10.3.0/bits/stl_vector.h:1383
    StarRocks#9 0x9b8358f in starrocks::BinaryColumnBase<unsigned int>::append(starrocks::Slice const&) /root/starrocks/be/src/column/binary_column.h:167
    StarRocks#10 0xd24e6d1 in void starrocks::fill_data_column_with_slot<(starrocks::LogicalType)17>(starrocks::Column*, void*) /root/starrocks/be/src/exec/schema_scanner/schema_helper.h:116
    StarRocks#11 0xd24e23d in void starrocks::fill_column_with_slot<(starrocks::LogicalType)17>(starrocks::Column*, void*) /root/starrocks/be/src/exec/schema_scanner/schema_helper.h:127
    StarRocks#12 0xd2808d3 in starrocks::SchemaLoadTrackingLogsScanner::fill_chunk(std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/schema_scanner/schema_load_tracking_logs_scanner.cpp:112
    StarRocks#13 0xd281dca in starrocks::SchemaLoadTrackingLogsScanner::get_next(std::shared_ptr<starrocks::Chunk>*, bool*) /root/starrocks/be/src/exec/schema_scanner/schema_load_tracking_logs_scanner.cpp:160
    StarRocks#14 0xd3f5dd5 in starrocks::pipeline::OlapSchemaChunkSource::_read_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/pipeline/scan/olap_schema_chunk_source.cpp:132
    StarRocks#15 0xd3d0b33 in starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking(starrocks::RuntimeState*, unsigned long, starrocks::workgroup::WorkGroup const*) /root/starrocks/be/src/exec/pipeline/scan/chunk_source.cpp:67
    StarRocks#16 0xc3deac7 in operator() /root/starrocks/be/src/exec/pipeline/scan/scan_operator.cpp:400
    StarRocks#17 0xc3e4345 in __invoke_impl<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    StarRocks#18 0xc3e41f3 in __invoke_r<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#19 0xc3e4068 in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#20 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#21 0xc916608 in starrocks::workgroup::ScanExecutor::worker_thread() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:70
    StarRocks#22 0xc915e2d in operator() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:34
    StarRocks#23 0xc917de3 in __invoke_impl<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    StarRocks#24 0xc9179b2 in __invoke_r<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#25 0xc91734f in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#26 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#27 0xa533efd in starrocks::FunctionRunnable::run() /root/starrocks/be/src/util/threadpool.cpp:58
    StarRocks#28 0xa5309bc in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:553
    StarRocks#29 0xa54d7ef in void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:73
    StarRocks#30 0xa54d2da in std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:95
    StarRocks#31 0xa54c223 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /opt/gcc/usr/include/c++/10.3.0/functional:416
    StarRocks#32 0xa54b191 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::operator()<, void>() /opt/gcc/usr/include/c++/10.3.0/functional:499
    StarRocks#33 0xa548ceb in void std::__invoke_impl<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) (/home/disk1/sr/be/lib/starrocks_be+0xa548ceb)
    StarRocks#34 0xa545659 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#35 0xa541944 in std::_Function_handler<void (), std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#36 0x9ae6d83 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#37 0xa518286 in starrocks::Thread::supervise_thread(void*) /root/starrocks/be/src/util/thread.cpp:364
    StarRocks#38 0x7f56ec0b0ea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
    StarRocks#39 0x7f56eb6cbb0c in clone (/lib64/libc.so.6+0xfeb0c)
```

Signed-off-by: trueeyu <[email protected]>
southernriver pushed a commit to southernriver/starrocks that referenced this pull request Nov 28, 2023
@trueeyu @guangxuCheng
[BugFix] Fix the mem leak of build in predicate for es external table (…
04ccf82 
…StarRocks#26631) (StarRocks#26805)

VExtLiteral is not deleted.

```
==76869==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 40 byte(s) in 1 object(s) allocated from:
    #0 0x9793a37 in operator new(unsigned long) ../../.././libsanitizer/asan/asan_new_delete.cpp:99
    StarRocks#1 0x129dd44e in operator()<int const> /root/starrocks/be/src/exec/es/es_predicate.cpp:416
    StarRocks#2 0x129dd817 in build_inpred_values<(starrocks::LogicalType)5, starrocks::EsPredicate::_build_in_predicate(const starrocks::Expr*, bool*)::<lambda(auto:29&)> > /root/starrocks/be/src/exec/es/es_predicate.cpp:366
    StarRocks#3 0x129d7824 in starrocks::EsPredicate::_build_in_predicate(starrocks::Expr const*, bool*) /root/starrocks/be/src/exec/es/es_predicate.cpp:416
    StarRocks#4 0x129d2f2d in starrocks::EsPredicate::_vec_build_disjuncts_list(starrocks::Expr const*) /root/starrocks/be/src/exec/es/es_predicate.cpp:185
    StarRocks#5 0x129d2a53 in starrocks::EsPredicate::build_disjuncts_list() /root/starrocks/be/src/exec/es/es_predicate.cpp:148
    StarRocks#6 0x128ab50d in starrocks::connector::ESDataSource::_build_conjuncts() /root/starrocks/be/src/connector/es_connector.cpp:123
    StarRocks#7 0x128aa7c5 in starrocks::connector::ESDataSource::open(starrocks::RuntimeState*) /root/starrocks/be/src/connector/es_connector.cpp:89
    StarRocks#8 0x9e33847 in starrocks::pipeline::ConnectorChunkSource::_open_data_source(starrocks::RuntimeState*) /root/starrocks/be/src/exec/pipeline/scan/connector_scan_operator.cpp:504
    StarRocks#9 0x9e33ed0 in starrocks::pipeline::ConnectorChunkSource::_read_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk>*) /root/starrocks/be/src/exec/pipeline/scan/connector_scan_operator.cpp:533
    StarRocks#10 0xabc8b65 in starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking(starrocks::RuntimeState*, unsigned long, starrocks::workgroup::WorkGroup const*) /root/starrocks/be/src/exec/pipeline/scan/chunk_source.cpp:67
    StarRocks#11 0x9e168c7 in operator() /root/starrocks/be/src/exec/pipeline/scan/scan_operator.cpp:396
    StarRocks#12 0x9e1c157 in __invoke_impl<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    StarRocks#13 0x9e1c005 in __invoke_r<void, starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#14 0x9e1be7a in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#15 0x992a271 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#16 0x9e519fa in starrocks::workgroup::ScanExecutor::worker_thread() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:67
    StarRocks#17 0x9e5121f in operator() /root/starrocks/be/src/exec/workgroup/scan_executor.cpp:31
    StarRocks#18 0x9e52c5d in __invoke_impl<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    StarRocks#19 0x9e52925 in __invoke_r<void, starrocks::workgroup::ScanExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#20 0x9e5249a in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#21 0x992a271 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#22 0x1181f23d in starrocks::FunctionRunnable::run() /root/starrocks/be/src/util/threadpool.cpp:58
    StarRocks#23 0x1181bff2 in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:553
    StarRocks#24 0x11837c4b in void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:73
    StarRocks#25 0x118375a4 in std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.
3.0/bits/invoke.h:95
    StarRocks#26 0x1183699b in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /opt/gcc/usr/include/c++/10.3.0/functional:416
    StarRocks#27 0x118352fd in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::operator()<, void>() /opt/gcc/usr/include/c++/10.3.0/functional:499
    StarRocks#28 0x11832361 in void std::__invoke_impl<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    StarRocks#29 0x1182fcc5 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110

```

Signed-off-by: trueeyu <[email protected]>
southernriver pushed a commit to southernriver/starrocks that referenced this pull request Nov 28, 2023
@trueeyu @guangxuCheng
[BugFix] Fix the mem_leak of regex_replace (StarRocks#24475) (StarRoc…
c5964d8 
…ks#24516)

Fixes #issue

```
mysql> SELECT regexp_replace('a b c', " ", "-");
+-----------------------------------+
| regexp_replace('a b c', ' ', '-') |
+-----------------------------------+
| a-b-c                             |
+-----------------------------------+
1 row in set (0.01 sec)
```

```
Direct leak of 2319 byte(s) in 1 object(s) allocated from:
    #0 0x974c2af in __interceptor_malloc ../../.././libsanitizer/asan/asan_malloc_linux.cpp:145
    StarRocks#1 0x14bdb529 in alloc_scratch /root/starrocks/thirdparty/src/hyperscan-5.4.0/src/scratch.c:121
    StarRocks#2 0x14bdbb3f in hs_clone_scratch /root/starrocks/thirdparty/src/hyperscan-5.4.0/src/scratch.c:399
    StarRocks#3 0x12d4f791 in regexp_replace_use_hyperscan /root/starrocks/be/src/exprs/string_functions.cpp:2846
    StarRocks#4 0x12d5044a in starrocks::StringFunctions::regexp_replace(starrocks::FunctionContext*, std::vector<std::shared_ptr<starrocks::Column>, std::allocator<std::shared_ptr<starrocks::Column> > > const&) /root/starrocks/be/src/exprs/string_functions.cpp:2913
    StarRocks#5 0xec8af37 in starrocks::VectorizedFunctionCallExpr::evaluate_checked(starrocks::ExprContext*, starrocks::Chunk*) /root/starrocks/be/src/exprs/function_call_expr.cpp:159
    StarRocks#6 0xd924e9b in starrocks::ExprContext::evaluate(starrocks::Expr*, starrocks::Chunk*, unsigned char*) /root/starrocks/be/src/exprs/expr_context.cpp:176
    StarRocks#7 0xd9246dc in starrocks::ExprContext::evaluate(starrocks::Chunk*, unsigned char*) /root/starrocks/be/src/exprs/expr_context.cpp:160
    StarRocks#8 0xab4f521 in starrocks::pipeline::ProjectOperator::push_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk> const&) /root/starrocks/be/src/exec/pipeline/project_operator.cpp:55
    StarRocks#9 0x9c9ec7c in starrocks::pipeline::PipelineDriver::process(starrocks::RuntimeState*, int) /root/starrocks/be/src/exec/pipeline/pipeline_driver.cpp:320
    StarRocks#10 0x12a59131 in starrocks::pipeline::GlobalDriverExecutor::_worker_thread() /root/starrocks/be/src/exec/pipeline/pipeline_driver_executor.cpp:154
    StarRocks#11 0x12a57871 in operator() /root/starrocks/be/src/exec/pipeline/pipeline_driver_executor.cpp:69
    StarRocks#12 0x12a61555 in __invoke_impl<void, starrocks::pipeline::GlobalDriverExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    StarRocks#13 0x12a60a5a in __invoke_r<void, starrocks::pipeline::GlobalDriverExecutor::initialize(int)::<lambda()>&> /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#14 0x12a5fd64 in _M_invoke /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#15 0x98e2d87 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#16 0x1172d057 in starrocks::FunctionRunnable::run() /root/starrocks/be/src/util/threadpool.cpp:58
    StarRocks#17 0x11729e0c in starrocks::ThreadPool::dispatch_thread() /root/starrocks/be/src/util/threadpool.cpp:553
    StarRocks#18 0x11745a65 in void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:73
    StarRocks#19 0x117453be in std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&) /opt/gcc/usr/include/c++/10.
3.0/bits/invoke.h:95
    StarRocks#20 0x117447b5 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /opt/gcc/usr/include/c++/10.3.0/functional:416
    StarRocks#21 0x11743117 in void std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>::operator()<, void>() /opt/gcc/usr/include/c++/10.3.0/functional:499
    StarRocks#22 0x1174017b in void std::__invoke_impl<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:60
    StarRocks#23 0x1173dadf in std::enable_if<is_invocable_r_v<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()>&>(std::_Bind<void (starrocks::ThreadP
ool::*(starrocks::ThreadPool*))()>&) /opt/gcc/usr/include/c++/10.3.0/bits/invoke.h:110
    StarRocks#24 0x11739b48 in std::_Function_handler<void (), std::_Bind<void (starrocks::ThreadPool::*(starrocks::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:291
    StarRocks#25 0x98e2d87 in std::function<void ()>::operator()() const /opt/gcc/usr/include/c++/10.3.0/bits/std_function.h:622
    StarRocks#26 0x11711bc8 in starrocks::Thread::supervise_thread(void*) /root/starrocks/be/src/util/thread.cpp:364
    StarRocks#27 0x7fc585af0ea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)

```

Signed-off-by: trueeyu <[email protected]>
@fzhedu fzhedu mentioned this pull request Jan 11, 2024
Merged
23 tasks
This was referenced Jan 23, 2024
Merged
Merged
Merged
Merged
@before-Sunrise before-Sunrise mentioned this pull request Apr 16, 2024
Merged
24 tasks
This was referenced Jul 5, 2024
Merged
Merged
This was referenced Sep 17, 2024
Closed
Closed
This was referenced Sep 25, 2024
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gengjun-git gengjun-git gengjun-git left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@wyb @CLAassistant @gengjun-git