Skip to content

Commit

Permalink
[BugFix] Be crash when doing compaction apply (#14513)
Browse files Browse the repository at this point in the history
This bug is introduced by #12068. To reduce the memory usage during apply, we don't preload all segments' primary keys and process segment by segment(#12068).

```
 ....
 uint32_t max_rowset_id = *std::max_element(info->inputs.begin(), info->inputs.end());
 Rowset* rowset = _get_rowset(max_rowset_id).get();
 .....
 for (size_t i = 0; i < _compaction_state->pk_cols.size(); i++) {
       // the rowset is not what we should load
       if (st = _compaction_state->load_segments(rowset, i); !st.ok()) {
            manager->index_cache().release(index_entry);
            _compaction_state.reset();
            std::string msg = strings::Substitute("_apply_compaction_commit error: load compaction state failed: $0 $1",
                                                  st.to_string(), debug_string());
            LOG(ERROR) << msg;
            _set_error(msg);
            return;
        }
        .....
    }
```
As the above code shown, we will load one segment for a loop. However, the `rowset` is not the output rowset after compaction but one of the input rowsets, so we may get an unexpected error.

(cherry picked from commit 287f82a)
  • Loading branch information
sevev authored and mergify[bot] committed Dec 5, 2022
1 parent 1d329ce commit ec1b0f3
Showing 1 changed file with 10 additions and 2 deletions.
12 changes: 10 additions & 2 deletions be/src/storage/tablet_updates.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1411,7 +1411,15 @@ void TabletUpdates::_apply_compaction_commit(const EditVersionInfo& version_info
_set_error(msg);
return;
}
if (!(st = _compaction_state->load(_get_rowset(rowset_id).get())).ok()) {
Rowset* output_rowset = _get_rowset(rowset_id).get();
if (output_rowset == nullptr) {
string msg = strings::Substitute("_apply_compaction_commit rowset not found tablet=$0 rowset=$1",
_tablet.tablet_id(), rowset_id);
LOG(ERROR) << msg;
_set_error(msg);
return;
}
if (!(st = _compaction_state->load(output_rowset)).ok()) {
manager->index_cache().release(index_entry);
_compaction_state.reset();
std::string msg = Substitute("_apply_compaction_commit error: load compaction state failed: $0 $1",
Expand Down Expand Up @@ -1442,7 +1450,7 @@ void TabletUpdates::_apply_compaction_commit(const EditVersionInfo& version_info
uint32_t max_src_rssid = max_rowset_id + rowset->num_segments() - 1;

for (size_t i = 0; i < _compaction_state->pk_cols.size(); i++) {
if (st = _compaction_state->load_segments(rowset, i); !st.ok()) {
if (st = _compaction_state->load_segments(output_rowset, i); !st.ok()) {
manager->index_cache().release(index_entry);
_compaction_state.reset();
std::string msg = Substitute("_apply_compaction_commit error: load compaction state failed: $0 $1",
Expand Down

0 comments on commit ec1b0f3

Please sign in to comment.