Auto-balance big parts in JBOD array #16481

amosbird · 2020-10-28T15:24:37Z

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Introduce a new merge tree setting min_bytes_to_rebalance_partition_over_jbod which allows assigning new parts to different disks of a JBOD volume in a balanced way.

Detailed description / Documentation draft:

JBOD volume is notorious in that only a small subset of disks can be fully utilized for a given query. This PR focuses on a particular common scenario: partition-wise queries. Now big new parts are assigned to different disks of a JBOD volume in a balanced way.
It might be useful for #16300.

amosbird · 2021-02-19T10:19:37Z

@azat Another uncaught exception from dtor. Please take a look. https://clickhouse-test-reports.s3.yandex.net/16481/5d150cce4778dd14f58dcff67435bdec1efa155b/stress_test_(thread).html#fail1

azat · 2021-02-19T19:03:55Z

@azat Another uncaught exception from dtor. Please take a look. https://clickhouse-test-reports.s3.yandex.net/16481/5d150cce4778dd14f58dcff67435bdec1efa155b/stress_test_(thread).html#fail1

Here is another patch - #20981

amosbird · 2021-02-23T17:10:37Z

Test failure looks unrelated https://clickhouse-test-reports.s3.yandex.net/16481/605dd9a971d72c91a721148e0c489c08dd4f86c7/integration_tests_(thread)/integration_run_test_storage_kafka_test_py_1.txt.out.log

alesapin

Maybe I'm a little confused, but we just have three types of parts:

Existing parts.
Currently merging source parts that will become outdated (included in 1).
Currently merging parts that will be added as a result of the merge.

For each new merged/fetched part we are trying to find a disk that has a minimum size of data in the same partition. So we just summing sizes by a disk of all existing parts from this same partition excluding parts 2) and adding parts 3). After that, we choose the smallest occupied disk.

The code looks slightly more complicated :)

Btw, how it will work with background moves? Also without tests merge is impossible.

alesapin · 2021-02-24T12:33:34Z

src/Storages/MergeTree/registerStorageMergeTree.cpp

+
+        // Pre-define a reasonable minimum size for the JBOD rebalancer
+        static constexpr size_t MIN_BYTES_TO_REBALANCE_OVER_JBOD = 100 * 1024 * 1024;
+        if (storage_settings->min_bytes_to_rebalance_partition_over_jbod > 0


We have a special place for it https://github.com/ClickHouse/ClickHouse/blob/master/src/Storages/MergeTree/MergeTreeSettings.cpp#L76-L101

alesapin · 2021-02-24T12:34:49Z

src/Storages/MergeTree/MergeTreeData.cpp

+    for (const auto & part : parts)
+    {
+        if (!storage.currently_submerging_parts.count(part))
+            LOG_WARNING(log, "currently_submerging_parts is missing parts. JBOD might lose balance");


Unclear message. Which part is missing? What does it mean?

alesapin · 2021-02-24T12:38:00Z

src/Storages/MergeTree/MergeTreeData.cpp

+
+        for (const auto & [name, emerging_part] : currently_emerging_parts)
+        {
+            // It's possible that the emerging parts are committed and get added twice. Thus a set is used to deduplicate.


How is it possible?

Because currently_emerging_parts is also maintained separately. We do commit before removing parts from it. Thus the previous getDataPartsStateRange(MergeTreeData::DataPartState::Committed) might contain (rarely) the same part.

alesapin · 2021-02-24T12:42:57Z

src/Storages/MergeTree/MergeTreeData.cpp

+                {
+                    auto name = part->volume->getDisk()->getName();
+                    auto it = disk_occupation.find(name);
+                    if (it != disk_occupation.end())


Otherwise, it's a LOGICAL_ERROR, isn't it?

It's possible that parts are stored on different volumes. We ignore them here.

alesapin · 2021-02-24T12:44:58Z

src/Storages/MergeTree/MergeTreeData.cpp

+                        {
+                            it->second += part->getBytesOnDisk();
+                            disk_parts[name].push_back(formatReadableSizeWithBinarySuffix(part->getBytesOnDisk()));
+                            big_parts.insert(part->name);


big_not_merging_parts?

alesapin · 2021-02-24T13:08:25Z

src/Storages/MergeTree/MergeTreeData.cpp

+
+            // Also include current submerging parts
+            for (const auto & part : covered_parts)
+                merging_parts.insert(part->name);


Why they are already not here? Ho we can have covered parts which are not in currently_submerging_parts? Also why we don't check?

part->isStoredOnDisk() && part->getBytesOnDisk() >= min_bytes_to_rebalance_partition_over_jbod && part_info.partition_id == part->info.partition_id

currently_submerging_parts only records parts in previous balanced reservation.

alesapin · 2021-02-24T13:10:45Z

src/Storages/MergeTree/MergeTreeData.cpp

+        for (const auto & [name, emerging_part] : currently_emerging_parts)
+        {
+            // It's possible that the emerging parts are committed and get added twice. Thus a set is used to deduplicate.
+            if (big_parts.find(name) == big_parts.end())


Again, no checks -- maybe this part from a different partition or small?

Ah, we should check partition here. It will be alway big though.

alesapin · 2021-02-24T13:12:03Z

src/Storages/MergeTree/MergeTreeData.cpp

+    {
+        const auto & disks = getStoragePolicy()->getVolume(max_volume_index)->getDisks();
+        std::map<String, size_t> disk_occupation;
+        std::map<String, std::vector<String>> disk_parts;


Add a comment: "Used only for logging"

alesapin · 2021-02-24T13:14:18Z

src/Storages/MergeTree/MergeTreeData.cpp

+            if (part->isStoredOnDisk() && part->getBytesOnDisk() >= min_bytes_to_rebalance_partition_over_jbod
+                && part_info.partition_id == part->info.partition_id)
+            {
+                merging_parts.insert(part->name);


big_submerging_parst_from_partition?

The set is used to remove some big valid parts out of calculation. It doesn't have currently submerging parts because those are not yet inserted into currently_submerging_parts, which only contains parts that participate in some other balanced activity.

alesapin · 2021-02-24T13:15:37Z

src/Storages/MergeTree/MergeTreeData.cpp

+    try
+    {
+        const auto & disks = getStoragePolicy()->getVolume(max_volume_index)->getDisks();
+        std::map<String, size_t> disk_occupation;


Why map and not unordered_map?

Disk names are usually sorted.

alesapin · 2021-03-09T11:58:34Z

tests/integration/test_jbod_balancer/test.py

+        for _ in range(10):
+            try:
+                print("Syncing replica")
+                node2.query("SYSTEM SYNC REPLICA tbl")


This query ignores timeout, which exception do we expect?

Added timeout = 10s. We can safely ignore any exceptions. The test checks the final state of disk balance.

alesapin · 2021-03-09T11:58:41Z

tests/integration/test_jbod_balancer/test.py

+                "insert into tmp2 select randConstant() % 2, randomPrintableASCII(16) from numbers(50)"
+            )
+
+        time.sleep(2)


Why we need it?

src/Storages/MergeTree/MergeTreeData.cpp

alesapin · 2021-03-09T16:20:37Z

src/Storages/MergeTree/MergeTreeData.cpp

+            WriteBufferFromOwnString log_str;
+            writeCString("\nbalancer: \n", log_str);
+            for (const auto & [disk_name, per_disk_parts] : disk_parts_for_logging)
+                writeString(fmt::format("  {}: [{}]\n", disk_name, boost::algorithm::join(per_disk_parts, ", ")), log_str);


btw, fmt::join looks prettier.

src/Storages/MergeTree/MergeTreeSettings.cpp

alesapin

Ok, quite isolated and useful code. Also, we have a setting that allows disabling this behavior.

src/Storages/MergeTree/MergeTreeSettings.h

sevirov · 2021-07-29T21:13:54Z

Internal documentation ticket: DOCSUP-12425

robot-clickhouse added the pr-improvement Pull request with some product improvements label Oct 28, 2020

amosbird marked this pull request as draft October 28, 2020 15:25

amosbird force-pushed the jbodbalance branch 2 times, most recently from c28def2 to e10faae Compare October 29, 2020 07:32

alesapin self-assigned this Nov 20, 2020

amosbird marked this pull request as ready for review February 18, 2021 08:52

amosbird force-pushed the jbodbalance branch from e10faae to e184235 Compare February 18, 2021 08:53

amosbird requested a review from alesapin February 18, 2021 08:56

amosbird force-pushed the jbodbalance branch 2 times, most recently from 9bc3ec3 to a36c60c Compare February 18, 2021 10:40

azat mentioned this pull request Feb 19, 2021

Improve backtrace catching on server failures in CI for stress tests #20462

Merged

amosbird force-pushed the jbodbalance branch from 5d150cc to 605dd9a Compare February 23, 2021 02:57

alesapin requested changes Feb 24, 2021

View reviewed changes

amosbird added 3 commits March 8, 2021 11:10

JBOD data balancer

e6522e1

Fix build

2b11a2a

update and add tests

2ec20c5

amosbird force-pushed the jbodbalance branch from 605dd9a to 2ec20c5 Compare March 8, 2021 09:40

alesapin reviewed Mar 9, 2021

View reviewed changes

src/Storages/MergeTree/MergeTreeData.cpp Show resolved Hide resolved

alesapin reviewed Mar 9, 2021

View reviewed changes

src/Storages/MergeTree/MergeTreeSettings.cpp Show resolved Hide resolved

update

e52b2fb

alesapin approved these changes Mar 11, 2021

View reviewed changes

src/Storages/MergeTree/MergeTreeSettings.h Outdated Show resolved Hide resolved

Disable new setting by default

5704405

alesapin merged commit cca05da into ClickHouse:master Mar 12, 2021

sevirov mentioned this pull request Oct 29, 2021

DOCSUP-12425: Document the min_bytes_to_rebalance_partition_over_jbod setting #30867

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auto-balance big parts in JBOD array #16481

Auto-balance big parts in JBOD array #16481

amosbird commented Oct 28, 2020 •

edited by alesapin

Loading

amosbird commented Feb 19, 2021

azat commented Feb 19, 2021

amosbird commented Feb 23, 2021

alesapin left a comment

alesapin Feb 24, 2021

amosbird Mar 8, 2021

alesapin Feb 24, 2021

amosbird Mar 8, 2021

alesapin Feb 24, 2021

amosbird Mar 8, 2021

alesapin Feb 24, 2021

amosbird Mar 8, 2021

alesapin Feb 24, 2021

alesapin Feb 24, 2021

amosbird Mar 8, 2021

alesapin Feb 24, 2021

amosbird Mar 8, 2021

alesapin Feb 24, 2021

alesapin Feb 24, 2021

amosbird Mar 8, 2021

alesapin Feb 24, 2021

amosbird Feb 24, 2021

alesapin Mar 9, 2021

amosbird Mar 9, 2021 •

edited

Loading

alesapin Mar 9, 2021

amosbird Mar 9, 2021

alesapin Mar 9, 2021

amosbird Mar 9, 2021

alesapin left a comment

sevirov commented Jul 29, 2021

Auto-balance big parts in JBOD array #16481

Auto-balance big parts in JBOD array #16481

Conversation

amosbird commented Oct 28, 2020 • edited by alesapin Loading

amosbird commented Feb 19, 2021

azat commented Feb 19, 2021

amosbird commented Feb 23, 2021

alesapin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amosbird Mar 9, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alesapin left a comment

Choose a reason for hiding this comment

sevirov commented Jul 29, 2021

amosbird commented Oct 28, 2020 •

edited by alesapin

Loading

amosbird Mar 9, 2021 •

edited

Loading