Feature database and table count limitation issue56347 #64781

XuJia0210 · 2024-06-04T07:26:38Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add server settings max_table_num_to_throw and max_database_num_to_throw to limit the number of databases or tables on CREATE queries.

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing):

Allow: All Required Checks
Allow: Stateless tests
Allow: Stateful tests
Allow: Integration Tests
Allow: Performance tests
Allow: All NOT Required Checks
Allow: batch 1, 2 for multi-batch jobs
Allow: batch 3, 4, 5, 6 for multi-batch jobs

Do not test
Upload binaries for special builds
Disable merge-commit
Disable CI cache

CLAassistant · 2024-06-04T07:26:44Z

All committers have signed the CLA.

robot-ch-test-poll · 2024-06-04T07:29:27Z

This is an automated comment for commit c32b97d with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check name	Description	Status
A Sync	If it fails, ask a maintainer for help	⏳ pending
CI running	A meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR	❌ failure
Integration tests	The integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests	❌ failure
Mergeable Check	Checks if all other necessary checks are successful	⏳ pending

Successful checks

Check name	Description	Status
Flaky tests	Checks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integration tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc	✅ success
PR Check	Checks correctness of the PR's body	✅ success
Style check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	✅ success

davenger · 2024-06-04T11:56:10Z

src/Interpreters/InterpreterCreateQuery.cpp

+    {
+        size_t db_count = DatabaseCatalog::instance().getDatabases().size();
+        // there's an invisible system database _temporary_and_external_tables, so we need to subtract 1
+        if (db_count > 0)


I think it makes sense to check that this hidden system DB is actually present && DatabaseCatalog::instance().isDatabaseExist(DatabaseCatalog::TEMPORARY_DATABASE)

hanfei1991 · 2024-06-06T20:58:54Z

tests/queries/0_stateless/03165_database_and_table_count_limitation_test.sh

+# Generate config.xml
+CONFIG_FILE="$TEMP_DIR/config.xml"
+cat > "$CONFIG_FILE" <<EOL
+<yandex>


We have left yandex :)

Don't do this in stateless test :( we write integration tests to test special config

hanfei1991 · 2024-06-06T21:04:31Z

tests/queries/0_stateless/03165_database_and_table_count_limitation_sql

@@ -0,0 +1,21 @@
+create database db1;


Don't do this .... we only have .sql and .reference in 0_stateless directory

hanfei1991 · 2024-06-10T20:49:47Z

I write an integration test to replace original one. Feel free to amend/revert if you do not agree :)

I have another question, why we don't care about system tables for tables limit, but have to care system and other default databases for limit? We can make the behaviour unified: only care about database/table which users create.

hanfei1991 · 2024-06-10T20:50:24Z

src/Interpreters/InterpreterCreateQuery.cpp

@@ -138,6 +145,18 @@ BlockIO InterpreterCreateQuery::createDatabase(ASTCreateQuery & create)
            throw Exception(ErrorCodes::DATABASE_ALREADY_EXISTS, "Database {} already exists.", database_name);
    }

+    if (auto max_db = getContext()->getGlobalContext()->getServerSettings().max_database_num_to_throw; max_db > 0)


We can write two lines, don't have to be tooo brief

hanfei1991 · 2024-06-10T20:50:31Z

src/Interpreters/InterpreterCreateQuery.cpp

@@ -1537,6 +1556,16 @@ bool InterpreterCreateQuery::doCreateTable(ASTCreateQuery & create,
        }
    }

+    if (UInt64 max_table = getContext()->getGlobalContext()->getServerSettings().max_table_num_to_throw; max_table > 0)


hanfei1991 · 2024-06-10T20:52:33Z

src/Interpreters/InterpreterCreateQuery.cpp

+        UInt64 table_count = CurrentMetrics::get(CurrentMetrics::AttachedTable);
+        if (table_count >= max_table)
+            throw Exception(ErrorCodes::TOO_MANY_TABLES,
+                            "Too many tables in the system. Current is {}, limit is {}. "


How about using one sentence:
Too many tables in ClickHouse. The limit (setting 'max_table_num_to_throw') is set to {}

Same for databases.

Besides, do not write period at the end.

max_table_num_to_throw max_database_num_to_throw

…st result file

…plicitly, change 'yandex' into 'clickhouse' in config.xml

…n_issue56347

robot-ch-test-poll added the pr-feature Pull request with new product feature label Jun 4, 2024

kssenii added the can be tested Allows running workflows for external contributors label Jun 4, 2024

davenger reviewed Jun 4, 2024

View reviewed changes

hanfei1991 self-assigned this Jun 6, 2024

hanfei1991 reviewed Jun 6, 2024

View reviewed changes

hanfei1991 reviewed Jun 10, 2024

View reviewed changes

hanfei1991 approved these changes Jun 13, 2024

View reviewed changes

hanfei1991 enabled auto-merge June 13, 2024 10:50

XuJia0210 and others added 11 commits June 14, 2024 08:16

Add 2 server_setting to limit numebr of table/db

bd0da13

max_table_num_to_throw max_database_num_to_throw

do not count systemStorage in AttachedTable metric

25e927a

modify DatabaseCommon.cpp compile errors; rename testcases, remove te…

0cdcc0a

…st result file

fix typo

06239e1

add a space to trigger CI after fixed a CI bug

446cc05

adjust database count checking DatabaseCatalog::TEMPORARY_DATABASE ex…

a2fd206

…plicitly, change 'yandex' into 'clickhouse' in config.xml

write integration tests

5ff65a1

fix style

f073a45

adjust code based on review comment

76d6bc3

fix type

555e166

trigger CI

d44d525

auto-merge was automatically disabled June 14, 2024 00:22
Head branch was pushed to by a user without write access

XuJia0210 closed this Jun 14, 2024

auto-merge was automatically disabled June 14, 2024 00:22
Pull request was closed

XuJia0210 force-pushed the feature_database_and_table_count_limitation_issue56347 branch from 8112250 to 594a0e9 Compare June 14, 2024 00:22

XuJia0210 reopened this Jun 14, 2024

XuJia0210 added 2 commits June 14, 2024 09:44

trigger CI

08b481a

not count syste tables

d0f2014

hanfei1991 and others added 4 commits June 14, 2024 15:28

add more tests

da18681

fix style error

f2a162a

fix the error system flush logs hang

d8379bf

Merge branch 'master' into feature_database_and_table_count_limitatio…

c32b97d

…n_issue56347

hanfei1991 enabled auto-merge June 18, 2024 13:53

hanfei1991 added this pull request to the merge queue Jun 19, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 19, 2024

hanfei1991 added this pull request to the merge queue Jun 19, 2024

Merged via the queue into ClickHouse:master with commit 5eca8db Jun 19, 2024
71 of 74 checks passed

robot-ch-test-poll added the pr-synced-to-cloud The PR is synced to the cloud repo label Jun 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature database and table count limitation issue56347 #64781

Feature database and table count limitation issue56347 #64781

XuJia0210 commented Jun 4, 2024 •

edited by Algunenano

Loading

CLAassistant commented Jun 4, 2024 •

edited

Loading

robot-ch-test-poll commented Jun 4, 2024 •

edited by robot-ch-test-poll2

Loading

davenger Jun 4, 2024

hanfei1991 Jun 6, 2024

hanfei1991 Jun 6, 2024

hanfei1991 commented Jun 10, 2024

hanfei1991 Jun 10, 2024

hanfei1991 Jun 10, 2024

hanfei1991 Jun 10, 2024 •

edited

Loading

Feature database and table count limitation issue56347 #64781

Feature database and table count limitation issue56347 #64781

Conversation

XuJia0210 commented Jun 4, 2024 • edited by Algunenano Loading

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI Settings (Only check the boxes if you know what you are doing):

CLAassistant commented Jun 4, 2024 • edited Loading

robot-ch-test-poll commented Jun 4, 2024 • edited by robot-ch-test-poll2 Loading

davenger Jun 4, 2024

Choose a reason for hiding this comment

hanfei1991 Jun 6, 2024

Choose a reason for hiding this comment

hanfei1991 Jun 6, 2024

Choose a reason for hiding this comment

hanfei1991 commented Jun 10, 2024

hanfei1991 Jun 10, 2024

Choose a reason for hiding this comment

hanfei1991 Jun 10, 2024

Choose a reason for hiding this comment

hanfei1991 Jun 10, 2024 • edited Loading

Choose a reason for hiding this comment

XuJia0210 commented Jun 4, 2024 •

edited by Algunenano

Loading

CLAassistant commented Jun 4, 2024 •

edited

Loading

robot-ch-test-poll commented Jun 4, 2024 •

edited by robot-ch-test-poll2

Loading

hanfei1991 Jun 10, 2024 •

edited

Loading