Skip to content

Add documentation for db resource group manager #4523

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
phd3 wants to merge 1 commit into from
Closed

Add documentation for db resource group manager #4523

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 109 additions & 19 deletions presto-docs/src/main/sphinx/admin/resource-groups.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,102 @@ it does not cause running queries to fail; instead new queries become queued.
A resource group may have sub-groups or may accept queries, but may not do both.

The resource groups and associated selection rules are configured by a manager, which is pluggable.
Add an ``etc/resource-groups.properties`` file with the following contents to enable
the built-in manager that reads a JSON config file:

There are two built-in managers: file-based and database-based. Add an ``etc/resource-groups.properties``
Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use a file-based or a database-based resource group manager:

  • Add a file etc/resource-groups.properties
  • Set the resource-groups.configuration-manager property to file or db
  • Add further configuration properties for the the desired manager.

file with the appropriate configurations to enable one of the two managers.

File Resource Group Manager
Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File resource group manager

---------------------------

This manager reads the resource group configuration from a JSON config file.
Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file resource group manager reads a JSON configuration file, specified with resource-groups.config-file


.. code-block:: none

resource-groups.configuration-manager=file
resource-groups.config-file=etc/resource-groups.json

Change the value of ``resource-groups.config-file`` to point to a JSON config file,
which can be an absolute path, or a path relative to the Presto data directory.
* ``resource-groups.config-file``: a JSON file storing the configurations, which can be an absolute
Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The path to the JSON file can be an absolute path, or a path relative to the Presto data directory.

path, or a path relative to the Presto data directory.

Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also does this file need to be on the coordinator and all workers?

Db Resource Group Manager
Copy link
Copy Markdown
Member

@Lewuathe Lewuathe Jul 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Db Resource Group Manager
Database Resource Group Manager

Might Database instead of Db be more descriptive?

Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Database resource group manager

-------------------------

This manager periodically loads the resource group configuration from a database.
Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The database resource group manager loads the configuration from a relational database.

Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to doc which ones are supported.

Copy link
Copy Markdown
Member

@hashhar hashhar Jul 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only MySQL is supported at the moment. It does mean it should work with MariaDB too but I'm not sure.


.. code-block:: none

resource-groups.configuration-manager=db
resource-groups.config-db-url=jdbc:mysql://localhost:3306/config_db?user=presto_admin
Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which databases are supported? How do users get JDBC driver in place?

resource-groups.max-refresh-interval=1m
resource-groups.exact-match-selector-enabled=true

* ``resource-groups.config-db-url`` (required): Database URL to load configuration from.
Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

specify more details about this JDBC URL

Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All properties in the following section require quite a bit more docs so I would suggest to remove usage of a bullet lest and just document them in multiple paragraphs one after the other.

* ``resource-groups.max-refresh-interval``: Time period for which the cluster will continue to accept
Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thats a odd name for what it does. Anyway .. remove future usage . otherwise ok

queries after refresh failures cause configuration to become stale. The default is ``1h``.
* ``resource-groups.exact-match-selector-enabled``: Boolean flag to enable usage of a selector requiring
an exact match of environment, source and query type.
Copy link
Copy Markdown
Member

@Lewuathe Lewuathe Jul 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the default value for DbResourceGroupConfig.exactMatchSelectorEnabled implicitly set false, should we put the default value here?

Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes ... also this needs to be explained more.. maybe with an example.


The configuration is loaded every second from the following tables under the specified db url. If the
Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The paragraph and following table needs to be broken up and reordered

  • First startup and automatic table creation
  • Document each table maybe with example row of data (including additional table for exact-match)
  • Document how we suggest users add data to the table (maybe example insert statement, update statement), also should they do that from Presto via catalog, or some other SQL UI directly on the db (probably that)
  • How it works (refresh interval...)

configuration ``resource-groups.exact-match-selector-enabled`` is set to ``true``, an additional table
is used for loading exact-match based selectors. If any of the tables do not exist, Presto creates them
on server startup.

+---------------------+-------------------------------------------------------+---------------------------------+
Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is way too much info for a table. I suggest we break this up into a separate section with proper code blocks and paragraphs that details more info.

| Table Name | Schema | Purpose |
+=====================+=======================================================+=================================+
| resource\_groups\_ | .. code-block:: sql |Store properties listed in |
| global\_properties | |:ref:`global-properties` |
| | name VARCHAR(128) NOT NULL PRIMARY KEY, | |
| | value VARCHAR(512) NULL, | |
| | CHECK (name in ('cpu_quota_period')) | |
+---------------------+-------------------------------------------------------+---------------------------------+
| resource\_groups | .. code-block:: sql |Store resource group |
| | |configuration described in |
| | resource_group_id BIGINT NOT NULL AUTO_INCREMENT, |:ref:`resource-group-properties`.|
| | name VARCHAR(250) NOT NULL, | |
| | soft_memory_limit VARCHAR(128) NOT NULL, | |
| | max_queued INT NOT NULL, | |
| | soft_concurrency_limit INT NULL, | |
| | hard_concurrency_limit INT NOT NULL, | |
| | scheduling_policy VARCHAR(128) NULL, | |
| | scheduling_weight INT NULL, | |
| | jmx_export BOOLEAN NULL, | |
| | soft_cpu_limit VARCHAR(128) NULL, | |
| | hard_cpu_limit VARCHAR(128) NULL, | |
| | parent BIGINT NULL, | |
| | environment VARCHAR(128) NULL, | |
| | PRIMARY KEY (resource_group_id), | |
| | FOREIGN KEY (parent) REFERENCES | |
| | resource_groups (resource_group_id) | |
+---------------------+-------------------------------------------------------+---------------------------------+
| selectors | .. code-block:: sql |Store selector rules |
| | |to match input queries |
| | resource_group_id BIGINT NOT NULL, |against resource groups, refer |
| | priority BIGINT NOT NULL, |to :ref:`selector-rules`. |
| | user_regex VARCHAR(512), |If there are multiple matches |
| | source_regex VARCHAR(512), |for a query, the one with the |
| | query_type VARCHAR(512), |highest priority is selected. |
| | client_tags VARCHAR(512), | |
| | selector_resource_estimate VARCHAR(1024), | |
| | FOREIGN KEY (resource_group_id) REFERENCES | |
| | resource_groups (resource_group_id) | |
+---------------------+-------------------------------------------------------+---------------------------------+
| exact\_match\_ | .. code-block:: sql |Store selector rules |
| source\_selectors | |for an exact match of source, |
| | environment VARCHAR(128), |environment and query type. |
| | source VARCHAR(512) NOT NULL, |``NULL`` values for query_type |
| | query_type VARCHAR(512), |and environment are treated as |
| | update_time DATETIME NOT NULL, |wildcards, and have a lower |
| | resource_group_id VARCHAR(256) NOT NULL, |priority than exact matches. |
| | PRIMARY KEY (environment, source, query_type), | |
| | UNIQUE ( | |
| | source, | |
| | environment, | |
| | query_type, | |
| | resource_group_id) | |
+---------------------+-------------------------------------------------------+---------------------------------+

.. _resource-group-properties:

Resource Group Properties
-------------------------
Expand Down Expand Up @@ -72,6 +158,8 @@ Resource Group Properties

* ``subGroups`` (optional): list of sub-groups.

.. _selector-rules:

Selector Rules
Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Selector rules

--------------

Expand All @@ -98,26 +186,13 @@ Selector Rules

Selectors are processed sequentially and the first one that matches will be used.

.. _global-properties:

Global Properties
Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Global propertie

-----------------

* ``cpuQuotaPeriod`` (optional): the period in which cpu quotas are enforced.

Providing Selector Properties
-----------------------------

The source name can be set as follows:

* CLI: use the ``--source`` option.

* JDBC: set the ``ApplicationName`` client info property on the ``Connection`` instance.

Client tags can be set as follows:

* CLI: use the ``--client-tags`` option.

* JDBC: set the ``ClientTags`` client info property on the ``Connection`` instance.

Example
-------

Expand Down Expand Up @@ -174,3 +249,18 @@ For the remaining users:

.. literalinclude:: resource-groups-example.json
:language: json

Providing Selector Properties
Copy link
Copy Markdown
Member

@mosabua mosabua Jul 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Providing selector properties

-----------------------------

The source name can be set as follows:

* CLI: use the ``--source`` option.

* JDBC: set the ``ApplicationName`` client info property on the ``Connection`` instance.

Client tags can be set as follows:

* CLI: use the ``--client-tags`` option.

* JDBC: set the ``ClientTags`` client info property on the ``Connection`` instance.