Skip to content

Low memory killer configuration naming improvements #12393

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
losipiuk merged 2 commits into from
May 15, 2022
Merged

Low memory killer configuration naming improvements #12393

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5453fc2
Rename least-wasted task killing policy to least-waste
losipiuk May 14, 2022
27353d0
Rename killer policy config
losipiuk May 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 5 additions & 7 deletions core/trino-main/src/main/java/io/trino/memory/MemoryManagerConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@
import io.airlift.configuration.Config;
import io.airlift.configuration.ConfigDescription;
import io.airlift.configuration.DefunctConfig;
import io.airlift.configuration.LegacyConfig;
import io.airlift.units.DataSize;
import io.airlift.units.Duration;

Expand Down Expand Up @@ -55,8 +54,7 @@ public LowMemoryQueryKillerPolicy getLowMemoryQueryKillerPolicy()
return lowMemoryQueryKillerPolicy;
}

@LegacyConfig("query.low-memory-killer.policy")
@Config("query.low-memory-query-killer.policy")
@Config("query.low-memory-killer.policy")
public MemoryManagerConfig setLowMemoryQueryKillerPolicy(LowMemoryQueryKillerPolicy lowMemoryQueryKillerPolicy)
{
this.lowMemoryQueryKillerPolicy = lowMemoryQueryKillerPolicy;
Expand All @@ -68,7 +66,7 @@ public LowMemoryTaskKillerPolicy getLowMemoryTaskKillerPolicy()
return lowMemoryTaskKillerPolicy;
}

@Config("query.low-memory-task-killer.policy")
@Config("task.low-memory-killer.policy")
public MemoryManagerConfig setLowMemoryTaskKillerPolicy(LowMemoryTaskKillerPolicy lowMemoryTaskKillerPolicy)
{
this.lowMemoryTaskKillerPolicy = lowMemoryTaskKillerPolicy;
Expand Down Expand Up @@ -203,7 +201,7 @@ public enum LowMemoryTaskKillerPolicy
{
NONE,
TOTAL_RESERVATION_ON_BLOCKED_NODES,
LEAST_WASTED,
LEAST_WASTE,
/**/;

public static LowMemoryTaskKillerPolicy fromString(String value)
Expand All @@ -213,8 +211,8 @@ public static LowMemoryTaskKillerPolicy fromString(String value)
return NONE;
case "total-reservation-on-blocked-nodes":
return TOTAL_RESERVATION_ON_BLOCKED_NODES;
case "least-wasted":
return LEAST_WASTED;
case "least-waste":
return LEAST_WASTE;
}

throw new IllegalArgumentException(format("Unrecognized value: '%s'", value));
Expand Down
2 changes: 1 addition & 1 deletion core/trino-main/src/main/java/io/trino/server/CoordinatorModule.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,7 @@ protected void setup(Binder binder)

bindLowMemoryTaskKiller(LowMemoryTaskKillerPolicy.NONE, NoneLowMemoryKiller.class);
bindLowMemoryTaskKiller(LowMemoryTaskKillerPolicy.TOTAL_RESERVATION_ON_BLOCKED_NODES, TotalReservationOnBlockedNodesTaskLowMemoryKiller.class);
bindLowMemoryTaskKiller(LowMemoryTaskKillerPolicy.LEAST_WASTED, LeastWastedEffortTaskLowMemoryKiller.class);
bindLowMemoryTaskKiller(LowMemoryTaskKillerPolicy.LEAST_WASTE, LeastWastedEffortTaskLowMemoryKiller.class);
bindLowMemoryQueryKiller(LowMemoryQueryKillerPolicy.NONE, NoneLowMemoryKiller.class);
bindLowMemoryQueryKiller(LowMemoryQueryKillerPolicy.TOTAL_RESERVATION, TotalReservationLowMemoryKiller.class);
bindLowMemoryQueryKiller(LowMemoryQueryKillerPolicy.TOTAL_RESERVATION_ON_BLOCKED_NODES, TotalReservationOnBlockedNodesQueryLowMemoryKiller.class);
Expand Down
4 changes: 2 additions & 2 deletions core/trino-main/src/test/java/io/trino/memory/TestMemoryManagerConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@ public void testDefaults()
public void testExplicitPropertyMappings()
{
Map<String, String> properties = ImmutableMap.<String, String>builder()
.put("query.low-memory-query-killer.policy", "none")
.put("query.low-memory-task-killer.policy", "none")
.put("query.low-memory-killer.policy", "none")
.put("task.low-memory-killer.policy", "none")
.put("query.low-memory-killer.delay", "20s")
.put("query.max-memory", "2GB")
.put("query.max-total-memory", "3GB")
Expand Down
2 changes: 1 addition & 1 deletion docs/src/main/sphinx/admin/fault-tolerant-execution.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ query.
The following cluster configuration changes are recommended to improve
fault-tolerant execution with a ``TASK`` retry policy:

* Set the ``query.low-memory-task-killer.policy``
* Set the ``task.low-memory-killer.policy``
:doc:`query management property </admin/properties-query-management>` to
``total-reservation-on-blocked-nodes``, or queries may
need to be manually killed if the cluster runs out of memory.
Expand Down
10 changes: 5 additions & 5 deletions docs/src/main/sphinx/admin/properties-query-management.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ stages of a query. You can use the following execution policies:
The number of partitions to use for processing distributed operations, such as
joins, aggregations, partitioned window functions and others.

``query.low-memory-query-killer.policy``
``query.low-memory-killer.policy``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* **Type:** :ref:`prop-type-string`
Expand All @@ -50,7 +50,7 @@ memory availability. Supports the following values:

Only applies for queries with task level retries disabled (``retry-policy`` set to ``NONE`` or ``QUERY``)

``query.low-memory-task-killer.policy``
``task.low-memory-killer.policy``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* **Type:** :ref:`prop-type-string`
Expand All @@ -63,7 +63,7 @@ memory availability. Supports the following values:
* ``total-reservation-on-blocked-nodes`` - Kill the tasks which are part of the queries
which has task retries enabled and are currently using the most memory specifically
on nodes that are now out of memory.
* ``least-wasted`` - Kill the tasks which are part of the queries
* ``least-waste`` - Kill the tasks which are part of the queries
which has task retries enabled and use significant amount of memory on nodes
which are now out of memory. This policy avoids killing tasks which are already
executing for a long time, so significant amount of work is not wasted.
Expand All @@ -79,8 +79,8 @@ memory availability. Supports the following values:
* **Default value:** ``5m``

The amount of time a query is allowed to recover between running out of memory
and being killed, if ``query.low-memory-query-killer.policy`` or
``query.low-memory-task-killer.policy`` is set to value differnt than ``none``.
and being killed, if ``query.low-memory-killer.policy`` or
``task.low-memory-killer.policy`` is set to value differnt than ``none``.

``query.max-execution-time``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down
2 changes: 1 addition & 1 deletion testing/trino-tests/src/test/java/io/trino/memory/TestMemoryManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ public void testOutOfMemoryKiller()
{
Map<String, String> properties = ImmutableMap.<String, String>builder()
.put("query.low-memory-killer.delay", "5s")
.put("query.low-memory-query-killer.policy", "total-reservation")
.put("query.low-memory-killer.policy", "total-reservation")
.buildOrThrow();

try (DistributedQueryRunner queryRunner = createQueryRunner(TINY_SESSION, properties)) {
Expand Down