Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions core/trino-spi/src/main/java/io/trino/spi/SplitWeight.java
Original file line number Diff line number Diff line change
Expand Up @@ -75,12 +75,29 @@ public String toString()
return BigDecimal.valueOf(value, -UNIT_SCALE).stripTrailingZeros().toPlainString();
}

/**
* Produces a {@link SplitWeight} from the raw internal value representation. This method is intended
* primarily for JSON deserialization, and connectors should use not call this factory method directly
* to construct {@link SplitWeight} instances. Instead, connectors should use {@link SplitWeight#fromProportion(double)}
* to avoid breakages that could arise if {@link SplitWeight#UNIT_VALUE} changes in the future.
*/
@JsonCreator
public static SplitWeight fromRawValue(long value)
{
return value == UNIT_VALUE ? STANDARD_WEIGHT : new SplitWeight(value);
}

/**
* Produces a {@link SplitWeight} that corresponds to the {@link SplitWeight#standard()} weight
* proportionally, ie: a parameter of <code>1.0</code> will be equivalent to the standard weight
Comment thread
pettyjamesm marked this conversation as resolved.
* and a value of <code>0.5</code> will be 1/2 of the standard split weight. Valid arguments
* must be greater than zero and finite. Connectors should prefer constructing split weights
* using this factory method rather than passing a raw integer value in case the integer representation
* of a standard split needs to change in the future.
*
* @param weight the proportional weight relative to a standard split, expressed as a double
* @return a {@link SplitWeight} with a raw value corresponding to the requested proportion
*/
public static SplitWeight fromProportion(double weight)
{
if (weight <= 0 || !Double.isFinite(weight)) {
Expand Down
21 changes: 13 additions & 8 deletions docs/src/main/sphinx/admin/properties-node-scheduler.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,18 @@ Splits
* **Default value:** ``100``

The target value for the total number of splits that can be running for
each worker node.
each worker node, assuming all splits have the standard split weight.

Using a higher value is recommended, if queries are submitted in large batches
(e.g., running a large group of reports periodically), or for connectors that
produce many splits that complete quickly. Increasing this value may improve
query latency, by ensuring that the workers have enough splits to keep them
fully utilized.
produce many splits that complete quickly but do not support assigning split
weight values to express that to the split scheduler. Increasing this value may
improve query latency, by ensuring that the workers have enough splits to keep
them fully utilized.

When connectors do support weight based split scheduling, the number of splits
assigned will depend on the weight of the individual splits. If splits are
small, more of them are allowed to be assigned to each worker to compensate.

Setting this too high wastes memory and may result in lower performance
due to splits not being balanced across workers. Ideally, it should be set
Expand All @@ -31,10 +36,10 @@ not higher.
* **Type:** :ref:`prop-type-integer`
* **Default value:** ``10``

The number of outstanding splits that can be queued for each worker node
for a single stage of a query, even when the node is already at the limit for
total number of splits. Allowing a minimum number of splits per stage is
required to prevent starvation and deadlocks.
The number of outstanding splits with the standard split weight that can be
queued for each worker node for a single stage of a query, even when the
node is already at the limit for total number of splits. Allowing a minimum
number of splits per stage is required to prevent starvation and deadlocks.

This value must be smaller than ``node-scheduler.max-splits-per-node``,
is usually increased for the same reasons, and has similar drawbacks
Expand Down