Limit number of nodes that execute writing stages

[radek-kondziolka]

Description

The option query.max-writer-node-count was added to limit number of nodes that take part in executing writer stages.
It was implemented by some changes in ScaledWriterScheduler (for unpartitioned data) and by adding the new rule LimitMaxWriterNodesCount to the optimizer (for partitioned data).

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( *) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

Documentation

Documentation update is need and will be done soon.

[sopel39]

Close #15877?

[radek-kondziolka]

Yes, #15877 was closed. Let's wait with merging until preparing PR to documentation staff.

[gaurav8297]

Instead of having a separate rule, you can do this directly inside AddExchange#visitTableWriter

[radek-kondziolka]

Yes, it could. I do it here because:

1. Most probably in the future we can make it adaptive basing on stats.
2. There is a rule that is similar in the sense of setting PartitioningScheme - ApplyPreferredTableExecutePartitioning. This rule is very simple, just basing on configuration toggle and could be part of AddExchanges as well.

This is why I decided to wrap it within another rule.

[gaurav8297]

TBH, I don't think we need a separate rule for this since we are just applying the configuration. In ApplyPreferredTableExecutePartitioning we are taking a decision based on estimates to use preferred partitioning or not. So, Its kinda makes sense.

[gaurav8297]

And in future if we ever make it adaptive based on stats we can always add a new rule and remove from AddExchanges. It shouldn't be a big change.

[gaurav8297]

WDYT? @sopel39

[sopel39]

I think the easier approach is preferred. If it's just few lines of code, then AddExchange#visitTableWriter is good enough.

Be sure to have proper testing coverage. With AddExchange I think you have to have BasePlanTest kind of tests

[radek-kondziolka]

@gaurav8297 , the rule LimitMaxWriterNodesCount is now much complicated. Especially, we skip the rule in some cases that is not skipped in AddExchange rule. I do not think that we should merge them. It is complicated it seems to be at the beginning.

[hashhar]

Have we also considered making the target catalog's connector participate in deciding how many writer tasks to use? Clusters can be connected to very homogenous systems so for example one might want all nodes to write to object storage or Kafka but only limited number of writers to databases.

[sopel39]

Have we also considered making the target catalog's connector participate in deciding how many writer tasks to use?

You mean round-robin writers or partitioned? For partitioned connector can always assigned fixed partition->node mapping

[hashhar]

For round-robin writers.

EDIT: Is it possible to partition writes on some column which is actually not part of the output? e.g. If I wanted to limit writes to 2 nodes maybe I can partition the output on some generated column and assign to two nodes. That would also achieve what I'm trying to think of.

[radek-kondziolka]

@hashhar ,

Is it possible to partition writes on some column which is actually not part of the output?

For now it is not possible.

[radek-kondziolka]

@hashhar , I am closing this one and I've opened this one: #16238
Beyond to that, I've added possibility for connectors to decide on number of tasks / nodes that take part in writing.