feat: Update scaling policy

[Kavinjsir]

Update:

• Prevent downscaling

Should Partially resolve: https://github.com/rilldata/rill-private-issues/issues/475

[Kavinjsir]

@begelundmuller Not quite sure how to figure the slots that is specified by "manual intervene".

OR if is possible/necessary to have a new method for the DB connect struct to find the info from the Database?

Would you have some recommendations?

[begelundmuller]

I guess you would need to add a column with a flag in the database. Or use an annotation to indicate it (such as has_sla or something else – the annotation could be automatically added if a user calls rill project edit --prod-slots ...).

[Kavinjsir]

@begelundmuller, following some discussions with the infra team, we've decided to temporarily disable down-scaling.

Regarding the disabling of scaling for manually-specified slots, I have left a comment seeking further assistance.

Concerning the cap at 20%, I am guessing that temporarily disabling down-scaling may suffice for our needs.
However, I am concerned that this cap might become a bottleneck, especially when scaling from smaller values (e.g., scaling from 4 to 8).
On the other hand, I'm cautious about introducing overly complex logic to determine the slots, considering we already have a recommended value for slots.

Additionally, I'm wondering about the has-sla tagging.
Is this a widely applicable tag?
If we disable scaling for projects tagged with has-sla, it might impact a large number of our production-level services, which are primarily scaled for flexibility and cost savings.

cc: @himadrisingh @kaspersjo

[himadrisingh]

Can we make this configurable through some property/env vars?

[begelundmuller]

Concerning the cap at 20%, I am guessing that temporarily disabling down-scaling may suffice for our needs. However, I am concerned that this cap might become a bottleneck, especially when scaling from smaller values (e.g., scaling from 4 to 8). On the other hand, I'm cautious about introducing overly complex logic to determine the slots, considering we already have a recommended value for slots.

A 20% cap (or similar) should only apply to downscaling. Upscaling should not have a cap since it might scale too slowly then. Since you disabled downscaling, this doesn't really matter right now.

Additionally, I'm wondering about the has-sla tagging. Is this a widely applicable tag? If we disable scaling for projects tagged with has-sla, it might impact a large number of our production-level services, which are primarily scaled for flexibility and cost savings.

You can ask @AndrewRTsao. I think he's the one who is using this annotation.

[AndrewRTsao]

As discussed previously, we are using the following tags:

• sla=prod to indicate projects that are considered in production
• sla=dev to indicate projects that are under active development
• dashboard_latency=10s for the expected dashboard latency before we trigger alerts (we are currently defaulting to 10s for duckdb-backed projects and 20s for Druid-backed)