Update Serverless docs for deprecations and new SDK #33
Conversation
| - `target_util` (float): A ratio that determines how much spare capacity (headroom) the serverless engine maintains. Default value is 0.9. | ||
| - `cold_mult`(float): A multiplier applied to your target capacity for longer-term planning (1+ hours). This parameter controls how much extra capacity the serverless engine will plan for in the future compared to immediate needs. Default value is 3.0. | ||
| - `test_workers` (integer): The number of different physical machines that a Workergroup should test during its initial "exploration" phase to gather performance data before transitioning to normal demand-based scaling. Default value is 3. | ||
| - `gpu_ram` (integer): The amount of GPU memory (VRAM) in gigabytes that your model or workload requires to run. This parameter tells the serverless engine how much GPU memory your model needs. Default value is 24. |
There was a problem hiding this comment.
We should just remove this one too
There was a problem hiding this comment.
Including the lines below too
There was a problem hiding this comment.
gpu_ram actually is used on the per-workergroup level, no need to remove it. Unless I'm misreading your comment
There was a problem hiding this comment.
Correct, but it looks weird to include that in the cli commands. Its almost always in the template. I was just pointing it out to simplify
There was a problem hiding this comment.
Check me if I'm wrong, but either we get the gpu_ram field for autogroups from the CLI here, or it defaults to 8. Check create_autojobs(request) in client.py and create__workergroup(args) in vast.py. I can't find any example of sourcing this parameter from the template.
There was a problem hiding this comment.
There was a problem hiding this comment.
| ## gpu\_ram | ||
|
|
||
| The amount of GPU memory (VRAM) in gigabytes that your model or workload requires to run. This parameter tells the serverless engine how much GPU memory your model needs. | ||
| The amount of GPU memory (VRAM) in gigabytes that your model or workload requires to run. This parameter tells the serverless engine how much GPU memory your model needs, and is primarily used to detect unusually long model load times. |
There was a problem hiding this comment.
Sure, we can remove this
Removes old deprecated worker group parameters
Adds request_idx to route payload/response
Includes instructions for using the vastai-sdk