feat: Add Get Token Metrics to GRPC server #3687
Conversation
Signed-off-by: Siddharth More <[email protected]>
✅ Deploy Preview for localai ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
siddimore
changed the title
| // Define the empty request | ||
| message MetricsRequest {} | ||
|
|
||
| message MetricsResponse { |
There was a problem hiding this comment.
what other metrics are good to be exposed out?
There was a problem hiding this comment.
Maybe not related to the processing request, but could be useful also to expose infos like how many parallel requests can be served
|
mmm I wonder if it would make sense to add a route also via http rest to leverage this, but shouldn't block merging this PR |
| @@ -374,3 +374,21 @@ func (c *Client) Rerank(ctx context.Context, in *pb.RerankRequest, opts ...grpc. | |||
| client := pb.NewBackendClient(conn) | |||
| return client.Rerank(ctx, in, opts...) | |||
| } | |||
|
|
|||
| func (c *Client) GetTokenMetrics(ctx context.Context, in *pb.MetricsRequest, opts ...grpc.CallOption) (*pb.MetricsResponse, error) { | |||
| if !c.parallel { | |||
There was a problem hiding this comment.
Code doesn't look formatted here - might need a go fmt
There was a problem hiding this comment.
forgot to fix this
@mudler yup i already did make those changes just did not add it yet because i had to investigate some build issue |
gotcha 👍 changes looks good here otherwise! |
Signed-off-by: Siddharth More <[email protected]>
@mudler added a route to http endpoints. I will update test and have this PR ready for review this week |
Description
This PR creates a GRPC method to expose some of the Token Metrics
TODO
Add tests
Signed commits