proxy: support timings for /infill from llama-server

[mostlygeek]

fixes: #463

Summary by CodeRabbit

• Bug Fixes
  • Fixed metrics collection for the /infill endpoint to properly extract timing data and accurately report token usage (input, output, and cached tokens) and performance metrics including throughput rates and response duration calculations.

[coderabbitai]

Walkthrough

The changes add special-case handling for /infill requests to extract timing metrics from the last element of parsed JSON array responses instead of from the root object, addressing metrics reporting failures for the /infill endpoint.

Changes

Cohort / File(s)	Summary
Metrics Monitor /infill Handling proxy/metrics_monitor.go, proxy/metrics_monitor_test.go	Adds conditional logic to extract timings from the last JSON array element for /infill endpoint responses. Includes tests for successful array parsing and empty array edge cases, verifying token counts and timing calculations.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• proxy: extract metrics for v1/messages #419: Modifies metrics extraction logic in proxy/metrics_monitor.go to adjust how timings are parsed from JSON responses.
• Include metrics from upstream chat requests #361: Refactors metrics extraction code in the same file, potentially affected by or related to this /infill-specific parsing change.
• Support llama-server's /infill endpoint #272: Introduces /infill endpoint support and establishes the initial metrics parsing/recording infrastructure that this PR extends.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding support for extracting timings/metrics from /infill responses through llama-server.
Linked Issues check	✅ Passed	The PR correctly addresses issue #463 by implementing special-case handling for /infill to extract timings from the JSON array response, restoring metrics reporting [#463].
Out of Scope Changes check	✅ Passed	All changes are scoped to /infill metrics handling in proxy/metrics_monitor.go and corresponding tests; no extraneous modifications detected.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch mg/infill-timings-463

---

No actionable comments were generated in the recent review. 🎉

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}