Skip to content

Recalculate Gemini-3.5-Flash swe-bench costs ($3.8253/instance) - #1179

Closed
all-hands-bot wants to merge 1 commit into
eval/Gemini-3.5-Flash/swe-bench-20260603-072155from
recalculate-costs/Gemini-3.5-Flash-swe-bench-20260605-005208
Closed

Recalculate Gemini-3.5-Flash swe-bench costs ($3.8253/instance)#1179
all-hands-bot wants to merge 1 commit into
eval/Gemini-3.5-Flash/swe-bench-20260603-072155from
recalculate-costs/Gemini-3.5-Flash-swe-bench-20260605-005208

Conversation

@all-hands-bot

Copy link
Copy Markdown
Collaborator

🧮 Cost Recalculation for swe-bench

Archive URL: https://results.eval.all-hands.dev/swebench/litellm_proxy-gemini-3-5-flash/26622025994/results.tar.gz

Pricing Notes

  • cache_read_price from metadata.json was used for cache hit costs.
  • ℹ️ cache_write_price was ignored in this calculation.

Token Usage Summary

Metric Value
Instances processed 500
Total prompt tokens 1,439,426,049
Cache read tokens 315,611,980
Non-cached prompt tokens 1,123,814,069
Completion tokens 19,953,716

Pricing Used (from metadata.json)

Token Type Cost per Million
Input (cache miss) $1.5000
Input (cache hit) $0.1500
Output $9.0000

Cost Breakdown

Component Tokens Rate Cost
Non-cached input 1,123,814,069 $1.5000/M $1685.7211
Cached input 315,611,980 $0.1500/M $47.3418
Output 19,953,716 $9.0000/M $179.5834
Total $1912.6463

Final Result

  • Total cost: $1912.6463
  • Average cost per instance: $3.8253

This comment was automatically generated by the recalculate-costs action.

Pricing from metadata.json:
Input (cache miss): $1.5/M tokens
Input (cache hit): $0.15/M tokens
Output: $9.0/M tokens
cache_write_price: ignored
New cost_per_instance: $3.8253
@all-hands-bot

Copy link
Copy Markdown
Collaborator Author

🧮 Cost Recalculation for swe-bench

Archive URL: https://results.eval.all-hands.dev/swebench/litellm_proxy-gemini-3-5-flash/26622025994/results.tar.gz

Pricing Notes

  • cache_read_price from metadata.json was used for cache hit costs.
  • ℹ️ cache_write_price was ignored in this calculation.

Token Usage Summary

Metric Value
Instances processed 500
Total prompt tokens 1,439,426,049
Cache read tokens 315,611,980
Non-cached prompt tokens 1,123,814,069
Completion tokens 19,953,716

Pricing Used (from metadata.json)

Token Type Cost per Million
Input (cache miss) $1.5000
Input (cache hit) $0.1500
Output $9.0000

Cost Breakdown

Component Tokens Rate Cost
Non-cached input 1,123,814,069 $1.5000/M $1685.7211
Cached input 315,611,980 $0.1500/M $47.3418
Output 19,953,716 $9.0000/M $179.5834
Total $1912.6463

Final Result

  • Total cost: $1912.6463
  • Average cost per instance: $3.8253

This comment was automatically generated by the recalculate-costs action.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants