google-genai: include thinking tokens in output token count

[Rima-ag]

Description

Updates the Google GenAI instrumentation to include thinking tokens in the total output token count.

Previously only candidate tokens were counted in the output_tokens, resulting in inaccurate cost estimations for models that produce separate thinking tokens.

Thinking tokens are reported in GenerateContentResponseUsageMetadata.thoughts_token_count.

#3890
Related to open-telemetry/semantic-conventions#3194

Type of change

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

• Unit tests

Does This PR Require a Core Repo Change?

• Yes. - Link to PR:
• No.

Checklist:

See contributing.md for styleguide, changelog guidelines, and more.

• Followed the style guidelines of this project
• Changelogs have been updated
• Unit tests have been added
• Documentation has been updated

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Rima-ag / name: Rima Al Ghossein (122daf0, 2331945, 5ca1807, 9d08a66, a1f2705, c5d0c98, d318cf9)
• ✅ login: Rima-ag / name: Rima-ag (57f0c66, 5ed7c8d, d4943fc)

[Rima-ag]

i just saw open-telemetry/semantic-conventions#3383 that adds gen_ai.usage.reasoning.output_tokens to spans, should i followup on this PR by filling this field once the semantic convention is merged?

[aabmass]

I think @DylanRussell can help decide here. It would be nice to get the general convention in as well. At least we should define a gcp.* one if it's not generalizable.

[DylanRussell]

Isn't open-telemetry/semantic-conventions#3383 a general convention for an attribute on both logs on spans ? Yes I think we should wait for that to be merged if this isn't urgent, there are still a couple unresolved things on that PR

[DylanRussell]

Ah forgot this was open and opened the exact same PR: #4493

[Rima-ag]

@DylanRussell I just resolved the conflicts, would you mind having a look?

I have a followup draft PR #4276 to report thinking tokens in a separate field too and will request review as soon as this one is merged

[DylanRussell]

Rima I think we should go ahead and merge my PR which is all ready: #4493