fix(ci): make litellm_internal_staging green (logging_testing + Bedrock Opus) - #29341
Closed
mateo-berri wants to merge 2 commits into
Closed
fix(ci): make litellm_internal_staging green (logging_testing + Bedrock Opus)#29341mateo-berri wants to merge 2 commits into
mateo-berri wants to merge 2 commits into
Conversation
…edaction PR #28909 hardened log_db_metrics to emit a minimal, non-sensitive event_metadata (only table_name when present, otherwise None) instead of dumping function_name, function_kwargs, and function_args onto the span. The test in test_log_db_redis_services was not updated and still asserted "function_name" in event_metadata, which raised TypeError (argument of type 'NoneType' is not iterable) and turned the logging_testing CI job red on litellm_internal_staging. Update test_log_db_metrics_success to assert event_metadata is None when no table_name is passed, and add test_log_db_metrics_event_metadata_is_safe as a regression guard verifying that only the table name surfaces and that sensitive kwargs (tokens, prisma client) are never dumped.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…count #29326 (the Bedrock CI account revert) dropped the grid_spec fail_reason workaround for the unentitled opus-4-7 cells, on the assumption that the reactivated account 888602223428 has flagship Opus. A live Bedrock converse call to us.anthropic.claude-opus-4-7 with that account (both us-east-1 and us-west-2) returns "anthropic.claude-opus-4-7 is not available for this account", while opus-4-6-v1 succeeds; the credentials are valid and only opus-4-7 is unentitled (access requires an AWS Sales request, not self-serve). The dropped workaround turned the 8 opus-4-7 cells red in llm_translation_testing on litellm_internal_staging. Restore the ModelEntry.fail_reason field and the pytest.xfail consumer, and mark the bedrock-claude-opus-4-7 cell xfail with the account reference updated to 888602223428. The cells now xfail (loud and documented) instead of failing the suite; remove the fail_reason once opus-4-7 access is granted.
Merged
7 tasks
Contributor
Author
|
Superseded by #29344. Same two commits, moved to the Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
litellm_internal_stagingCI was red. GitHub Actions was fully green (75/75), but the CircleCIbuild_and_testworkflow on staging HEAD (f11c12d, pipeline #79824) had two failing jobs:logging_testingandllm_translation_testing. This PR fixes both.What this fixes
1.
logging_testing.#28909hardenedlog_db_metricsto emit a minimal, non-sensitiveevent_metadata(onlytable_namewhen present, otherwiseNone) instead of dumpingfunction_name,function_kwargs, andfunction_argsonto the span; the function name still rides on the separatecall_typeargument.test_log_db_redis_services.pywas not updated and still asserted"function_name" in event_metadata, so on the no-table_namepathevent_metadataisNoneand the assertion raisedTypeError: argument of type 'NoneType' is not iterable.This updates
test_log_db_metrics_successto assertevent_metadata is Nonefor that path and addstest_log_db_metrics_event_metadata_is_safe, a regression guard that only the table name surfaces and that sensitive kwargs (tokens, prisma client) are never dumped.2.
llm_translation_testing.All 8 failures were
test_reasoning_effort_grid[bedrock_converse-bedrock-claude-opus-4-7-*]withBedrockException - {"message":"anthropic.claude-opus-4-7 is not available for this account..."}. The revert in#29326dropped thegrid_specfail_reasonxfail workaround for the unentitled Opus cells on the assumption that the reactivated Bedrock account (888602223428) has flagship Opus. Live Bedrockconversecalls with that account (in both us-east-1 and us-west-2) confirmus.anthropic.claude-opus-4-6-v1works butus.anthropic.claude-opus-4-7returns "not available for this account", so opus-4-7 is genuinely unentitled there and access "requires an AWS Sales request, not self-serve". The CircleCIAWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEYalready point at 888602223428, so no credential rotation was needed.This restores the
ModelEntry.fail_reasonfield and thepytest.xfailconsumer, and marks thebedrock-claude-opus-4-7cell xfail (account reference updated to 888602223428). The 8 cells now xfail (loud and documented) instead of failing the suite; remove thefail_reasononce opus-4-7 access is granted.Stacked PR
#29327(Opus 4.8 reasoning-effort grid) depends on thefail_reasonmechanism this PR restores, so it has been rebased to branch off this PR. Its opus-4-8 Azure/Vertex/Bedrock cells use the same xfail guard, with the Bedrock account reference aligned to 888602223428.Pre-Submission checklist
Proof of Fix
This is a CI-only test fix; the proof is the two previously-red CircleCI jobs turning green. Before (staging HEAD pipeline #79824, jobs 1743867 / 1743865):
After (this branch, locally, with the CI Bedrock creds for the xfail path):
The CircleCI run on this PR is the live proof for both jobs.
Type
🐛 Bug Fix
✅ Test
Changes
tests/logging_callback_tests/test_log_db_redis_services.py: alignevent_metadataassertions with the redacted contract from#28909and add a redaction regression test.tests/llm_translation/reasoning_effort_grid/grid_spec.pyandtest_reasoning_effort_grid.py: restore theModelEntry.fail_reasonfield andpytest.xfailconsumer, and mark the unentitledbedrock-claude-opus-4-7cell xfail (account 888602223428).