Repository navigation
Dont require defining evals twice - #707
Conversation
## Walkthrough
The static definition of allowed evaluation tags was replaced with a dynamic approach that reads pytest marker tags from the `pyproject.toml` file at runtime. Helper functions were added to extract and return these tags, and necessary imports were updated to support TOML parsing and type handling. The `tomli` package was added as a development dependency for Python versions below 3.11 to enable TOML parsing. Two new pytest markers, including one for Kafka-related tests, were also introduced.
## Changes
| File(s) | Change Summary |
|-----------------------------|---------------------------------------------------------------------------------------------------------------------|
| tests/llm/utils/constants.py | Replaced static `ALLOWED_EVAL_TAGS` with a dynamically generated `Literal` based on `pyproject.toml` markers; added `_get_allowed_eval_tags()` and `get_allowed_tags_list()` helper functions; updated imports for TOML parsing and typing utilities. |
| pyproject.toml | Added `tomli` as a development dependency for Python <3.11 under `[tool.poetry.group.dev.dependencies]`; added new pytest markers `"kafka: Tests involving Kafka functionality"` and `"missing-tool: Tests where required tools are unavailable"` under `[tool.pytest.ini_options.markers]`. |
## Estimated code review effort
🎯 2 (Simple) | ⏱️ ~8 minutes
## Possibly related PRs
- robusta-dev/holmesgpt#597: Adds new static tags to the `ALLOWED_EVAL_TAGS` literal and updates `pyproject.toml` markers, closely related to the dynamic tag generation changes in this PR.Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
tests/llm/utils/constants.py(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
tests/llm/utils/constants.py (2)
Learnt from: nherment
PR: #436
File: tests/llm/utils/mock_utils.py:240-249
Timestamp: 2025-06-05T12:23:27.634Z
Learning: The holmesgpt project uses Python >= 3.10 and prefers modern type hint syntax like list[str], dict[str, int] over importing equivalent types from the typing module like List[str], Dict[str, int].
Learnt from: nherment
PR: #610
File: .github/workflows/llm-evaluation.yaml:39-42
Timestamp: 2025-07-08T08:45:41.069Z
Learning: The robusta-dev/holmesgpt codebase has comprehensive existing validation for Azure environment variables (AZURE_API_BASE, AZURE_API_KEY, AZURE_API_VERSION) and MODEL in tests/llm/utils/classifiers.py, tests/llm/conftest.py, and holmes/core/llm.py. Don't suggest adding redundant validation logic.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Pre-commit checks
- GitHub Check: Pre-commit checks
🔇 Additional comments (2)
tests/llm/utils/constants.py (2)
44-44: Assignment looks good once function issues are resolved.This assignment is straightforward and will work correctly once the error handling issues in
_get_allowed_eval_tags()are addressed.
48-50: Good utility function for debugging.This function provides a clean way to inspect the allowed tags at runtime. The implementation is correct and will work properly once the Literal type construction is fixed in
_get_allowed_eval_tags().
2 participants
No description provided.