feat(eval): add eval configs example - #19
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR introduces example configuration and documentation updates for the evaluation framework, modernizes dependency management, and consolidates memory backend handling by removing legacy code.
- Added new dependencies to
pyproject.tomlfor evaluation configs, environment loading, and model support. - Refactored the
get_clientand search/response scripts to unify thememosbackend (removingmemos_mos) and added atop_kparameter. - Updated CLI choices across scripts and streamlined the README to reference a
.env-exampleandconfigs-exampledirectory.
Reviewed Changes
Copilot reviewed 8 out of 10 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| pyproject.toml | Added scipy, python-dotenv, pydantic, langgraph, and langmem to project dependencies |
| evaluation/scripts/locomo/locomo_search.py | Unified memos backend code, removed legacy blocks, and added top_k parameter support |
| evaluation/scripts/locomo/locomo_ingestion.py | Removed legacy memos and memos_mos ingestion logic; now uses consolidated backend |
| evaluation/README.md | Simplified setup instructions, removed LongMemEval references, and pointed to run_locomo_eval.sh |
| evaluation/.env-example | Introduced an example .env with placeholder API keys and model settings |
Comments suppressed due to low confidence (4)
evaluation/scripts/locomo/locomo_ingestion.py:126
- The
ingest_sessionfunction no longer includes a branch to handle thememosframe, so data will not be ingested for that backend. Restoring or adding appropriate ingestion logic formemosis needed.
)
evaluation/scripts/locomo/locomo_search.py:445
- The help string for the
--libflag lists frameworks but omitslangmem(and optionallyopenai), leading to a mismatch with thechoiceslist. Please update the help text to reflect the actual available options.
help="Specify the memory framework (zep or memos or mem0 or mem0_graph)",
evaluation/README.md:27
- [nitpick] The instructions now refer to
run_locomo_eval.shbut don't explain where to find or how to configure it, nor the steps for ingestion and search. Consider elaborating the evaluation workflow or including the script contents for clarity.
### LoCoMo Evaluation
pyproject.toml:54
- [nitpick] Added dependency lines appear to lack the same indentation as existing entries in the
[tool.poetry.dependencies]section, which may affect readability or formatting. Consider aligning them with the existing entries.
scipy = "^1.10.1"
| bert-score = "^0.3.13" | ||
| scipy = "^1.10.1" | ||
| python-dotenv = "^1.1.1" | ||
| pydantic = "^2.11.7" |
There was a problem hiding this comment.
This dependency (pydantic) can be removed since it is implicitly required by other dependencies and does not need to be explicitly specified. After deleting this line, run poetry lock to update the poetry.lock.
该依赖项(pydantic)可以被移除,因为它已被其他依赖项隐式引入,无需显式指定。删除此行后,请运行 poetry lock 以更新 poetry.lock。
|
@Ki-Seki I have solved the two problems. |
feat: update installation instructions and add platform compatibility notes
* feat(eval): add eval dependencies * feat(eval): add configs example * docs(eval): update README.md * feat(eval): remove the dependency (pydantic)
* fix: aysnc node updated_at not being updated * fix: fix restored_memories version bug * feat: optimize preserved memory dealing in mem versions * fix: fix test * feat: temporarily delete get_maybe_merged logic for benchmarks
Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Please delete options that are not relevant.
Checklist:
Maintainer Checklist