feat(eval): add openai memory on locomo - eval guide by Duguce · Pull Request #54 · MemTensor/MemOS

Duguce · 2025-07-10T10:17:20Z

Description

Summary: add openai memory on locomo - eval guide

Fix: #(issue)

Reviewer: @hush-cd

Checklist:

I have performed a self-review of my own code | 我已自行检查了自己的代码
I have commented my code in hard-to-understand areas | 我已在难以理解的地方对代码进行了注释
I have added tests that prove my fix is effective or that my feature works | 我已添加测试以证明我的修复有效或功能正常
I have added necessary documentation (if applicable) | 我已添加必要的文档（如果适用）
I have linked the issue to this PR (if applicable) | 我已将 issue 链接到此 PR（如果适用）
I have mentioned the person who will review this PR | 我已提及将审查此 PR 的人

Copilot

Pull Request Overview

This PR adds support for evaluating OpenAI’s Memory feature on the LoCoMo dataset by providing automation scripts, a detailed guide, and an evaluation tool.

Introduces run_openai_eval.sh to orchestrate the LoCoMo OpenAI memory evaluation pipeline
Adds a step-by-step Markdown guide for manual memory extraction and consolidation
Implements locomo_openai.py to automate question answering against stored memories
Updates the main README.md to reference the new guide

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
evaluation/scripts/run_openai_eval.sh	Shell script to run the OpenAI-based evaluation steps
evaluation/scripts/locomo/openai_memory_locomo_eval_guide.md	Manual guide for extracting and consolidating ChatGPT memories
evaluation/scripts/locomo/locomo_openai.py	Python script to query stored memories and evaluate responses
evaluation/README.md	README update linking to the OpenAI memory evaluation guide

Comments suppressed due to low confidence (1)

evaluation/README.md:36

[nitpick] It would be helpful to include the exact command for running the new evaluation script (e.g., ./evaluation/scripts/run_openai_eval.sh --version <version>), so users don’t have to look it up.

✍️ For evaluating OpenAI's native memory feature with the LoCoMo dataset, please refer to the detailed guide: [OpenAI Memory on LoCoMo - Evaluation Guide](./scripts/locomo/openai_memory_locomo_eval_guide.md).

Copilot · 2025-07-10T10:18:40Z

+echo "Running locomo_openai.py..."
+python scripts/locomo/locomo_openai.py --version $VERSION
+if [ $? -ne 0 ]; then
+    echo "Error running locomo_openai.py."
+    exit 1
+fi
+
+echo "Running locomo_eval.py..."
+python scripts/locomo/locomo_eval.py --lib $LIB --version $VERSION --num_runs $NUM_RUNS
+if [ $? -ne 0 ]; then
+    echo "Error running locomo_eval.py"
+    exit 1
+fi
+
+echo "Running locomo_metric.py..."
+python scripts/locomo/locomo_metric.py --lib $LIB --version $VERSION


The relative path scripts/locomo/locomo_openai.py will be resolved from the current working directory, which may not be the script’s location. Consider using the script’s directory for paths (e.g., DIR=$(dirname "$0") and then python "$DIR"/locomo/locomo_openai.py --version "$VERSION").

Suggested change

echo "Running locomo_openai.py..."

python scripts/locomo/locomo_openai.py --version $VERSION

if [ $? -ne 0 ]; then

echo "Error running locomo_openai.py."

exit 1

fi

echo "Running locomo_eval.py..."

python scripts/locomo/locomo_eval.py --lib $LIB --version $VERSION --num_runs $NUM_RUNS

if [ $? -ne 0 ]; then

echo "Error running locomo_eval.py"

exit 1

fi

echo "Running locomo_metric.py..."

python scripts/locomo/locomo_metric.py --lib $LIB --version $VERSION

DIR=$(dirname "$0")

echo "Running locomo_openai.py..."

python "$DIR"/scripts/locomo/locomo_openai.py --version $VERSION

if [ $? -ne 0 ]; then

echo "Error running locomo_openai.py."

exit 1

fi

echo "Running locomo_eval.py..."

python "$DIR"/scripts/locomo/locomo_eval.py --lib $LIB --version $VERSION --num_runs $NUM_RUNS

if [ $? -ne 0 ]; then

echo "Error running locomo_eval.py"

exit 1

fi

echo "Running locomo_metric.py..."

python "$DIR"/scripts/locomo/locomo_metric.py --lib $LIB --version $VERSION

Copilot · 2025-07-10T10:18:41Z

+class OpenAIPredict:
+    def __init__(self, model="gpt-4o-mini"):
+        self.model = model
+        self.openai_client = OpenAI(
+            api_key=os.getenv("OPENAI_API_KEY"), base_url=os.getenv("OPENAI_BASE_URL")
+        )
+        self.results = defaultdict(list)
+
+    def search_memory(self, idx):
+        with open(f"openai_memory/{idx}.txt", encoding="utf-8") as file:
+            memories = file.read().strip().replace("\n\n", "\n")
+
+        return memories, 0
+


The code assumes an openai_memory/ directory exists and hardcodes that path, but the guide uses openai_inputs/ and doesn’t mention creating openai_memory/. Either parameterize this directory or document/setup the folder before reading.

Suggested change

class OpenAIPredict:

def __init__(self, model="gpt-4o-mini"):

self.model = model

self.openai_client = OpenAI(

api_key=os.getenv("OPENAI_API_KEY"), base_url=os.getenv("OPENAI_BASE_URL")

)

self.results = defaultdict(list)

def search_memory(self, idx):

with open(f"openai_memory/{idx}.txt", encoding="utf-8") as file:

memories = file.read().strip().replace("\n\n", "\n")

return memories, 0

class OpenAIPredict:

def __init__(self, model="gpt-4o-mini", memory_dir="openai_memory"):

self.model = model

self.memory_dir = memory_dir

os.makedirs(self.memory_dir, exist_ok=True)

self.openai_client = OpenAI(

api_key=os.getenv("OPENAI_API_KEY"), base_url=os.getenv("OPENAI_BASE_URL")

)

self.results = defaultdict(list)

def search_memory(self, idx):

memory_file_path = os.path.join(self.memory_dir, f"{idx}.txt")

with open(memory_file_path, encoding="utf-8") as file:

memories = file.read().strip().replace("\n\n", "\n")

return memories, 0

Copilot · 2025-07-10T10:18:41Z

+    def search_memory(self, idx):
+        with open(f"openai_memory/{idx}.txt", encoding="utf-8") as file:
+            memories = file.read().strip().replace("\n\n", "\n")
+
+        return memories, 0
+


Currently search_memory always returns 0 for the search time. For accurate metrics, consider measuring and returning the actual file‐read duration using time.time().

Suggested change

def search_memory(self, idx):

with open(f"openai_memory/{idx}.txt", encoding="utf-8") as file:

memories = file.read().strip().replace("\n\n", "\n")

return memories, 0

def search_memory(self, idx):

t1 = time.time()

with open(f"openai_memory/{idx}.txt", encoding="utf-8") as file:

memories = file.read().strip().replace("\n\n", "\n")

t2 = time.time()

search_time = (t2 - t1) * 1000 # Convert to milliseconds

return memories, search_time

hush-cd

The file evaluation/scripts/locomo/openai_memory_locomo_eval_guide.md requires partial revisions.

hush-cd · 2025-07-10T12:24:38Z

+4.  **Save Memories:**
+    * Click on **Manage** in the memory confirmation to view the newly generated memories.
+    * Create a new local `.txt` file with the same name as the input file (e.g., `0-D1.txt`).
+    * Copy each memory entry from ChatGPT and paste it into the new file, with each memory on a new line.


It is important to emphasize that once all sessions under a single conversation have been processed and their memories saved, all memories in ChatGPT must be deleted(Settings -> Personalization -> Manage memories -> Delete all) before starting memory extraction for the next conversation. This prevents interference between memories from different conversations.

hush-cd · 2025-07-10T12:26:24Z

+## 3. Considerations
+
+-   **Account Differences:** Be aware of potential differences between free and Plus accounts, such as context length limitations and the number of memories that can be stored.
+-   **Granularity:** The current process adds memories at a **session** level. This is due to the context window limitations of the web UI. Evaluating at a full **conversation** level at once might be preferable if the context window allows.


Memory extraction should be performed at the session level only. It is not recommended to input the entire conversation at once, as this often results in very few memories being extracted, leading to significant information loss.

Duguce · 2025-07-10T15:15:07Z

@hush-cd has been reviewed; Please help me merge this branch @Ki-Seki

* feat(eval): add eval dependencies * feat(eval): add configs example * docs(eval): update README.md * feat(eval): remove the dependency (pydantic) * feat(eval): add run locomo eval script * fix(eval): delete about memos redundant search branches * chore: fix format * feat(eval): add openai memory on locomo - eval guide * docs(eval): modify openai memory on locomo - eval guide

Duguce and others added 13 commits July 7, 2025 22:43

feat(eval): add eval dependencies

4ef7418

feat(eval): add configs example

ddad8c1

docs(eval): update README.md

d623791

Merge branch 'MemTensor:dev' into dev

3365de4

feat(eval): remove the dependency (pydantic)

ed68e36

Merge branch 'MemTensor:dev' into dev

4e99031

feat(eval): add run locomo eval script

41368b9

Merge branch 'MemTensor:dev' into dev

9477539

fix(eval): delete about memos redundant search branches

8cd9361

chore: fix format

a900910

Merge branch 'MemTensor:dev' into dev

204bd27

Merge branch 'MemTensor:dev' into dev

5d68ed9

feat(eval): add openai memory on locomo - eval guide

42e9366

Copilot AI review requested due to automatic review settings July 10, 2025 10:17

Copilot AI reviewed Jul 10, 2025

View reviewed changes

Merge branch 'dev' into dev

f3d1d5d

hush-cd reviewed Jul 10, 2025

View reviewed changes

Duguce and others added 2 commits July 10, 2025 22:56

Merge branch 'MemTensor:dev' into dev

9bada64

docs(eval): modify openai memory on locomo - eval guide

7881029

hush-cd approved these changes Jul 10, 2025

View reviewed changes

Ki-Seki merged commit 97fdb06 into MemTensor:dev Jul 10, 2025

Navigation Menu

Uh oh!

feat(eval): add openai memory on locomo - eval guide - #54

feat(eval): add openai memory on locomo - eval guide#54
Ki-Seki merged 16 commits into
MemTensor:devfrom
Duguce:dev

Duguce commented Jul 10, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jul 10, 2025

Uh oh!

Copilot AI Jul 10, 2025

Uh oh!

Copilot AI Jul 10, 2025

Uh oh!

hush-cd left a comment

Uh oh!

hush-cd Jul 10, 2025

Uh oh!

hush-cd Jul 10, 2025

Uh oh!

Duguce commented Jul 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Duguce commented Jul 10, 2025

Description

Checklist:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

hush-cd left a comment

Choose a reason for hiding this comment

Uh oh!

hush-cd Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

hush-cd Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

Duguce commented Jul 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants