Skip to content

feat(eval): add run locomo eval script - #28

Merged
Ki-Seki merged 7 commits into
MemTensor:devfrom
Duguce:dev
Jul 8, 2025
Merged

feat(eval): add run locomo eval script#28
Ki-Seki merged 7 commits into
MemTensor:devfrom
Duguce:dev

Conversation

@Duguce

@Duguce Duguce commented Jul 8, 2025

Copy link
Copy Markdown
Contributor

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes # (issue)

Type of change

Please delete options that are not relevant.

  • The last PR missed the run locomo eval script, submitting a supplement now. @Ki-Seki

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Please delete options that are not relevant.

  • Unit Test
  • Test Script (please provide)

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have checked my code and corrected any misspellings

Maintainer Checklist

  • closes #xxxx (Replace xxxx with the GitHub issue number)
  • Made sure Checks passed

Copilot AI review requested due to automatic review settings July 8, 2025 12:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces a new shell script to sequentially run all steps of the Locomo evaluation pipeline.

  • Adds run_locomo_eval.sh to orchestrate ingestion, search, response generation, evaluation, and metric calculation.
  • Sets default parameters (LIB, VERSION, WORKERS, TOPK) and checks exit codes for each step.

@@ -0,0 +1,44 @@
#!/bin/bash

Copilot AI Jul 8, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding set -euo pipefail after the shebang to ensure the script exits on any error and detects undefined variables.

Copilot uses AI. Check for mistakes.
Comment on lines +9 to +42
echo "Running locomo_ingestion.py..."
CUDA_VISIBLE_DEVICES=0 python scripts/locomo/locomo_ingestion.py --lib $LIB --version $VERSION --workers $WORKERS
if [ $? -ne 0 ]; then
echo "Error running locomo_ingestion.py"
exit 1
fi

echo "Running locomo_search.py..."
CUDA_VISIBLE_DEVICES=0 python scripts/locomo/locomo_search.py --lib $LIB --version $VERSION --top_k $TOPK --workers $WORKERS
if [ $? -ne 0 ]; then
echo "Error running locomo_search.py"
exit 1
fi

echo "Running locomo_responses.py..."
python scripts/locomo/locomo_responses.py --lib $LIB --version $VERSION
if [ $? -ne 0 ]; then
echo "Error running locomo_responses.py."
exit 1
fi

echo "Running locomo_eval.py..."
python scripts/locomo/locomo_eval.py --lib $LIB --version $VERSION --workers $WORKERS --num_runs 3
if [ $? -ne 0 ]; then
echo "Error running locomo_eval.py"
exit 1
fi

echo "Running locomo_metric.py..."
python scripts/locomo/locomo_metric.py --lib $LIB --version $VERSION
if [ $? -ne 0 ]; then
echo "Error running locomo_metric.py"
exit 1
fi

Copilot AI Jul 8, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] The error-checking and execution pattern is repeated for each script; consider extracting a helper function (e.g., run_step) to reduce duplication and improve readability.

Suggested change
echo "Running locomo_ingestion.py..."
CUDA_VISIBLE_DEVICES=0 python scripts/locomo/locomo_ingestion.py --lib $LIB --version $VERSION --workers $WORKERS
if [ $? -ne 0 ]; then
echo "Error running locomo_ingestion.py"
exit 1
fi
echo "Running locomo_search.py..."
CUDA_VISIBLE_DEVICES=0 python scripts/locomo/locomo_search.py --lib $LIB --version $VERSION --top_k $TOPK --workers $WORKERS
if [ $? -ne 0 ]; then
echo "Error running locomo_search.py"
exit 1
fi
echo "Running locomo_responses.py..."
python scripts/locomo/locomo_responses.py --lib $LIB --version $VERSION
if [ $? -ne 0 ]; then
echo "Error running locomo_responses.py."
exit 1
fi
echo "Running locomo_eval.py..."
python scripts/locomo/locomo_eval.py --lib $LIB --version $VERSION --workers $WORKERS --num_runs 3
if [ $? -ne 0 ]; then
echo "Error running locomo_eval.py"
exit 1
fi
echo "Running locomo_metric.py..."
python scripts/locomo/locomo_metric.py --lib $LIB --version $VERSION
if [ $? -ne 0 ]; then
echo "Error running locomo_metric.py"
exit 1
fi
# Helper function to run a script and check for errors
run_step() {
local script=$1
shift
echo "Running $script..."
"$@"
if [ $? -ne 0 ]; then
echo "Error running $script"
exit 1
fi
}
run_step "locomo_ingestion.py" CUDA_VISIBLE_DEVICES=0 python scripts/locomo/locomo_ingestion.py --lib $LIB --version $VERSION --workers $WORKERS
run_step "locomo_search.py" CUDA_VISIBLE_DEVICES=0 python scripts/locomo/locomo_search.py --lib $LIB --version $VERSION --top_k $TOPK --workers $WORKERS
run_step "locomo_responses.py" python scripts/locomo/locomo_responses.py --lib $LIB --version $VERSION
run_step "locomo_eval.py" python scripts/locomo/locomo_eval.py --lib $LIB --version $VERSION --workers $WORKERS --num_runs 3
run_step "locomo_metric.py" python scripts/locomo/locomo_metric.py --lib $LIB --version $VERSION

Copilot uses AI. Check for mistakes.
@Ki-Seki
Ki-Seki merged commit 6148813 into MemTensor:dev Jul 8, 2025
tianxing02 pushed a commit to tianxing02/MemOS that referenced this pull request Feb 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants