Skip to content

Revise PR #339: the fabricated sha256 pins were removed but replaced with a fabricated byte size, wrong by 18.5x, and a false claim that the file does not exist on this machine #342

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jaylfc wants to merge 1 commit into from
Closed

Revise PR #339: the fabricated sha256 pins were removed but replaced with a fabricated byte size, wrong by 18.5x, and a false claim that the file does not exist on this machine #342

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 63 additions & 3 deletions benchmarks/data/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,10 @@ before trusting a published number.
- What it is: LongMemEval-S, the oracle variant of the LongMemEval long-term
memory benchmark, 500 questions, each with a haystack of conversation
sessions and the gold answer session ids.
- Used by: `benchmarks/longmemeval_enhanced.py` and the other
`longmemeval_*` runners. It is the source of the published 97.0% Recall@5
headline (see `benchmarks/REPRODUCE-longmemeval.md`).
- Used by: `combo_benchmark.py`, `longmemeval_enhanced.py`,
`realworld_llm_benchmark.py`, `embedding_comparison.py`,
`longmemeval_ku_runner.py`, `realworld_pipeline_benchmark.py`,
`fusion_shootout.py`, `longmemeval_matrix.py`, `variations_sweep.py`.
- Size: 277383467 bytes (about 265 MiB).
- Question count: 500.
- sha256: `d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442`
Expand All @@ -31,3 +32,62 @@ oracle set). Place the 500-question file at
`benchmarks/data/longmemeval_s_full.json` and confirm the checksum above. A
canonical pinned copy is also kept on the project bench host under the repo's
`benchmarks/data/` directory.

## longmemeval_oracle.json

- What it is: LongMemEval-S oracle set, the original oracle variant loaded by
`benchmarks/longmemeval_runner.py` and `benchmarks/recall_v2_benchmark.py`.
Each question has a haystack of conversation sessions and the gold answer
session ids.
- Relationship to `longmemeval_s_full.json`: A distinct oracle variant; not a
subset or cleaned derivative of the pinned 500-question file. Both are oracle
variants from the upstream LongMemEval project but differ in question selection
and composition.
- Question count: 500 (same as `longmemeval_s_full.json`).

@kilo-code-bot kilo-code-bot Bot Aug 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: Question count asserted without independent source

"Question count: 500 (same as longmemeval_s_full.json)" is stated as fact without citing a verifiable source. Given this PR's purpose of removing unverifiable numbers from the README, this claim should either be sourced from the upstream project or marked as unverified (as done for longmemeval_s_cleaned.json).

Reply with @kilocode-bot fix it to have Kilo Code address this issue.

- Byte size: 15388478 bytes (about 14.7 MiB), measured with `stat -c %s` on
a verified copy found in sibling repositories at
`/home/jay/Development/taOSr1-catalog/`, `/home/jay/Development/tinyagentos/`,

@kilo-code-bot kilo-code-bot Bot Aug 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: Hardcoded local filesystem paths exposed in shared README

These absolute paths expose the author's personal username and local directory structure in a public repository:

  • /home/jay/Development/taOSr1-catalog/
  • /home/jay/Development/tinyagentos/
  • /home/jay/Development/taOS-turnslice/

They are meaningless to other developers and should be removed or replaced with a general description (e.g., "verified copies in sibling repositories").

Reply with @kilocode-bot fix it to have Kilo Code address this issue.

and `/home/jay/Development/taOS-turnslice/`.
- sha256: `821a2034d219ab45846873dd14c14f12cfe7776e73527a483f9dac095d38620c`

@kilo-code-bot kilo-code-bot Bot Aug 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: Dataset metadata cannot be verified by external reviewers

The byte size and sha256 for longmemeval_oracle.json are documented as measured from "verified copies found in sibling repositories", but those repositories are not accessible to other developers. Since the file does not exist in this repository, no external reviewer can independently confirm these values. This recreates the same verification gap that caused the previous fabrication issue.

Reply with @kilocode-bot fix it to have Kilo Code address this issue.

(64 hex, valid). Verify with: `shasum -a 256 benchmarks/data/longmemeval_oracle.json`.
- How to obtain it: Get the LongMemEval-S oracle set from the upstream LongMemEval
project. Place the file at `benchmarks/data/longmemeval_oracle.json` and confirm
the checksum above. This file is distinct from `longmemeval_s_full.json` and is
not derived from it.

To verify a copy when available:

```bash
shasum -a 256 benchmarks/data/longmemeval_oracle.json
# expect: 821a2034d219ab45846873dd14c14f12cfe7776e73527a483f9dac095d38620c
```

*(Note: this file does not currently exist in this repository; three verified
copies exist in sibling repositories with the measurements above. Obtain from the
upstream LongMemEval project and confirm the checksum before use.)*

## longmemeval_s_cleaned.json

- What it is: LongMemEval-S cleaned variant, having questions with ambiguous
or invalid answer sessions removed. Used by `benchmarks/longmemeval_granularity.py`.
- Relationship to `longmemeval_s_full.json`: A cleaned derivative of
`longmemeval_s_full.json` obtained by filtering out questions with ambiguous
or invalid answer session ids. No verified copy is available in this repository.
- Byte size: not available (no verified copy located).
- Question count: not available (no verified copy located).
- sha256: NOT YET PINNED - no verified copy has been hashed; obtain the file from
the upstream LongMemEval project cleaned set and run
`shasum -a 256 benchmarks/data/longmemeval_s_cleaned.json`
to verify. Previously the script `benchmarks/scripts/clean_longmemeval.py` was
cited as the derivation method, but that script does not exist in this repository.
The byte size and question count previously stated were unverifiable and are
removed.

To verify a copy when available:

```bash
shasum -a 256 benchmarks/data/longmemeval_s_cleaned.json
```

*(Note: this file does not currently exist in this repository. No verified copy
has been located. Obtain from the upstream LongMemEval project cleaned set.)*
4 changes: 4 additions & 0 deletions changelog.d/tsk-mln5lz-longmemeval-readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
### Fixed
- Replaced fabricated byte size for `longmemeval_oracle.json` (was 285212416, measured 15388478) with correct value from verified copies in sibling repos
- Removed unverifiable byte size, question count, and "13 removed" derivation from `longmemeval_s_cleaned.json`, replaced with NOT YET PINNED status
- Corrected existence claims from "does not exist on this machine" to "does not exist in this repository; obtain from upstream"