Fix missing evaluator for SWE-bench

[ludwig-n]

Fixes an issue introduced by #825 where this line was failing because SweBenchGenerationTask has no evaluator attribute.

Edit: the coderabbit summary is nonsense, I'm just overriding the existing apply_evaluation_hook function to be a no-op for SWE-bench because we don't use an evaluator here (yet).

Summary by CodeRabbit

• Refactor

  • Introduced an asynchronous evaluation hook to the evaluation workflow, enabling a standardized post-generation processing step.
  • Current behavior remains unchanged; data passes through without modification.

• Chores

  • Prepared the evaluation pipeline for future extensibility with a non-disruptive hook point.

[coderabbitai]

Walkthrough

Introduces an asynchronous hook method apply_evaluation_hook(self, data_point) to SweBenchGenerationTask in nemo_skills/inference/eval/swebench.py. The method currently returns the input data_point unchanged, with a comment indicating evaluation is performed directly after generation.

Changes

Cohort / File(s)	Summary
Evaluation Hook Addition nemo_skills/inference/eval/swebench.py	Added async method apply_evaluation_hook(self, data_point) that returns data_point unchanged; comment notes evaluation occurs immediately after generation.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant U as Caller
    participant T as SweBenchGenerationTask
    rect rgb(240,248,255)
    note over T: Generation phase
    U->>T: generate(data_point)
    T-->>U: generated_output
    end
    rect rgb(245,255,240)
    note over T: New hook (post-generation)
    U->>T: apply_evaluation_hook(data_point)
    T-->>U: data_point (unchanged)
    end
    rect rgb(255,250,240)
    note over T: Evaluation phase
    U->>T: evaluate(generated_output, data_point)
    T-->>U: evaluation_result
    end

Loading

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Poem

I twitch my ears at hooks so new,
A nibble of code, no change in view—
Pass-through carrot, crisp and plain,
Post-gen hop, then tests remain.
In fields of eval, I bound with cheer,
Small tweak today—more growth next year! 🥕🐇

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title Check	✅ Passed	The title precisely identifies the main change of restoring the missing evaluator for the SWE-bench task, directly reflecting the pull request’s objective to fix the evaluator attribute error in SweBenchGenerationTask.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

✨ Finishing touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch ludwig-n/swe-bench-fix-evaluator

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Kipok]

@ludwig-n would be great to add slurm tests for swe-bench, so that we can catch issues like this in the future automatically