Handle none output in wmtp24++

[Froxyy-dev]

Summary by CodeRabbit

Bug Fixes

• Fixed handling of missing translation data in metrics evaluation to prevent processing errors.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds a None-check in the translation metrics evaluation module. If the generation value is None during update, it is replaced with an empty string before being appended to the predictions list.

Changes

Cohort / File(s)	Change Summary
None-check for generation value nemo_skills/evaluation/metrics/translation_metrics.py	Added defensive None-check to replace None generation values with empty strings before appending to predictions list

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly relates to the main change: adding a None-check for generation output in the wmtp24++ metric, replacing None with an empty string.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch mwiniarek/wmtp24pp-handle-none

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

nemo_skills/evaluation/metrics/translation_metrics.py (1)

61-71: None-handling is correct; just confirm empty-string vs skipping behavior

Using an empty string when generation is None is a reasonable way to keep preds/gts aligned and avoid sacrebleu type errors; it will treat “no output” as a worst-case hypothesis, which is often what you want for metrics. If that’s the intended semantics, this looks good as-is.

If instead you’d prefer to ignore examples with missing generations (so they don’t affect BLEU at all), an alternative would be:

 for pred in predictions:
     src_lang = pred["source_language"]
     tgt_lang = pred["target_language"]
     generation = pred["generation"]
     ground_truth = pred["translation"]

-    if generation is None:
-        generation = ""
-
-    self.translation_dict[f"{src_lang}->{tgt_lang}"]["preds"].append(generation)
-    self.translation_dict[f"{src_lang}->{tgt_lang}"]["gts"].append(ground_truth)
+    if generation is not None:
+        self.translation_dict[f"{src_lang}->{tgt_lang}"]["preds"].append(generation)
+        self.translation_dict[f"{src_lang}->{tgt_lang}"]["gts"].append(ground_truth)

From a correctness standpoint, your current change is fine; it just encodes a particular policy about how missing outputs should influence metrics.

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 774cef6 and fb94624.

📒 Files selected for processing (1)

• nemo_skills/evaluation/metrics/translation_metrics.py (1 hunks)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)

• GitHub Check: pre-commit
• GitHub Check: unit-tests

[gwarmstrong]

Hi @Froxyy-dev looks good to me! Please update with signoff--you can see the DCO guidance on how to do this: https://github.com/NVIDIA-NeMo/Skills/pull/1091/checks?check_run_id=57708510000

[Froxyy-dev]

Hi @gwarmstrong, I've signed the DCO!