Publish evaluation metrics

[vrigal]

Based on #589 (required for valid evaluation task naming).

Closes #519

Based on Bastien's work (#558)

[skip ci]

[vrigal]

I tried testing by setting {"training_config": { "wandb-publication": True }} in the training configuration (taskcluster.translations_taskgraph.parameters) and change parameters to avoid caching, all in a temporary commit (TRASHME).

However no train nor evaluation task is started in Taskcluster (marked unscheduled). @eu9ene do you have right to start this task for example https://firefox-ci-tc.services.mozilla.com/tasks/ce0zK1kgSxC_cwZesDXPLQ ? (if it works it should publish that eval metric to a new group ci_b_lxggt6TXCnpi1qsjeIUQ on moz-translations workspace project ru-en).

[eu9ene]

I tried testing by setting {"training_config": { "wandb-publication": True }} in the training configuration (taskcluster.translations_taskgraph.parameters) and change parameters to avoid caching, all in a temporary commit (TRASHME).

However no train nor evaluation task is started in Taskcluster (marked unscheduled). @eu9ene do you have right to start this task for example https://firefox-ci-tc.services.mozilla.com/tasks/ce0zK1kgSxC_cwZesDXPLQ ? (if it works it should publish that eval metric to a new group ci_b_lxggt6TXCnpi1qsjeIUQ on moz-translations workspace project ru-en).

The train-backwards task is failed, that's why it doesn't start. It's something about: Publication failed: argument --wandb-project: conflicting option string: --wandb-project

[eu9ene]

Depends on #611 to run CI

[eu9ene]

We should not forget to check that the newly added COMET metric is being published.

[eu9ene]

This will likely work after the bug fix but we should refactor the code reuse without adding arguments to another script. Also please double check that the recently added COMET metric is working. We'll need green CI and everything published on W&B to verify that this works. Thanks for pushing on this one, we really need it for the release!

[vrigal]

Also please double check that the recently added COMET metric is working.

This metric was not specified. I will open a new Issue/PR for this.

It is actually ignored by the online publication. Actual support for .metrics files will not work (from old experiments and offline Taskcluster uploads). It will raise AssertionError: file must contain exactly 2 float values.

[vrigal]

This would also benefit from #610.

[vrigal]

Evaluation data has been published to W&B: https://wandb.ai/moz-translations/ru-en/groups/ci_bVp4cnGJSgCeY2pK5aKK0A/workspace

I found two other things:

• Teacher-1 is not correctly parsed by the regex (label evaluate-teacher-flores-devtest-ru-en-1), because it detects ru-en-1 as part of the dataset (instead of the language at the end of the label). In case that extra language part is always present, it will be a 1 character patch.
• I noticed that simple logs can be displayed as bar charts on W&B, once the grouping option is disabled (see our custom workspace). That would be a great option in order to remove Tables/custom charts for evaluation metrics and homogenize all charts in the future.

⚠️ note: Commit 9d6cfa2 must be dropped before merging

[eu9ene]

@vrigal things don't look right here: https://wandb.ai/moz-translations/ru-en/groups/ci_bVp4cnGJSgCeY2pK5aKK0A/workspace. Evals use different run names than training:

Teacher-1 is not correctly parsed by the regex (label evaluate-teacher-flores-devtest-ru-en-1), because it detects ru-en-1 as part of the dataset (instead of the language at the end of the label). In case that extra language part is always present, it will be a 1 character patch.

Yes, I see, this is a must-fix before merging. Please also add "sacrebleu_aug-upper_wmt19" to https://github.com/mozilla/firefox-translations-training/blob/f5a6af83849f08cce3e7b8b9d06cc2a0456f2c68/taskcluster/translations_taskgraph/parameters.py#L91 so that we can tests that the augmentation part of the dataset is parsed correctly.

[vrigal]

Publication should be fine now (flores & sacrebleu): https://wandb.ai/moz-translations/ru-en/groups/ci_Z45R3e22TWCzyqQ5UFdWdQ/workspace

I updated the regex so it supports labels ending with -1, merged run names in the label parser (backward -> backwards and finetuned -> finetune) and dropped the temporary commit used to trigger the complete CI.

[eu9ene]

The issue in CI is #549