Exclude start stage tasks from existing tasks

[eu9ene]

This fixes the edge case where we have alignments-original -> alignments-backtranslated both marked as stage: alignments-teacher and want to restart both of them. We should probably split them later to two stages but not now because it breaks the ability to reuse previous graphs.

[bhearsum]

Is this a generally good thing, or could there be groups in previous_group_ids where we'd want to overwrite one task from it but not another? The safer way to do this may be to allow a list of labels, substrings or prefixes to be specified in the training config, and filter anything matching them out of existing_tasks.

(We do something similar for optimizations. We can't directly use that here, but we could use the same technique to keep things flexible.

[eu9ene]

Is this a generally good thing, or could there be groups in previous_group_ids where we'd want to overwrite one task from it but not another? The safer way to do this may be to allow a list of labels, substrings or prefixes to be specified in the training config, and filter anything matching them out of existing_tasks.

(We do something similar for optimizations. We can't directly use that here, but we could use the same technique to keep things flexible.

@bhearsum It probably depends on the situation but usually, I want to restart all the tasks with start_stage. This is basically a hotfix so that I can at least have a way to schedule what I need. So I suggest we merge it for now and do a more comprehensive analysis in #719

[bhearsum]

Is this a generally good thing, or could there be groups in previous_group_ids where we'd want to overwrite one task from it but not another? The safer way to do this may be to allow a list of labels, substrings or prefixes to be specified in the training config, and filter anything matching them out of existing_tasks.
(We do something similar for optimizations. We can't directly use that here, but we could use the same technique to keep things flexible.

@bhearsum It probably depends on the situation but usually, I want to restart all the tasks with start_stage. This is basically a hotfix so that I can at least have a way to schedule what I need. So I suggest we merge it for now and do a more comprehensive analysis in #719

I think it's probably fine on release, seeing as it doesn't necessarily live forever there. I'm more hesitant to take a quick hack on main. Ultimately I consider this up to you and @gregtatum though, so I'll defer to him on this PR.

[gregtatum]

Approving for the release branch. Can you rewrite the merge commit to have a message like "RELEASE ONLY: Exclude start stage tasks from existing tasks" so that we don't accidentally cherry pick it back?

[eu9ene]

I think I'm doing fine with just cherry-picking this into a separate branch so we don't have to merge it at all if you have doubts it's the right solution here