How to use whisper to synchronize existing subtitles?

[skittlesvampir]

Hello everyone,

let's say I already have a subtitle file that has the correct sentences but the timings are way off. Like so bad that none of the other subtitle synchronization softwares can fix it. I'm talking different framerates, commercial blocks, all that.

Now I can use whisper to automatically create subtitles with matching timings, but the quality of the sentences is worse.

What would be a good way to use the timings of one subtitle file and the sentences from another reference subtitle file to create one excellent subtitle file? Whisper sometimes skips background noises or misunderstands whole sentences which makes things a little difficult.

Has somebody maybe implemented a similiar scenario in the past and could give me some hints on how to build this?

[jongwook]

Forced alignment is not supported out of the box but you can refer to this notebook example. This method only works for 30 seconds or shorter, but I can imagine you can write a logic to shift the text and the audio and repeat the same alignment process on a longer audio.

[skittlesvampir]

I don't fully understand everything I see but that seems really helpful. Thank you!
I will mark this question as solved as soon as I managed to understand what the notebook says.

[EtienneAb3d]

@skittlesvampir

What would be a good way to use the timings of one subtitle file and the sentences from another reference subtitle file to create one excellent subtitle file? Whisper sometimes skips background noises or misunderstands whole sentences which makes things a little difficult.
Has somebody maybe implemented a similiar scenario in the past and could give me some hints on how to build this?

What you are looking for is here:
https://github.com/EtienneAb3d/WhisperTimeSync

😉

[Tex2002ans]

let's say I already have a subtitle file that has the correct sentences but the timings are way off. Like so bad that none of the other subtitle synchronization softwares can fix it. I'm talking different framerates, commercial blocks, all that.

Now I can use whisper to automatically create subtitles with matching timings, but the quality of the sentences is worse.

I'm having a little trouble following your explanation, but let me see if I understand by splitting it into different cases.

---

Case A

• Subtitle 1 with good translation, but bad timing.

Case B

• Subtitle 1 with bad translation, but good timing.
• Subtitle 2 with good translation, but bad timing.

Case C

• Subtitle 1 with bad translation, but good timing.
• Subtitle 2 just a good translation, but with zero timings at all.

And you want to align all of these better to the actual video/audio.

---

For Case A

Back in 2019, I had success using "Alass 2":

• https://github.com/kaegi/alass

It would adjust all the timings to match the sound in the video more closely.

And like its Usage states, I was able to run:

alass movie.mp4 incorrect_subtitle.srt output.srt

I successfully used it on multiple harder cases too, where the subtitles were slowly drifting further and further away from the audio... so by the time I was at the end of the show, original subtitles were MANY seconds off.

For Case B

Same, you can just run the GOOD subtitles (Subtitle 2) through the tool to align it better. Hopefully that would squish your issue.

For Case C

Looks like @EtienneAb3d 's link might work.

[EtienneAb3d]

@Tex2002ans
WhisperTimeSync can be used in both cases B and C (if the reference text file is also in SRT format, its timestamps will be removed).

[skittlesvampir]

I've tried alass countless times but it isn't reliable enough for my use case, I would like something that compares subtitles by text.

[EtienneAb3d]

I would like something that compares subtitles by text.

This is what WhisperTimeSync does.

[Tex2002ans]

WhisperTimeSync can be used in both cases B and C (if the reference text file is also in SRT format, its timestamps will be removed).

Great! I'll have to check it out.

Like I said, I haven't really ran across such poor subtitle timing since 2019, so never really ventured too much into other tools. I suspect things probably got better at aligning since 2019 too! :)

[EtienneAb3d]

I suspect things probably got better at aligning since 2019 too! :)

I'm an expert on that alignment subject: this SRT to TXT/SRT aligner is a light open-source version of an industrial aligner I'm working on for more than 25 years. It was proprietary-only in 2019.
Its algorithm has been tested/improved on hundreds of millions of XML/HTML/TXT document pages in all languages, including cross-lingual alignments (not in this light open-source version).
If needed, I know dozens of ways to improve it to this specific SRT alignment question, possibly depending on the kind of text you have to align.
But, really, why didn't you take 5 mn to give it a try? 🤔

[skittlesvampir]

But, really, why didn't you take 5 mn to give it a try? 🤔

Not sure if you mean me or him, but I actually trued out WhusperTimeSync.

The postitive aspects first: On one of my problematic videos, it worked PERFECTLY. I've never encountered such a reliable aligner.

But... On another video, it didn't work as great. For that video, Whisper skipped some sentences altogether, and your software doesn't seem to handle that properly.

I want the exakt subtitles as in the file I pulled from OpenSubtitles.com but adjusting the timestamps with whisper. I would want something like Alass, but it compares the subtitles content.

That being said, I'm currently working on my own solution. (Unless you can help me with my problem, that would be great.)

[EtienneAb3d]

Unless you can help me with my problem, that would be great

You may share your test with me, so that I could give it a try myself.

Skipping several sentences is a real problem. You may also try this, to get a better result from Whisper:
https://github.com/EtienneAb3d/WhisperHallu
😉

[skittlesvampir]

I've recreated the subtitle files using a better model (small.en) using whisper stable-ts and now, the generated file seems to contain all texts.

But after syncing with WhisperTimeSync (lots of words are still misunderstood), the timings don't match anymore. They start with being off-by-one, but get more and more desynchronized over time. Do you have an idea what could cause this?

[EtienneAb3d]

@skittlesvampir
small.en is a low quality model. Perhaps you should at least consider medium.en.
Can you share your data so that I can analyse?

[skittlesvampir]

I've created a seperate issue: EtienneAb3d/WhisperTimeSync#14

I will close this discussion, as WhisperTimeSync would solve my problem.

[EtienneAb3d]

@skittlesvampir
Bug fixed: when the accurate text is also in SRT format, both timestamps were in output
🙃