Translation overhaul

[Mark-Giesen]

I'm thinking about a major overhaul of the multilanguage implementation of Hedy. A little research let me to a few promising ideas. First there is gettext (wikipage). Old and tested and integrated in Python. Using this will standardize translations and therefore open the option of using other tools like Weblate. The latter might be a great tool for our translators, see this instruction video.

Before you get as excited as me, there are some gotchas. Weblate is not guaranteed to work on windows dev-machines (I use windows), but they welcome fixes :-). Some tedious work will come from this, because all strings will have to be changed in the code.

The best part though is that the translation process will be completely decoupled from development. Strings will be distilled from code and added into the translation pipeline and the translations will be put back into the product every time they're updated.

All comments, ideas, problems etc. are very welcome. Is this a path to pursue?

[Felienne]

I'm thinking about a major overhaul of the multilanguage implementation of Hedy. A little research let me to a few promising ideas. First there is gettext (wikipage). Old and tested and integrated in Python. Using this will standardize translations and therefore open the option of using other tools like Weblate. The latter might be a great tool for our translators, see this instruction video.

Weblate looks awesome and like somethiing we totally want. I have no knowledge of any of this, so for me it is not entirely cler what the relationship between gettext and weblate is?

The best part though is that the translation process will be completely decoupled from development. Strings will be distilled from code and added into the translation pipeline and the translations will be put back into the product every time they're updated.

Yes, that would be awesome!

All comments, ideas, problems etc. are very welcome. Is this a path to pursue?

I think this is a great path, but... there it is also a huge huge overhaul. So the big question is what small things can we try now to convince ourselves that this is the best way forward, to build our confidence that yes this is the way to go? And... is it feasible with you leading this while also being on a Windows machine?

[Mark-Giesen]

If you want continuous translation (and you want that) you need to separate development from translation. Development is done in ONE language and translatable strings have to be recognizable within the code. I think there are two ways:

1. you load a file with strings from the right language up front and use those through something like a dictionary with one language being the key, the other the value.
2. you capture all translatable strings in a function (say _()) and have that do the translation at runtime.

Hedy does some of the translation with the first method and some with a variant of the second (if text is level dependent). The toughest part (I think) is to get these two methods merged into one.

If you use method 1, files with strings usually have two languages in them (key and value). This makes it hard to keep all languages in sync.
If you use method 2, you can imagine that a tool could find all translatable strings in code and put them aside for translation for every language. This is what gettext does. It puts it in special files and keeps track of changes and additions. These files are what many Translation Management Service (TMS) like Weblate use to support the translators. They show the translators what's new and what changed. Translators using Weblate "ask" for the newest strings and they are picked from GitHub (or others) and after translation they push it back (or do a pullrequest).

So, I'm proposing a clear cut between development and translation using a tool (i.e. gettext) to extract strings from code on every build of the product. Anytime a translator wants to do some work, he/she gets the latest strings (using Weblate) with the latest translations and updates it as far as he/she likes and pushes it back.
gettext is old, tested and has a lot of users, but it's not the only way. Serge is another option. Some TMS can read yaml if it adheres to a strict structure (regular expression), so maybe the existing structure is usable.

[Mark-Giesen]

Steps forward are probably:

1. find out if we can merge the different methods within Hedy (I believe there's a third one in the UI) into one.
2. Decide if it's a problem if the developer of this part has to be working on a *nix machine. I use Windows and it has apparently full *nix support these days, so only I'm the problem. I could even buy an Apple.
3. After these 2 steps, we narrowed down the options (or not) and decide the way forward.

[Felienne]

Hedy does some of the translation with the first method and some with a variant of the second (if text is level dependent). The toughest part (I think) is to get these two methods merged into one.

I agree that this is the toughest part, so maybe we should take a stab at that first? Because it will make it easier to do further changes but also has value in itself, so yeah let's do your step forward number 1. "find out if we can merge the different methods within Hedy into one"

[comradekingu]

@Mark-Giesen
Happy to help. Not sure if you intend to host Weblate on Windows.
That part didn't make any sense, because it is just a web interface from the user point of view.

[Mark-Giesen]

@comradekingu, thanks for the offer, are you acquainted with Weblate?
For the user it doesn't matter indeed, but for the developers working on this part it might. The backend is python and not tested on windows, but they say "it might work". Just installing it on a linux server or having it hosted by Webslate are valid options. The only problem might be that I'm using windows and might want to tweak it, or run it locally.

[Mark-Giesen]

I have another option under investigation though: Serge.io. This should work in combination with Weblate, but it is well tested in combination with Zing, both are developed en used by Evernote. Anyone with expertise in any of these tools is invited to share there thought and knowledge.