[2.0.x] UTF-8 language translation support by yhfudev · Pull Request #10213 · MarlinFirmware/Marlin

yhfudev · 2018-03-25T03:01:52Z

Description

This is the UTF-8 language support for 2.0.x branch. It supports UTF-8 language translation header files, with the font data generated by scripts automatically.

It has been tested with following hardware:

HD44780 (Japanese version)
- REPRAP_DISCOUNT_SMART_CONTROLLER
U8G devices
- REPRAP_DISCOUNT_FULL_GRAPHIC_SMART_CONTROLLER

The document(howto) for the new language engine is in the file buildroot/share/fonts/uxggenpages.md.

It would be better to change the suffix of the language file to reflect the language/country.
For example: jp_kana(Japaneses with kana), zh_CN(Simplified Chinese), zh_TW(Traditional Chinese) etc.
Since all of the language files are UTF-8 encoded, so we don't need the utf8 suffixes now.

To regenerate the font data header for languages (from Marlin/src/lcd/language/language_*.h to Marlin/src/lcd/dogm/language_data_*.h), you may need to run the script (as described in the document buildroot/share/fonts/uxggenpages.md):

    # compile the bdf2u8g with patch buildroot/share/fonts/u8glib-bdf2u8g.patch
    # and then
    cd Marlin/
    ../buildroot/share/fonts/genallfont.sh

the following files can be renamed:

    Marlin/src/lcd/language/language_pl-DOGM.h      ==>  language_pl.h
    Marlin/src/lcd/language/language_pt-br_utf8.h   ==>  language_pt-br.h
    Marlin/src/lcd/language/language_kana_utf8.h    ==>  language_jp-kana.h
    Marlin/src/lcd/language/language_fr_utf8.h      ==>  language_fr.h
    Marlin/src/lcd/language/language_es_utf8.h      ==>  language_es.h
    Marlin/src/lcd/language/language_cz_utf8.h      ==>  language_cz.h
    Marlin/src/lcd/language/language_pt_utf8.h      ==>  language_pt.h
    Marlin/src/lcd/language/language_sk_utf8.h      ==>  language_sk.h

The following files can be removed after merging this patch set:

    Marlin/src/lcd/language/language_pl-HD44780.h
    Marlin/src/lcd/language/language_kana.h
    Marlin/src/lcd/language/language_cn.h
    Marlin/src/lcd/utf_mapper.h
    Marlin/src/lcd/dogm/dogm_font_data_HD44780_C.h
    Marlin/src/lcd/dogm/dogm_font_data_HD44780_J.h
    Marlin/src/lcd/dogm/dogm_font_data_HD44780_W.h
    Marlin/src/lcd/dogm/dogm_font_data_ISO10646_1_PL.h
    Marlin/src/lcd/dogm/dogm_font_data_ISO10646_1_tr.h
    Marlin/src/lcd/dogm/dogm_font_data_ISO10646_5_Cyrillic.h
    Marlin/src/lcd/dogm/dogm_font_data_ISO10646_CN.h
    Marlin/src/lcd/dogm/dogm_font_data_ISO10646_CZ.h
    Marlin/src/lcd/dogm/dogm_font_data_ISO10646_SK.h
    Marlin/src/lcd/dogm/dogm_font_data_ISO10646_Greek.h
    Marlin/src/lcd/dogm/dogm_font_data_ISO10646_Kana.h
    Marlin/src/lcd/dogm/dogm_font_data_Marlin_symbols.h

Benefits

The language maintainer can use Unicode to translate their languages and don't need care about the font if the font contains all of the glyph of their languages.
And the code developer won't need to create utf8 wrapper for each of language.

Related Issues

It was with the help of UTF-8 language translation support #5441 by @thinkyhead which is rebuild of UTF-8 language translation supports #4704 from the origin discussion The Language Translations #4539.
For the size of firmware of the new UTF-8 engine, please see the discussions in Language files - new system #5501.

thinkyhead · 2018-04-07T01:26:30Z

@yhfudev — Are you certain that we can remove dogm_font_data_Marlin_symbols.h? It is still included in several places, and I assume the symbols are not available elsewhere.

yhfudev · 2018-04-08T17:23:24Z

@thinkyhead Yes, it can be removed. I put the symbols to the first 9 slots(1-9) in the font file buildroot/share/fonts/ISO10646-0-3.bdf

yhfudev · 2018-04-08T17:29:03Z

The hd44780 driver has its own font data defined in the file ultralcd_impl_HD44780.h, and I tested and it works.

… into devel-2.0-utf8

…are/fonts/ISO10646-0-3.bdf

…fault font data.

thinkyhead · 2018-04-09T02:32:07Z

Because I've been making changes and rebasing this branch, your forced update seems to have trounced my fixes. You should have pulled this afresh from your origin repository before applying more changes, like so:

git checkout devel-2.0-utf8
git fetch origin
git reset --hard origin/devel-2.0-utf8

I will try to patch it back together as best I can.

yhfudev · 2018-04-09T02:50:59Z

@thinkyhead
Yes, the merging code conflict would be very chance to happen after I changed the language name and removed the unused font data files. After I notice you were trying to do the font data file deletion from repo, I do it instead to avoid unawareness errors, since I can test and conform it works with my hardware.

So, what's your plan for the next step?

thinkyhead · 2018-04-09T02:55:02Z

Well, I should have been more communicative and let you know I had squashed and made other changes. I've just rebased both my version and your version to the latest bugfix-2.0.x so that I can compare apples to apples, and all I really need to do is see where I made formatting changes and fix it up. I'll simply push whatever changes I make as an added commit for now, so you can continue to use your current workflow.

thinkyhead · 2018-04-09T03:03:02Z

Ok, all set! Just formatting changes. If you just use "Pull" from Github Desktop we'll be in sync.

Actually, are there any more changes that you want to make? If this passes Travis CI then there's nothing preventing it from being merged. Marlin 1.1.9 is on the verge of being released, so we'll all be focusing on 2.0.x from now on.

yhfudev · 2018-04-09T03:09:33Z

That's great news you decide to merge this branch!

I'm done with the changes. I tested both of the dogm and hd44780 after removing unused files and it now works.
Some language can also shows in hd44780 with the help of remapping the char to similar glyph.

I'll also test your update with my changes later.

thinkyhead · 2018-04-09T07:04:36Z

Great! I'll only be holding this till 1.1.9 release. I was hoping for yesterday… and then today… but now maybe tomorrow due to some very minor issues that need to be checked out.

thinkyhead · 2018-04-13T01:15:02Z

Merged so that language updates are applied correctly.

Marlin 1.1.9 will be released very soon. No need to address every last issue, which can always be fixed in a quick-patch later as 1.1.10.

rmoravcik · 2018-04-13T11:02:09Z

Hi @yhfudev, I just tried your modification and some characters with accents looks different than the one without accents. It seems to me they we created using a different font. For example character ť looks different then t. č,ž is bigger than c,z and so on...

Here is an picture of characters used by slovak language displayed in bugfix-1.1.x (left side) and in bugfix-2.0.x (right side).

I used these commands to display the characters:

M117 aäAÄaáAÁeéEÉiíIÍlĺLĹ
M117 oóOÓoôOÔrŕRŔuúUÚyýYÝ
M117 cčCČdďDĎlľLĽnňNŇsšSŠ
M117 tťTŤzžZŽ

yhfudev · 2018-04-13T17:58:20Z

Hi, @rmoravcik,

The fonts were changed in the latest PR.
It's now using the font file buildroot/share/fonts/ISO10646-0-3.bdf.
Slovak (sk) was added recently, and it had its own font data when added.

I'm not speak Slovak so I want to know which one is preferred? Thanks!

Fix #10395 Followup to #10213

yhfudev · 2018-04-15T00:01:47Z

Hi, @rmoravcik
The current font file seems mixed with three different size fonts, can you please renew the missing glyph of SK in the file ISO10646-0-2.bdf updated by the PR #10417? So I can merge it to ISO10646-0-3.bdf for generating language data.
Thanks!

lehuuloc-0104 · 2019-08-23T15:52:00Z

Description

This is the UTF-8 language support for 2.0.x branch. It supports UTF-8 language translation header files, with the font data generated by scripts automatically.

It has been tested with following hardware:

HD44780 (Japanese version)

REPRAP_DISCOUNT_SMART_CONTROLLER

U8G devices

REPRAP_DISCOUNT_FULL_GRAPHIC_SMART_CONTROLLER

The document(howto) for the new language engine is in the file buildroot/share/fonts/uxggenpages.md.

It would be better to change the suffix of the language file to reflect the language/country.
For example: jp_kana(Japaneses with kana), zh_CN(Simplified Chinese), zh_TW(Traditional Chinese) etc.
Since all of the language files are UTF-8 encoded, so we don't need the utf8 suffixes now.

To regenerate the font data header for languages (from Marlin/src/lcd/language/language_*.h to Marlin/src/lcd/dogm/language_data_*.h), you may need to run the script (as described in the document buildroot/share/fonts/uxggenpages.md):
    # compile the bdf2u8g with patch buildroot/share/fonts/u8glib-bdf2u8g.patch
    # and then
    cd Marlin/
    ../buildroot/share/fonts/genallfont.sh
the following files can be renamed:
    Marlin/src/lcd/language/language_pl-DOGM.h      ==>  language_pl.h
    Marlin/src/lcd/language/language_pt-br_utf8.h   ==>  language_pt-br.h
    Marlin/src/lcd/language/language_kana_utf8.h    ==>  language_jp-kana.h
    Marlin/src/lcd/language/language_fr_utf8.h      ==>  language_fr.h
    Marlin/src/lcd/language/language_es_utf8.h      ==>  language_es.h
    Marlin/src/lcd/language/language_cz_utf8.h      ==>  language_cz.h
    Marlin/src/lcd/language/language_pt_utf8.h      ==>  language_pt.h
    Marlin/src/lcd/language/language_sk_utf8.h      ==>  language_sk.h
The following files can be removed after merging this patch set:
    Marlin/src/lcd/language/language_pl-HD44780.h
    Marlin/src/lcd/language/language_kana.h
    Marlin/src/lcd/language/language_cn.h
    Marlin/src/lcd/utf_mapper.h
    Marlin/src/lcd/dogm/dogm_font_data_HD44780_C.h
    Marlin/src/lcd/dogm/dogm_font_data_HD44780_J.h
    Marlin/src/lcd/dogm/dogm_font_data_HD44780_W.h
    Marlin/src/lcd/dogm/dogm_font_data_ISO10646_1_PL.h
    Marlin/src/lcd/dogm/dogm_font_data_ISO10646_1_tr.h
    Marlin/src/lcd/dogm/dogm_font_data_ISO10646_5_Cyrillic.h
    Marlin/src/lcd/dogm/dogm_font_data_ISO10646_CN.h
    Marlin/src/lcd/dogm/dogm_font_data_ISO10646_CZ.h
    Marlin/src/lcd/dogm/dogm_font_data_ISO10646_SK.h
    Marlin/src/lcd/dogm/dogm_font_data_ISO10646_Greek.h
    Marlin/src/lcd/dogm/dogm_font_data_ISO10646_Kana.h
    Marlin/src/lcd/dogm/dogm_font_data_Marlin_symbols.h
Benefits

The language maintainer can use Unicode to translate their languages and don't need care about the font if the font contains all of the glyph of their languages.

And the code developer won't need to create utf8 wrapper for each of language.

Related Issues

It was with the help of UTF-8 language translation support #5441 by @thinkyhead which is rebuild of UTF-8 language translation supports #4704 from the origin discussion The Language Translations #4539.

For the size of firmware of the new UTF-8 engine, please see the discussions in Language files - new system #5501.

Hi @yhfudev ,
I had the test.bdf font file I needed, I wanted to display the glyphs for my language.
Ex: glyph is in position 7861
I need to run the file [genallfont.sh] to compile data font, but I don't know how to run it, though try many ways like installing ubuntu [bash] on win10 but still can't run the file [genallfont.sh]
Can you help me to run it?

yhfudev · 2019-08-23T23:11:33Z

You need to compile the bdf2u8g binary first(using get-bdf2u8g.sh), and then run the script genallfont.sh.

You may read the file Marlin/buildroot/share/fonts/uxggenpages.md as reference.

It would be better if you can post the output of error messages so I can told you what need to do next.

yhfudev added 11 commits March 24, 2018 20:46

Font and tool additions for UTF-8 language

b6e6b31

lib additions for UTF-8 language

254322c

Language and language_data files for UTF-8

3ad764d

LCD code updates for UTF-8 system

9a283e9

Build and CI support for UTF-8 language

91f4bb0

Update the Russian language font data for the recent changes

066d5fe

Add missing language font data

07fd983

Rremove compile warnings

3bd031e

Update the travis CI scripts

1287e94

share LCD_CLASS definition for HD44780 devices

42f6f01

use default memcpy_P() in case of no such function

5a5b05a

yhfudev mentioned this pull request Mar 26, 2018

[2.0.x] Support utf8 unicode for test language #10220

Merged

thinkyhead force-pushed the bugfix-2.0.x branch 2 times, most recently from 9f46c52 to 889fd5f Compare March 31, 2018 23:59

thinkyhead added PR: Improvement C: Language labels Apr 3, 2018

thinkyhead force-pushed the bugfix-2.0.x branch 3 times, most recently from 68147f7 to e8e6026 Compare April 7, 2018 00:48

thinkyhead mentioned this pull request Apr 8, 2018

[RFC, 2.0.x] Proposed Menu Reorganization #10317

Closed

yhfudev added 7 commits April 8, 2018 15:04

Merge branch 'bugfix-2.0.x' of https://github.com/MarlinFirmware/Marlin…

10b03a0

… into devel-2.0-utf8

convert the font data for u8g including the first 9 Marlin glyph.

df4ba55

remove unused fonts which was merged in the default font buildroot/sh…

d5caaa3

…are/fonts/ISO10646-0-3.bdf

remove file dogm_font_data_Marlin_symbols.h, since it is merged in de…

226f25c

…fault font data.

fix hd44780 sorted search table

63de445

remove unused language files; change language suffix;

041dafb

update example configure files for language names

62ac75d

yhfudev added 2 commits April 8, 2018 21:55

add char sim for hd44780.

4dbef4e

update language data for recent updates.

9433038

update hd44780 search table

7125d70

Tweaky spacing changes

f006273

fix hd44780 search table

41f6866

thinkyhead merged commit c96412a into MarlinFirmware:bugfix-2.0.x Apr 13, 2018

thinkyhead mentioned this pull request Apr 14, 2018

2.0.x - 13 April - u8g.h bug (again) #10395

Closed

thinkyhead added a commit that referenced this pull request Apr 14, 2018

u8g.h => U8glib.h

e0cb8ff

Fix #10395 Followup to #10213

yhfudev deleted the devel-2.0-utf8 branch April 14, 2018 22:56

yhfudev mentioned this pull request Apr 14, 2018

[2.0.x] Update fonts data and scripts #10417

Merged

yhfudev mentioned this pull request Apr 15, 2018

[2.0.x] Fix u8g path #10419

Merged

Uh oh!

Conversation

yhfudev commented Mar 25, 2018 • edited by thinkyhead Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Benefits

Related Issues

Uh oh!

thinkyhead commented Apr 7, 2018

Uh oh!

yhfudev commented Apr 8, 2018

Uh oh!

yhfudev commented Apr 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thinkyhead commented Apr 9, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yhfudev commented Apr 9, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thinkyhead commented Apr 9, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thinkyhead commented Apr 9, 2018

Uh oh!

yhfudev commented Apr 9, 2018

Uh oh!

thinkyhead commented Apr 9, 2018

Uh oh!

thinkyhead commented Apr 13, 2018

Uh oh!

rmoravcik commented Apr 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yhfudev commented Apr 13, 2018 • edited by thinkyhead Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yhfudev commented Apr 15, 2018

Uh oh!

lehuuloc-0104 commented Aug 23, 2019

Description

Benefits

Related Issues

Uh oh!

yhfudev commented Aug 23, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yhfudev commented Mar 25, 2018 •

edited by thinkyhead

Loading

yhfudev commented Apr 8, 2018 •

edited

Loading

thinkyhead commented Apr 9, 2018 •

edited

Loading

yhfudev commented Apr 9, 2018 •

edited

Loading

thinkyhead commented Apr 9, 2018 •

edited

Loading

rmoravcik commented Apr 13, 2018 •

edited

Loading

yhfudev commented Apr 13, 2018 •

edited by thinkyhead

Loading