Add a tool to detect possible unused language keys and untranslated keys

[lunny]

When do some translations, I found some keys have never been used. So that I wrote a tool to help to avoid unused language keys.

• A new tool to detect unused language keys has been introduced. How to use

make i18n-check

the tool will search all .go(except test go files) or .tmpl files to find "$key" to detect whether the language keys are being used. This cannot cover all the situations, i.e. some keys are composited dynamically. So that manually check is necessary. There is a whitelist with glob syntax to manually skip such a situation.

• It also detected the untranslated keys
• The tool also be used in the CI for pull request to check whether unused language keys are missed.
• This PR also fixes all unused and misused language keys.
• This PR also moved backport-locale to tools/i18n/.

[hiifong]

I think you can add this command to the Makefile.

[silverwind]

Can you move the tool to tools directory? It fits better there.

Also maybe add a convenience make target for it.

[wxiaoguang]

1. I believe it can be optimized to finish in a few seconds (at most)
2. It still lacks the ability to check missing keys like Tr("the.key.is.missing.in.locale")
3. We need to declare these non-static keys at the place where they are used, but not make a tool to "know everything"
4. We should never introduce any new top-level single-word keys like Tr("error"), it is unsearchable and make it impossible to maintain.

[silverwind]

We should never introduce any new top-level single-word keys like Tr("error"), it is unsearchable and make it impossible to maintain.

I disagree, sometimes you have simple words to translate, and top-level is ideal for them. Unsearchability is a tooling issue.

[silverwind]

I believe it can be optimized to finish in a few seconds (at most)

Yes, I suppose this is because it serially walks the files while such a operation could and should be parallelized (maybe with limited concurrency to stay within OS open file number limits).

[wxiaoguang]

We should never introduce any new top-level single-word keys like Tr("error"), it is unsearchable and make it impossible to maintain.

I disagree, sometimes you have simple words to translate, and top-level is ideal for them. Unsearchability is a tooling issue.

It's impossible to build a correct tool to resolve the "tooling issue". If you think yes, please show a feasible solution.

[wxiaoguang]

I believe it can be optimized to finish in a few seconds (at most)

Yes, I suppose this is because it serially walks the files while such a operation could and should be parallelized (maybe with limited concurrency to stay within OS open file number limits).

I think it's purely a algorithm problem, no parallelization is needed ......

[silverwind]

I believe it can be optimized to finish in a few seconds (at most)

Yes, I suppose this is because it serially walks the files while such a operation could and should be parallelized (maybe with limited concurrency to stay within OS open file number limits).

I think it's purely a algorithm problem, no parallelization is needed ......

Yeah, I was just refering to filepath.WalkDir which is presumably a serial operation, but likely still fast enough.

[silverwind]

We should never introduce any new top-level single-word keys like Tr("error"), it is unsearchable and make it impossible to maintain.

I disagree, sometimes you have simple words to translate, and top-level is ideal for them. Unsearchability is a tooling issue.

It's impossible to build a correct tool to resolve the "tooling issue". If you think yes, please show a feasible solution.

Could introduce a prefix to all translations like tr-error which makes the strings greppable via tr-error\b for example.

[TheFox0x7]

I skimmed through this and if I'm understanding correctly it's looking through all files for each key.
Why not inverse this and collect used keys first, then compare them with the list?

something similar in shell (fish but I don't think any specific syntax from it is used). bare in mind it probably is missing keys as it's a quick PoC.

rg "TrN? \"([\w._]*)\"" -o --no-filename --no-line-number -r '$1' >keys #find all Tr(N)s from templates
rg "Tr?N\(\"(.*)\"\)" -o --no-filename --no-line-number -r '$1' >> keys # find all Tr(N)s from go files
sort keys | uniq > keys_in_code # sort for comparison
rg "(.*) = .*" options/locale/locale_en-US.ini -N -r '$1' | sort |uniq > keys_in_locale # strip to keys and sort
comm keys_in_locale keys_in_code -32 # list every key in locale which wasn't in code

Disregard - this does not take locale being in ini format so it's wrong.

[wxiaoguang]

I skimmed through this and if I'm understanding correctly it's looking through all files for each key.
Why not inverse this and collect used keys first, then compare them with the list?

The challenge is like this:

key := "k1"
if ... {
    key = "k2"
}
ctx.Locale.Tr(key)

[TheFox0x7]

Ah. True, that complicates things.

[lunny]

Can you move the tool to tools directory? It fits better there.

Also maybe add a convenience make target for it.

Another locale tool, backport-locale, is located under the build directory. Therefore, I think it makes sense to keep them together. How about moving the entire build directory under tools, as tools/build?

[lunny]

OK. I pushed a commit. Now it will take about 15s so that I also introduced it in CI for pull request.

[lunny]

1. I believe it can be optimized to finish in a few seconds (at most)

improve, now it's about 15s.

2. It still lacks the ability to check missing keys like Tr("the.key.is.missing.in.locale")

Yes. This tool just check unused keys but not check missed keys.

3. We need to declare these non-static keys at the place where they are used, but not make a tool to "know everything"
4. We should never introduce any new top-level single-word keys like Tr("error"), it is unsearchable and make it impossible to maintain.

[silverwind]

Still would like the script moved to tools directory. build is for build-related scripts only.

[wxiaoguang]

#34737 (comment)

Still I think it needs to address 2 & 3