Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: invalid template parameters in translations prevent substitution #46326

Open
5 of 8 tasks
dvtate opened this issue Jul 5, 2024 · 11 comments
Open
5 of 8 tasks

[Bug]: invalid template parameters in translations prevent substitution #46326

dvtate opened this issue Jul 5, 2024 · 11 comments

Comments

@dvtate
Copy link

dvtate commented Jul 5, 2024

⚠️ This issue respects the following points: ⚠️

Bug description

I was uploading some files to my Nextcloud instance which I have the language set to Danish and noticed this message "{sekunder} sekunder tilbage". I assumed the translator mistakenly translated the template parameter causing it to not get substituted.

After seeing this I didn't find it in the any of the l10n/*.json files but did find it in some compiled javascript files in dist/ which seems suspicious:

[server]$ grep -Rl "{sekunder} sekunder tilbage" .
./dist/files-main.js.map
./dist/files-init.js
./dist/files-main.js
./dist/files-init.js.map

Why does this happen??

I also found some problems in the l10n json files. (for example)

So I threw together a script to get all the translations with fields not in the original English version.
image

I think we should include this script (or some better-written version of it) as a CI/unit test before releasing.

check_translations.py
import subprocess
import glob
import os
import json
import re

translation_dirs = (subprocess
	.check_output([ 'find', '.', '-type', 'd', '-name', 'l10n' ])
	.decode('utf-8')
	.split('\n'))

files = [
	# [file path, parsed json]
]

for d in translation_dirs:
	for path in glob.glob(d + "/*.json"):
		if os.stat(path).st_size != 0:
			with open(path) as file:
				try:
					files.append([path, json.load(file)])
				except Exception as e:
					print('json load failed: ', e)


for t in files:
	for t_en in t[1]['translations']:
		t_to = t[1]['translations'][t_en]
		en_fields = re.findall('\\{([^}]+)\\}', t_en)
		if type(t_to) == list: # plurals?
			for tt in t_to:
				for f in re.findall('\\{([^}]+)\\}', tt):
					if f not in en_fields:
						print(t[0] + ': invalid field {' + f + '}')
		else:
			for f in re.findall('\\{([^}]+)\\}', t_to):
				if f not in en_fields:
					print(t[0] + ': invalid field {' + f + '}')

Steps to reproduce

  1. set language to danish
  2. upload a bunch of files on a slow machine in the files app
  3. time remaining translation is wrong (probably others too)

Expected behavior

Translations should probably not translate the template params

Installation method

Other Community project

Nextcloud Server version

29

Operating system

Other

PHP engine version

PHP 8.2

Web server

Nginx

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

  • Default user-backend (database)
  • LDAP/ Active Directory
  • SSO - SAML
  • Other

Configuration report

{
    "system": {
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "logfile": "\/var\/log\/nextcloud\/nextcloud.log",
        "loglevel": 3,
        "apps_paths": [
            {
                "path": "\/usr\/share\/webapps\/nextcloud\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/lib\/nextcloud\/apps",
                "url": "\/wapps",
                "writable": true
            }
        ],
        "trusted_domains": [
            "localhost",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "htaccess.RewriteBase": "\/",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "29.0.3.4",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "enable_previews": true,
        "enabledPreviewProviders": [
            "OC\\Preview\\Movie",
            "OC\\Preview\\PNG",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\GIF",
            "OC\\Preview\\BMP",
            "OC\\Preview\\XBitmap",
            "OC\\Preview\\MP3",
            "OC\\Preview\\MP4",
            "OC\\Preview\\TXT",
            "OC\\Preview\\MKV",
            "OC\\Preview\\AVI",
            "OC\\Preview\\mov",
            "OC\\Preview\\Image"
        ],
        "maintenance": false,
        "theme": "",
        "memories.vod.ffmpeg": "\/usr\/bin\/ffmpeg",
        "memories.vod.ffprobe": "\/usr\/bin\/ffprobe",
        "memories.vod.path": "\/var\/lib\/nextcloud\/apps\/memories\/bin-ext\/go-vod-amd64",
        "memories.vod.disable": false,
        "memories.exiftool_no_local": true,
        "memories.db.triggers.fcu": true
    }
}

List of activated Apps

No response

Nextcloud Signing status

No response

Nextcloud Logs

No response

Additional info

I have installed nextcloud from the Arch Linux repos

@dvtate dvtate added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Jul 5, 2024
@kesselb
Copy link
Contributor

kesselb commented Jul 6, 2024

Good finding, thank you 👍

Maybe that problem could be solved by adding a translation check on transifex: https://help.transifex.com/en/articles/6241794-setting-translation-checks#h_317a8b70f5

@rakekniven
Copy link
Member

@dvtate Can you provide the list of defective placeholders?
I will fix them.
Minutes ago I added translation checks and if it works like intended it should fix the issue.

@rakekniven rakekniven added feature: language l10n and translations and removed bug 0. Needs triage Pending check for reproducibility or if it fits our roadmap labels Jul 6, 2024
@rakekniven rakekniven self-assigned this Jul 6, 2024
@dvtate
Copy link
Author

dvtate commented Jul 8, 2024

I pulled and re-ran the script and here's the list of invalid placeholders: list.txt

I'm not sure how to check the bundles in the dist/ folder but it seems there's there's still some problems there.

I used the attached python script to generate this list translation_check.py.txt

@dvtate
Copy link
Author

dvtate commented Jul 9, 2024

My script might not be checking all the translations

@susnux susnux added the bug label Aug 15, 2024
@szaimen szaimen added 0. Needs triage Pending check for reproducibility or if it fits our roadmap 29-feedback labels Aug 29, 2024
@rakekniven
Copy link
Member

A lot of the strings have been fixed.

Could you re-run your script next week after release of NC30?
After that I will write a message to all translators and ask them to check their lines.

@dvtate
Copy link
Author

dvtate commented Sep 10, 2024

@rakekniven sure, just ping me. I'm happy to contribute where I can. The script is also attached in a previous post.

Would this release make the translations in dist/ synced with the l10n .json files? If so that would probably be very helpful for finding and fixing things.

@dvtate
Copy link
Author

dvtate commented Oct 31, 2024

@rakekniven It seems to be improving but there are still some mistakes. I would recommend making a CI to check this.

[tate@archbox server]$ python translation_check.py 
./apps/comments/l10n/af.json: invalid field {outeur}
./apps/comments/l10n/af.json: invalid field {lêer}
./apps/comments/l10n/af.json: invalid field {outeur}
./apps/comments/l10n/af.json: invalid field {lêer}
./apps/comments/l10n/sq.json: invalid field {autori}
./apps/dav/l10n/el.json: invalid field {ημερολόγιο}
./apps/dav/l10n/es_419.json: invalid field {el calendario calendar}
./apps/dav/l10n/es_AR.json: invalid field {el calendario calendar}
./apps/dav/l10n/es_CL.json: invalid field {el calendario calendar}
./apps/dav/l10n/es_CO.json: invalid field {el calendario calendar}
./apps/dav/l10n/es_CR.json: invalid field {el calendario calendar}
./apps/dav/l10n/es_DO.json: invalid field {el calendario calendar}
./apps/dav/l10n/es_GT.json: invalid field {el calendario calendar}
./apps/dav/l10n/es_HN.json: invalid field {el calendario calendar}
./apps/dav/l10n/es_NI.json: invalid field {el calendario calendar}
./apps/dav/l10n/es_PA.json: invalid field {el calendario calendar}
./apps/dav/l10n/es_PE.json: invalid field {el calendario calendar}
./apps/dav/l10n/es_PR.json: invalid field {el calendario calendar}
./apps/dav/l10n/es_PY.json: invalid field {el calendario calendar}
./apps/dav/l10n/es_SV.json: invalid field {el calendario calendar}
./apps/dav/l10n/es_UY.json: invalid field {el calendario calendar}
./apps/dav/l10n/sq.json: invalid field {aktori}
./apps/dav/l10n/sq.json: invalid field {kalendarin}
./apps/dav/l10n/sq.json: invalid field { todo}
./apps/files/l10n/af.json: invalid field {file}
./apps/files/l10n/ar.json: invalid field {اسم المجلد}
./apps/files/l10n/ar.json: invalid field {اسم الملف}
./apps/files/l10n/fa.json: invalid field {نسبی}
./apps/files/l10n/fa.json: invalid field {پرونده}
./apps/files/l10n/fa.json: invalid field {پرونده}
./apps/files/l10n/ga.json: invalid field {comhad}
./apps/files/l10n/ga.json: invalid field { source}
./apps/files/l10n/ga.json: invalid field { source}
./apps/files/l10n/hu.json: invalid field {extension}
./apps/files/l10n/hu.json: invalid field {fileName}
./apps/files/l10n/id.json: invalid field {user}
./apps/files/l10n/ka_GE.json: invalid field {destionation}
./apps/files/l10n/ka_GE.json: invalid field {destionation}
./apps/files/l10n/mn.json: invalid field {user}
./apps/files/l10n/nl.json: invalid field {bestand}
./apps/files/l10n/nl.json: invalid field {oorsprong}
./apps/files/l10n/nl.json: invalid field {bestemming}
./apps/files/l10n/nl.json: invalid field {andere}
./apps/files/l10n/sc.json: invalid field {folderCount}
./apps/files/l10n/sc.json: invalid field {folderCount}
./apps/files/l10n/sk.json: invalid field {file}
./apps/files/l10n/sr.json: invalid field {fileName}
./apps/files/l10n/sr.json: invalid field {targetName}
./apps/files/l10n/sr.json: invalid field {fileName}
./apps/files/l10n/sv.json: invalid field {fileName}
./apps/files/l10n/th.json: invalid field {dirname}
./apps/files/l10n/vi.json: invalid field { quota }
./apps/files/l10n/vi.json: invalid field {tương đối}
./apps/files/l10n/vi.json: invalid field {tập tin}
./apps/files/l10n/vi.json: invalid field {mục tiêu}
./apps/files/l10n/vi.json: invalid field {mục tiêu}
./apps/files/l10n/vi.json: invalid field {Đã sử dụng}
./apps/files/l10n/zh_CN.json: invalid field {displayName}
./apps/files/l10n/zh_CN.json: invalid field {dirName}
./apps/files/l10n/zh_CN.json: invalid field {fileName}
./apps/files/l10n/zh_CN.json: invalid field {fileName}
./apps/files/l10n/zh_HK.json: invalid field {extension}
./apps/files_external/l10n/ko.json: invalid field {status Message}
./apps/files_reminders/l10n/hu.json: invalid field {name}
./apps/files_sharing/l10n/az.json: invalid field {uzaq}
./apps/files_sharing/l10n/eo.json: invalid field {group}
./apps/files_sharing/l10n/eo.json: invalid field {group}
./apps/files_sharing/l10n/fa.json: invalid field {node }
./apps/files_sharing/l10n/fa.json: invalid field {اشتراک}
./apps/files_sharing/l10n/fa.json: invalid field {user }
./apps/files_sharing/l10n/fa.json: invalid field {اشتراک}
./apps/files_sharing/l10n/fa.json: invalid field {گروه}
./apps/files_sharing/l10n/fa.json: invalid field {user }
./apps/files_sharing/l10n/fa.json: invalid field {کاربر}
./apps/files_sharing/l10n/fa.json: invalid field {صاحب}
./apps/files_sharing/l10n/fa.json: invalid field {کاربر}
./apps/files_sharing/l10n/fa.json: invalid field {صاحب}
./apps/files_sharing/l10n/fa.json: invalid field {صاحب}
./apps/files_sharing/l10n/fa.json: invalid field {ابتکار}
./apps/files_sharing/l10n/fa.json: invalid field {ابتکار}
./apps/files_sharing/l10n/fa.json: invalid field {سرور}
./apps/files_sharing/l10n/fa.json: invalid field {زمان نسبی }
./apps/files_sharing/l10n/fa.json: invalid field { دارنده}
./apps/files_sharing/l10n/fa.json: invalid field {گروه}
./apps/files_sharing/l10n/fa.json: invalid field {دارنده}
./apps/files_sharing/l10n/fa.json: invalid field {حلقه}
./apps/files_sharing/l10n/fa.json: invalid field {مالک }
./apps/files_sharing/l10n/fa.json: invalid field {مکالمه}
./apps/files_sharing/l10n/fa.json: invalid field {صاحب }
./apps/files_sharing/l10n/fa.json: invalid field {صاحب}
./apps/files_sharing/l10n/fa.json: invalid field {از راه دور}
./apps/files_sharing/l10n/ga.json: invalid field {aisteoir}
./apps/files_sharing/l10n/ga.json: invalid field {aisteoir}
./apps/files_sharing/l10n/ga.json: invalid field {aisteoir}
./apps/files_sharing/l10n/ga.json: invalid field { fillteán}
./apps/files_sharing/l10n/ga.json: invalid field { index}
./apps/files_sharing/l10n/nb.json: invalid field {indeks}
./apps/files_sharing/l10n/ru.json: invalid field {group}
./apps/files_sharing/l10n/sl.json: invalid field {user}
./apps/files_sharing/l10n/sq.json: invalid field { }
./apps/files_sharing/l10n/sq.json: invalid field {aktori}
./apps/files_sharing/l10n/sq.json: invalid field {përdoruesin}
./apps/files_sharing/l10n/sq.json: invalid field {aktori}
./apps/files_sharing/l10n/sq.json: invalid field {skedarë}
./apps/files_sharing/l10n/sq.json: invalid field {përdoruesin}
./apps/files_sharing/l10n/sq.json: invalid field {skedarë}
./apps/files_sharing/l10n/sv.json: invalid field {labe}
./apps/files_sharing/l10n/vi.json: invalid field {tệp}
./apps/files_sharing/l10n/vi.json: invalid field {người khởi xướng}
./apps/files_sharing/l10n/vi.json: invalid field {thư mục}
./apps/files_sharing/l10n/vi.json: invalid field {máy chủ}
./apps/files_versions/l10n/es_AR.json: invalid field {archivo}
./apps/files_versions/l10n/ga.json: invalid field { version.label}
./apps/files_versions/l10n/sc.json: invalid field {documentu}
./apps/settings/l10n/ar.json: invalid field {توصيات جمعية الـ w3c}
./apps/settings/l10n/br.json: invalid field {actor] En deus ouzhpennet {user}
./apps/settings/l10n/br.json: invalid field {tiken}
./apps/settings/l10n/da.json: invalid field {licens}
./apps/settings/l10n/el.json: invalid field {icenseopen}
./apps/settings/l10n/fa.json: invalid field {time}
./apps/settings/l10n/ga.json: invalid field {nasc}
./apps/settings/l10n/ga.json: invalid field {tús}
./apps/settings/l10n/ga.json: invalid field {deireadh}
./apps/settings/l10n/ga.json: invalid field {w3c-moladh}
./apps/settings/l10n/ga.json: invalid field {scór}
./apps/settings/l10n/ga.json: invalid field {airí}
./apps/settings/l10n/ga.json: invalid field {maoin}
./apps/settings/l10n/ga.json: invalid field {méid}
./apps/settings/l10n/ga.json: invalid field {scór}
./apps/settings/l10n/gl.json: invalid field {actor}
./apps/settings/l10n/he.json: invalid field {version}
./apps/settings/l10n/hr.json: invalid field {grupa}
./apps/settings/l10n/hr.json: invalid field {Licenseopen}
./apps/settings/l10n/hr.json: invalid field {autor}
./apps/settings/l10n/hr.json: invalid field {licenca}
./apps/settings/l10n/id.json: invalid field {commmunityopen}
./apps/settings/l10n/ko.json: invalid field {linkstart}
./apps/settings/l10n/ko.json: invalid field {linkend}
./apps/settings/l10n/nb.json: invalid field {tid}
./apps/settings/l10n/nb.json: invalid field {indeks}
./apps/settings/l10n/sk.json: invalid field {actor}
./apps/settings/l10n/sq.json: invalid field {aktori}
./apps/settings/l10n/sq.json: invalid field {madhësia}
./apps/settings/l10n/tr.json: invalid field {w3c-önerisi}
./apps/settings/l10n/uk.json: invalid field {linkstart}
./apps/settings/l10n/uk.json: invalid field {linkend}
./apps/settings/l10n/vi.json: invalid field {tác giả}
./apps/settings/l10n/vi.json: invalid field {giấy phép}
./apps/sharebymail/l10n/bg.json: invalid field {файл}
./apps/sharebymail/l10n/bg.json: invalid field {имейл}
./apps/sharebymail/l10n/el.json: invalid field {αρχείο}
./apps/sharebymail/l10n/nb.json: invalid field {fil}
./apps/sharebymail/l10n/sq.json: invalid field {skedarin}
./apps/sharebymail/l10n/sq.json: invalid field {aktori}
./apps/sharebymail/l10n/sq.json: invalid field {skedarin}
./apps/theming/l10n/es_AR.json: invalid field {pautas}
./apps/theming/l10n/et_EE.json: invalid field {guidlines}
./apps/theming/l10n/lv.json: invalid field {vadlīnijām}
./apps/theming/l10n/lv.json: invalid field {piesaistīts}
./apps/theming/l10n/sq.json: invalid field {udhëzimet}
./apps/theming/l10n/zh_HK.json: invalid field {linked}
./apps/twofactor_backupcodes/l10n/fa.json: invalid field {نام}
./apps/twofactor_backupcodes/l10n/sk.json: invalid field {použité}
./apps/twofactor_backupcodes/l10n/sk.json: invalid field {celkom}
./apps/updatenotification/l10n/ga.json: invalid field {leagan}
./apps/updatenotification/l10n/it.json: invalid field {versione}
./apps/updatenotification/l10n/lb.json: invalid field {Versioun}
./apps/user_ldap/l10n/ga.json: invalid field {nthFreastalaí}
./apps/user_status/l10n/es_EC.json: invalid field {icono}
./apps/user_status/l10n/es_EC.json: invalid field {mensaje}
./apps/user_status/l10n/es_EC.json: invalid field {mensaje}
./apps/user_status/l10n/es_EC.json: invalid field {icono}
./apps/weather_status/l10n/it.json: invalid field {unità}
./apps/weather_status/l10n/nb.json: invalid field {temperatur}
./apps/weather_status/l10n/nb.json: invalid field {temperatur}
./apps/weather_status/l10n/nb.json: invalid field {temperatur}
./apps/weather_status/l10n/nb.json: invalid field {temperatur}
./apps/workflowengine/l10n/mn.json: invalid field {байрлал}
./core/l10n/eu.json: invalid field {bertsioa}
./core/l10n/lo.json: invalid field {ຊື່}
./core/l10n/lo.json: invalid field {ຊື່}
./core/l10n/lo.json: invalid field {ປະເພດ}
./core/l10n/lo.json: invalid field {ປະເພດ}
./core/l10n/vi.json: invalid field {tập tin}
./core/l10n/vi.json: invalid field {mục tiêu}
./core/l10n/vi.json: invalid field {mục tiêu}
./core/l10n/zh_CN.json: invalid field {类型}

@dvtate
Copy link
Author

dvtate commented Oct 31, 2024

I'm also still seeing the bug that originally made me post this in nextcloud 30.
Image
Image

@rakekniven
Copy link
Member

rakekniven commented Oct 31, 2024

It was me fixing the bug one by one.
There is no other way.
Many languages are affected and a lot of the strings are marked as reviewed. So notifying translators does not help and the manually notification costs a lot of time.

No CI check needed for the future. I adapted some settings on Transifex to protect such JavaScript placeholders.

Will fix the others as soon as I have some time and mood.

@dvtate
Copy link
Author

dvtate commented Oct 31, 2024

Sounds like a good plan. Most of the failures are pretty inconsequential so not a huge deal. I just wanted to let you know. Thanks!

My output is different to yours. Are you using masterbranch to run your script?

@joshtrichards joshtrichards added 2. developing Work in progress and removed 0. Needs triage Pending check for reproducibility or if it fits our roadmap labels Nov 5, 2024
@rakekniven
Copy link
Member

rakekniven commented Nov 11, 2024

I'm also still seeing the bug that originally made me post this in nextcloud 30. Image Image

Hello @dvtate ,

just checked your specific case and unfortunately the cause is the same as for french.
See https://help.nextcloud.com/t/telecharger-vs-televerser/208750/11 for details.

For reference:
#49211

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants