Support serving of backend translations #12453

emlove · 2018-02-16T00:38:30Z

Description

This PR introduces the notion of backend-defined translations. When platforms are loaded, they will be able to register additional translation strings that can be served to clients. Fixes home-assistant/architecture#9

Still quite a ways from MVP, but I wanted to get some better visibility into our path forward and potentially some early feedback.

Source files

We'll manage our source files with git. Sources will be checked in adjacent to the component file. If the component is the main file for a Python package (i.e. __init__.py), the strings will be stored in a file named strings.json. Otherwise, they'll be stored in a file named strings.<platform_name>.json. Examples: (homeassistant/components/zwave/strings.json, homeassistant/components/sensor/strings.season.json)

Lokalise downloads

The Lokalise downloads will also be checked in to the repo, but shouldn't ever be edited except through the translation_download script. They'll be stored in a hidden directory .translations, also adjacent to the component. Similarly to the source files above, they'll be named with the format <language_code>.json, or <component_name>.<language_code>.json. Examples: (homeassistant/components/zwave/.translations/en.json, homeassistant/components/sensor/.translations/season.de.json)

Frontend PR: home-assistant/frontend#894

Docs PR: home-assistant/home-assistant.io#4787

Lokalise temporary project: https://lokalise.co/project/104514435a91b786c77a78.62627628/

TODO:

Register platform translations with the frontend component when platforms are first setup
Determine how translation sources will be checked in to the repo. (Equivalent of src/translations/en.json on the frontend.)
Determine strategy to synchronize / distribute full translation sets from Lokalize through to the PyPI package.
Tests
Create actual Lokalise project before merging
Add balloobbot keys for Lokalise in backend Travis CI job

Future PR:

EDIT: Migrated to home-assistant/frontend#498 for tracking

~~Developer docs~~
~~Allow clients to fetch translations for a single component~~
~~Notify clients when component with translations is set up~~
~~Determine backend fetch caching and strategy (Possibly include etags with the response)~~

houndci-bot · 2018-02-16T00:38:38Z

homeassistant/components/frontend/__init__.py

@@ -19,7 +19,8 @@
 from homeassistant.components.http import HomeAssistantView
 from homeassistant.components.http.auth import is_trusted_ip
 from homeassistant.config import find_config_file, load_yaml_config_file
-from homeassistant.const import CONF_NAME, EVENT_THEMES_UPDATED
+from homeassistant.const import (


'homeassistant.const.EVENT_TRANSLATIONS_UPDATED' imported but unused

balloob · 2018-02-17T07:10:14Z

homeassistant/components/frontend/__init__.py

+        """Return themes."""
+        hass = request.app['hass']
+
+        resources = hass.data[DATA_TRANSLATIONS].get(language)


I would prefer if we structure this like we just restructured service descriptions: on demand.

So have this view look at which components are loaded (hass.config.components), look at their file names and see if they have files. For example, the light folder would look like:

__init__.py

services.yaml

strings.json

strings.nl.json

Bonus points if the frontend would be able to fetch the strings on a per component basis. So if it sees a string for component.light.setup.bla, it will only fetch the light strings from the backend.

houndci-bot · 2018-02-17T22:41:23Z

homeassistant/helpers/translation.py

+    output = {}
+    for key, value in data.items():
+        if isinstance(value, dict):
+            output.update(recursive_flatten('{}{}.'.format(prefix, key), value))


line too long (80 > 79 characters)

houndci-bot · 2018-02-17T22:41:23Z

homeassistant/helpers/translation.py

+    output = {}
+    for key, value in data.items():
+        if isinstance(value, dict):
+            output.update(recursive_flatten('{}{}.'.format(prefix, key), value))


line too long (80 > 79 characters)

andrey-git

We should either have a single file per component or break the world on duplicate keys.

andrey-git · 2018-02-18T06:56:11Z

homeassistant/components/frontend/__init__.py

+
+    @asyncio.coroutine
+    def get(self, request, language):
+        """Return themes."""


andrey-git · 2018-02-18T07:02:03Z

homeassistant/helpers/translation.py

+            platform = component.split('.', 1)[1]
+            filename = 'strings.{}.json'.format(platform)
+        else:
+            filename = 'strings.json'


This won't work for components without their own dir

houndci-bot · 2018-02-24T21:40:32Z

script/translations_upload_merge.py

+
+    os.makedirs("build", exist_ok=True)
+
+    json_util.save_json(os.path.join("build", "translations-upload.json"), translations)


line too long (88 > 79 characters)

houndci-bot · 2018-02-24T21:40:33Z

script/translations_download_split.py

+
+
+
+def main():


too many blank lines (3)

houndci-bot · 2018-02-24T21:40:33Z

script/translations_download_split.py

+#!/usr/bin/env python3
+"""Merge all translation sources into a single JSON file."""
+import glob
+import itertools


'itertools' imported but unused

houndci-bot · 2018-02-24T21:40:33Z

homeassistant/helpers/translation.py

+            name = component
+
+
+        module = get_component(component)


too many blank lines (2)

houndci-bot · 2018-02-24T21:40:33Z

script/translations_upload_merge.py

+
+    os.makedirs("build", exist_ok=True)
+
+    json_util.save_json(os.path.join("build", "translations-upload.json"), translations)


line too long (88 > 79 characters)

houndci-bot · 2018-02-24T21:40:33Z

script/translations_download_split.py

+
+
+
+def main():


too many blank lines (3)

houndci-bot · 2018-02-24T21:40:33Z

script/translations_download_split.py

+#!/usr/bin/env python3
+"""Merge all translation sources into a single JSON file."""
+import glob
+import itertools


'itertools' imported but unused

houndci-bot · 2018-02-24T21:40:34Z

homeassistant/helpers/translation.py

+            name = component
+
+
+        module = get_component(component)


too many blank lines (2)

balloob · 2018-02-26T04:04:40Z

I think that we should only accept the format components/<component name>/strings.json.

Edit: I misread. You used the word component but meant platform. For platforms we should indeed stick to components/<component name>/strings.<platform name>.json

balloob · 2018-02-26T04:05:50Z

(btw no access to temporary lokalise project)

balloob · 2018-02-26T04:09:28Z

homeassistant/helpers/translation.py

+        if domain not in resources:
+            resources[domain] = {}
+
+        # Add the translations for this component to the domain resources.


Something I was thinking about the other day, especially now that we're doing more entity based config: should we store platform as an attribute in the state machine?

This one is going to get massive for sensor component.

The platform as an attribute would certainly be a big help here. It would let us completely avoid this problem and keep the translations namespaced separately. That's something we could add later though if we decide to make it available, as the conflict only exists in the API response. All of our translations at rest are stored separately per platform.

And FWIW this will only return translations for loaded components, so although we'll have lots of translations merged, each instance should only serve the translations it's using.

balloob · 2018-02-26T04:11:06Z

homeassistant/helpers/translation.py

+    components = hass.config.components
+
+    # Load missing files
+    missing = set()


FYI Sets can be subtracted from one another.

missing = hass.config.components - set(translation_cache)

balloob · 2018-02-26T04:13:09Z

homeassistant/helpers/translation.py

+            load_translations_files, missing)
+
+    # Update cache
+    for component in components:


Are you calculating missing again here?

balloob · 2018-02-26T04:13:58Z

homeassistant/helpers/translation.py

+    # Update cache
+    for component in components:
+        if component not in translation_cache:
+            json_file = component_translation_file(component, language)


I wonder if we can send a dictionary mapping component -> file to load_translation_files so we don't have to generate all the keys again.

balloob · 2018-02-26T04:14:47Z

homeassistant/helpers/translation.py

+async def async_get_translations(hass, language):
+    """Return all backend translations."""
+    if language == 'en':
+        resources = await async_get_component_resources(hass, language)


You can move this before the if and turn the else into an if language != 'en'

balloob

It looks great. I realize I was a bit too picky with my comments, you can ignore them.

Get to take advantage of the slick Python 3.5 dict merging here.

emlove · 2018-02-26T23:17:42Z

No problem, it's all great advice per usual.

I thought I made the Lokalise project public, but maybe it's because it's a trial. In any case, there's not much there. I mostly just linked it for an example of how the namespacing will look in Lokalise. I attached a screenshot to the PR description.

I'm going to go ahead and get the docs ready and start setting up the live Lokalise project. It'll probably be a few days before I can get to it.

emlove · 2018-03-01T01:21:10Z

OK, I've got the project spinned up on Lokalise. The last step should be to get the balloobbot credentials copied from the polymer build job to the backend master build job in Travis. After that we can push the button and see if it works.

I've also put up a small PR to update the translation docs to include a link to the new project. home-assistant/home-assistant.io#4787

balloob · 2018-03-01T03:30:09Z

Added LOKALISE_TOKEN env var to the Travis builds of this repo.

balloob · 2018-03-01T03:33:47Z

🤞

emlove requested a review from a team as a code owner February 16, 2018 00:38

homeassistant added integration: frontend core small-pr PRs with less than 30 lines. cla-signed labels Feb 16, 2018

houndci-bot reviewed Feb 16, 2018

View reviewed changes

This was referenced Feb 16, 2018

Load and subscribe to backend translations home-assistant/frontend#894

Merged

RFC: Backend translations home-assistant/architecture#9

Closed

balloob reviewed Feb 17, 2018

View reviewed changes

houndci-bot reviewed Feb 17, 2018

View reviewed changes

emlove force-pushed the backend-translations branch from eb45f41 to 4084719 Compare February 17, 2018 22:45

homeassistant added the cla-signed label Feb 17, 2018

andrey-git reviewed Feb 18, 2018

View reviewed changes

emlove mentioned this pull request Feb 18, 2018

RFC: Adding Default Hass Intents to Components #12087

Closed

houndci-bot reviewed Feb 24, 2018

View reviewed changes

emlove added 7 commits February 24, 2018 16:43

Add view to support backend translation fetching

d01424b

Load backend translations from component json

6db8da6

Translations for season sensor

bb2f328

Scripts to merge and unpack Lokalise translations

e70bafc

Fix copy paste error

4f6529b

Serve post-lokalise translations to frontend

b613332

Linting

83336a4

emlove force-pushed the backend-translations branch from 0caa343 to 83336a4 Compare February 24, 2018 21:46

homeassistant added the cla-signed label Feb 24, 2018

emlove added 4 commits February 24, 2018 16:54

Auto-deploy translations with Travis

cb3bb00

Commit post-lokalise translation files

c8afa71

Split logic into more helper functions

936b36e

Fall back to English for missing keys

dd5b365

emlove added 3 commits February 24, 2018 20:04

Remove unnecessary file check

8895207

Convert translation helper to async/await

5f0ff59

Convert translation helper tests to async/await

9edf9dd

emlove changed the title ~~[WIP] Support serving of backend translations~~ Support serving of backend translations Feb 25, 2018

emlove added the Ready for review label Feb 25, 2018

emlove mentioned this pull request Feb 25, 2018

Translations Roadmap home-assistant/frontend#498

Closed

25 tasks

balloob removed the small-pr PRs with less than 30 lines. label Feb 26, 2018

balloob reviewed Feb 26, 2018

View reviewed changes

balloob approved these changes Feb 26, 2018

View reviewed changes

emlove added 3 commits February 26, 2018 17:50

Use set subtraction to find missing_components

3006593

load_translation_files use component->file mapping

1ae2b3c

Remove duplicated resources fetching

2287676

Get to take advantage of the slick Python 3.5 dict merging here.

homeassistant added the cla-signed label Feb 26, 2018

Switch to live project ID

dcd9f9d

emlove mentioned this pull request Mar 1, 2018

Backend translations home-assistant/home-assistant.io#4787

Merged

2 tasks

balloob merged commit b434ffb into dev Mar 1, 2018

balloob deleted the backend-translations branch March 1, 2018 03:31

balloob mentioned this pull request Mar 9, 2018

0.65 #12995

Merged

OttoWinter mentioned this pull request Mar 10, 2018

Text will not be displayed / translations not loaded on Login and Logout #13035

Closed

home-assistant locked and limited conversation to collaborators Jul 26, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support serving of backend translations #12453

Support serving of backend translations #12453

emlove commented Feb 16, 2018 •

edited

Loading

houndci-bot Feb 16, 2018

balloob Feb 17, 2018

houndci-bot Feb 17, 2018

houndci-bot Feb 17, 2018

andrey-git left a comment

andrey-git Feb 18, 2018

andrey-git Feb 18, 2018

houndci-bot Feb 24, 2018

houndci-bot Feb 24, 2018

houndci-bot Feb 24, 2018

houndci-bot Feb 24, 2018

houndci-bot Feb 24, 2018

houndci-bot Feb 24, 2018

houndci-bot Feb 24, 2018

houndci-bot Feb 24, 2018

balloob commented Feb 26, 2018 •

edited

Loading

balloob commented Feb 26, 2018

balloob Feb 26, 2018

balloob Feb 26, 2018

emlove Feb 26, 2018

emlove Feb 26, 2018

balloob Feb 26, 2018 •

edited

Loading

balloob Feb 26, 2018

balloob Feb 26, 2018

balloob Feb 26, 2018

balloob left a comment

emlove commented Feb 26, 2018

emlove commented Mar 1, 2018

balloob commented Mar 1, 2018

balloob commented Mar 1, 2018


		os.makedirs("build", exist_ok=True)

		json_util.save_json(os.path.join("build", "translations-upload.json"), translations)

Support serving of backend translations #12453

Support serving of backend translations #12453

Conversation

emlove commented Feb 16, 2018 • edited Loading

Description

Source files

Lokalise downloads

TODO:

Future PR:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrey-git left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

balloob commented Feb 26, 2018 • edited Loading

balloob commented Feb 26, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

balloob Feb 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

balloob left a comment

Choose a reason for hiding this comment

emlove commented Feb 26, 2018

emlove commented Mar 1, 2018

balloob commented Mar 1, 2018

balloob commented Mar 1, 2018

emlove commented Feb 16, 2018 •

edited

Loading

balloob commented Feb 26, 2018 •

edited

Loading

balloob Feb 26, 2018 •

edited

Loading