Message extraction

[RicSala]

Is your feature request related to a problem? Please describe.

I love next-intl and it's my go to app for translations, but the lack of a string extraction system is a big disadvantage vs other i18n libraries.

Managing json files manually is a mess: as soon as you start changing the text of your app we...

1. Start getting translations that are not completely sure if are used anymore
2. start doubting whether if every string is within a t method (vs harcoded)

Extensions like i18n Ally do help, but are not a complete solution as other libraries have (see Lingui or Tolgee).

This is, imho, the only thing that keeps next-intl from being the obvious choice for i18n for nextjs apps.

Describe the solution you'd like

Any, as ar as there is a way to make sure everything is translated and every translated string is being used (and it's not just there because it WAS used)

Describe alternatives you've considered

Lingui, Tolgee

[amannn]

Hey, there's been some previous discussion on this here: #456

To some extent, you can already use other tools define and extract messages from a component.

I've recently started wondering if message extraction could be a helpful addition at some point, but I haven't fully made my mind up yet.

Potential benefits:

1. Purge unused messages automatically
2. Easier for AIs to generate
3. No need to rely on a declaration for a JSON file since we can infer values as part of a call site
4. Easier prototyping since there's no need for managing keys
5. Automatic invalidation of target locales when source locale changes

Drawbacks:

1. Tooling like i18n Ally won't work (for editing non-source translations)
2. Relies on a build step
3. A bit less context from the ID for AI-translated messages
4. Translations can be lost if typo fixed in source locale

---

API ideas

Separate namespace

import {useTranslations} from 'next-intl/extracted';

const t = useTranslations();
t('Hello {name}', {name: 'Jane'});

What's not ideal is that we need yet another namespace for ìmport {getTranslations} from 'next-intl/server/extracted'.

Chaining

import {useTranslations} from 'next-intl';

const t = useTranslations().extract();
t('Hello {name}', {name: 'Jane'});

Avoids having to add new namespaces.

A question here is if we should allow useTranslations('Namespace').extract(…). Potentially this could indicate using a custom namespace instead of an autogenerated one.

Separate hook

import {useExtractedTranslations} from 'next-intl';

const t = useExtractedTranslations();
t('Hello {name}', {name: 'Jane'});

Benefit is less overloading, but the downside is more exported functionality.

Key extractor

import {extract, useTranslations} from 'next-intl';

const t = useTranslations();
t(extract`Hello {name}`, {name: 'Jane'})

A bit more verbose, but a potential benefit is that you can extract keys in random places outside of a component (not sure if we want to advocate for this though).

---

Questions:

• Can a key be optionally supplied?
• Can a description be supplied?
• Is there a tradeoff to AI translation quality if a key is automatically generated?
• Which information should be encoded in the generated key? Aspects like the module name (or even file path to be unique) are candidates. But also: Does the human-readable label influence the key name? If so, what happens if the label is slightly changed? Will this create an entirely new key? Or should we encode the AST path? That would in turn create new keys for minor changes in the markup. Seems like there are some tradeoffs here.
• Is changing a label in the source locale in an external system like a TMS a use case? This would lead to a weird situation where the code contains a label that doesn't appear in this form.

---

We can use this issue to collect interest from the community! Some userland experiments like the ones mentioned in #456 are certainly welcome!

[alexop1000]

I'm working on this as a separate tool! Let me know any specific features you are looking for.

[amannn]

That's really cool! Have you thought about the API yet?

I was thinking about something along these lines:

import {useTranslations} from 'next-intl/extracted';

// Or as a separate lib:
// import {useTranslations} from 'next-intl-extracted';

const t = useTranslations('Greeting');
t('hello', 'Hello {firstName}!', {firstName: 'Jane'});

Declaring the default translation inline could be an AI-friendly approach similar to what Tailwind did with CSS (instead of, let's say, moving all translations to a separate object outside of the component).

Going even further, you could even make the key optional:

const t = useTranslations();
t('Hello {firstName}', {firstName: 'Jane'});

It's pretty elegant, but I can see some hurdles:

• Keys need to be generated automatically
• AIs that are used for translating this label into another language have a harder time inferring the context without an explicit key (if we at least use the module name in the generated key, maybe this could help at least a bit)
• Statically analyzing used keys in Client Components for Automatic tree-shaking of messages #1 might have to consider this somehow

Might be helpful to research a bit further into prior art with message extraction …

I haven't thought too much about this yet, just leaving this here for inspiration!

[alexop1000]

Just to be clear, it's not a tool specifically for next-intl. Although, that is the initial focus.
I'm not sure it is close to what you have discussed, but I am pretty confident it will accomplish what the problem is: "to extract messages from the source code and generate the JSON automatically".

It's getting pretty close to release ready (probably 1-2 weeks), I'm just testing it on some larger codebases currently. Will send link here when it's open sourced.

[amannn]

That's really cool! I really haven't made my mind up here, so I'm happy to see any other perspectives.

Definitely leave a note here once there's something to play with! 🙌

[z9ekaMen]

Just want to check if this feature have any plan for release? My project is migrated from i18next and i18next-parser to next-intl due to supportive matter on app router. But the parser problem set a drawback for the team due to a ICU plural problem. @amannn

[amannn]

There are no immediate release plans here, sorry. I'm planning to explore this topic further later this year, but it will take a bit of time—sorry.

If you rely on message extraction currently, you can consider a workaround such as the one shared here: #456 (comment).

[ChristianIvicevic]

For message extraction I have written a Next.js plugin that you could check out until next-intl gets some form of first class support for it. It parses your codebase and extracts i18n keys and warns you about unused keys (which it can prune automatically if configured): https://github.com/ChristianIvicevic/intl-watcher

[newme616]

For message extraction I have written a Next.js plugin that you could check out until next-intl gets some form of first class support for it. It parses your codebase and extracts i18n keys and warns you about unused keys (which it can prune automatically if configured): https://github.com/ChristianIvicevic/intl-watcher

this is awesome, thank you so much!

btw have you seen https://github.com/lingodotdev/lingo.dev they seem to achieve pretty good results (haven't tried though)