Skip to content

feat: AOT compilation with icu-to-json (experiment) #705

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

feat: AOT compilation with icu-to-json (experiment) #705

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
ee8279c
experiment: Try out `icu-to-json`
amannn Dec 6, 2023
7997a50
Some more tests passing
amannn Dec 6, 2023
0f25794
Fix more tests
amannn Dec 7, 2023
e89de9c
Remove unused dependency
amannn Dec 7, 2023
2dc39b5
Some cleanup
amannn Dec 7, 2023
3fa4253
Fix more tests
amannn Dec 7, 2023
44337ea
Add comment
amannn Dec 7, 2023
69bfe0c
Cleanup
amannn Dec 7, 2023
1d44910
Cleanup
amannn Dec 7, 2023
7afec20
Use `compileToJson` instead of `compile`
amannn Dec 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/use-intl/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@
],
"dependencies": {
"@formatjs/ecma402-abstract": "^1.11.4",
"intl-messageformat": "^9.3.18"
"icu-to-json": "0.0.20"
},
"peerDependencies": {
"react": "^16.8.0 || ^17.0.0 || ^18.0.0"
Expand Down
9 changes: 9 additions & 0 deletions packages/use-intl/src/core/MessageFormat.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
import type {compile} from 'icu-to-json/compiler';

type MessageFormat = Omit<
ReturnType<typeof compile>,
// TODO: Do we need the args?
'args'
Copy link
Owner Author

@amannn amannn Dec 7, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this be removed from the parsed result? See also https://github.com/amannn/next-intl/pull/705/files#r1418720864

Copy link

@jantimon jantimon Dec 8, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use compileToJson instead it's the same as compile just without args

args are helpful to generate types or to optimize which formatters should be included to a bundle.
e.g. if date is used you might want to add a date formatter

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, thanks!

>;

export default MessageFormat;
7 changes: 3 additions & 4 deletions packages/use-intl/src/core/MessageFormatCache.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
// eslint-disable-next-line import/no-named-as-default -- False positive
import type IntlMessageFormat from 'intl-messageformat';
import MessageFormat from './MessageFormat';

type MessageFormatCache = Map<
/** Format: `${locale}.${namespace}.${key}.${message}` */
string,
IntlMessageFormat
string, // Could simplify the key here
MessageFormat
>;

export default MessageFormatCache;
63 changes: 0 additions & 63 deletions packages/use-intl/src/core/convertFormatsToIntlMessageFormat.tsx
View file
Open in desktop

This file was deleted.

111 changes: 31 additions & 80 deletions packages/use-intl/src/core/createBaseTranslator.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +1,18 @@
// eslint-disable-next-line import/no-named-as-default -- False positive
import IntlMessageFormat from 'intl-messageformat';
import {
cloneElement,
isValidElement,
ReactElement,
ReactNode,
ReactNodeArray
} from 'react';
import {evaluateAst} from 'icu-to-json';
import {compile} from 'icu-to-json/compiler';
import {ReactElement} from 'react';
import AbstractIntlMessages from './AbstractIntlMessages';
import Formats from './Formats';
import {InitializedIntlConfig} from './IntlConfig';
import IntlError, {IntlErrorCode} from './IntlError';
import MessageFormat from './MessageFormat';
import MessageFormatCache from './MessageFormatCache';
import TranslationValues, {
MarkupTranslationValues,
RichTranslationValues
} from './TranslationValues';
import convertFormatsToIntlMessageFormat from './convertFormatsToIntlMessageFormat';
import {defaultGetMessageFallback, defaultOnError} from './defaults';
import getFormatters from './getFormatters';
import MessageKeys from './utils/MessageKeys';
import NestedKeyOf from './utils/NestedKeyOf';
import NestedValueOf from './utils/NestedValueOf';
Expand Down Expand Up @@ -56,34 +51,6 @@ function resolvePath(
return message;
}

function prepareTranslationValues(values: RichTranslationValues) {
if (Object.keys(values).length === 0) return undefined;

// Workaround for https://github.com/formatjs/formatjs/issues/1467
const transformedValues: RichTranslationValues = {};
Object.keys(values).forEach((key) => {
let index = 0;
const value = values[key];

let transformed;
if (typeof value === 'function') {
transformed = (chunks: ReactNode) => {
const result = value(chunks);

return isValidElement(result)
? cloneElement(result, {key: key + index++})
: result;
};
} else {
transformed = value;
}

transformedValues[key] = transformed;
});

return transformedValues;
}

function getMessagesOrError<Messages extends AbstractIntlMessages>({
messages,
namespace,
Expand Down Expand Up @@ -132,23 +99,6 @@ export type CreateBaseTranslatorProps<Messages> = InitializedIntlConfig & {
messagesOrError: Messages | IntlError;
};

function getPlainMessage(candidate: string, values?: unknown) {
if (values) return undefined;

const unescapedMessage = candidate.replace(/'([{}])/gi, '$1');

// Placeholders can be in the message if there are default values,
// or if the user has forgotten to provide values. In the latter
// case we need to compile the message to receive an error.
const hasPlaceholders = /<|{/.test(unescapedMessage);

if (!hasPlaceholders) {
return unescapedMessage;
}

return undefined;
}

export default function createBaseTranslator<
Messages extends AbstractIntlMessages,
NestedKey extends NestedKeyOf<Messages>
Expand Down Expand Up @@ -196,7 +146,7 @@ function createBaseTranslatorImpl<
values?: RichTranslationValues,
/** Provide custom formats for numbers, dates and times. */
formats?: Partial<Formats>
): string | ReactElement | ReactNodeArray {
): string | ReactElement | Array<ReactElement> {
if (messagesOrError instanceof IntlError) {
// We have already warned about this during render
return getMessageFallback({
Expand Down Expand Up @@ -224,7 +174,7 @@ function createBaseTranslatorImpl<

const cacheKey = joinPath([locale, namespace, key, String(message)]);

let messageFormat: IntlMessageFormat;
let messageFormat: MessageFormat;
if (messageFormatCache?.has(cacheKey)) {
messageFormat = messageFormatCache.get(cacheKey)!;
} else {
Expand All @@ -251,19 +201,8 @@ function createBaseTranslatorImpl<
return getFallbackFromErrorAndNotify(key, code, errorMessage);
}

// Hot path that avoids creating an `IntlMessageFormat` instance
const plainMessage = getPlainMessage(message as string, values);
if (plainMessage) return plainMessage;

try {
messageFormat = new IntlMessageFormat(
message,
locale,
convertFormatsToIntlMessageFormat(
{...globalFormats, ...formats},
timeZone
)
);
messageFormat = compile(message);
} catch (error) {
return getFallbackFromErrorAndNotify(
key,
Expand All @@ -276,14 +215,29 @@ function createBaseTranslatorImpl<
}

try {
const formattedMessage = messageFormat.format(
// @ts-expect-error `intl-messageformat` expects a different format
// for rich text elements since a recent minor update. This
// needs to be evaluated in detail, possibly also in regards
// to be able to format to parts.
prepareTranslationValues({...defaultTranslationValues, ...values})
const allValues = {...defaultTranslationValues, ...values};
// TODO: The return type seems to be a bit off, not sure if
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are two different ways to run the precompiled icu messages:

  1. run
  2. evaluateAst

the idea is that run will always return a string and evaluateAst will always return an array.

const t = (key, args) => run(messages[key], lang, args);

icu-to-json is framework independent so it would require you to wrap a react Fragment around the result:

t.rich = (key, args) => <>{evaluateAst(messages[key], lang, args)}</>;

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, that helps with the fragment—thanks!

I was wondering about this case:

Screenshot 2023-12-11 at 21 24 31

It seems like the args determine the return value. However, the function b has already been called at this point (having returned a React.JSX.Element) and date should be turned into a string.

Therefore the returned value should have the type (string | React.JSX.Element)[], right?

Or am I missing something?

// this should be handled in `icu-to-json` or here.
const evaluated = evaluateAst(
messageFormat.json,
locale,
allValues,
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I pass the values from the user or args from the parsed result here?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the args are not needed for formatters however the args can tell you during compile time which formatters need to be available to run the code.

in your case there is a very special formatter wich you should add for full react support

it allows processing the children before they are passed to the given tag function:

{
   tag: (children: Array<string | ReactNode>, locale: string) => 
         children.map((value, i) => typeof value === "string"
            ? value 
            : <Fragment key={`f-${i}`}>{value}</Fragment>
}

e.g. for a translation like <foo>Hello <b>{name}</b></foo>

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is also the baseTag formatter which is a fallback which kicks in if the end-user did not define a tag.

by default the tag is converted to a string but you might also wrap in a fragment or even allow some tags to be used as html:

// for a compiled message "<b>Hello {name}</b>" 
const formatters = {
  baseTag: (Tag, chilren) => {
      // allowlist:
      if (["b", "strong", "p"].include(Tag)) { 
        return <Tag>{children}</Tag>
      }
      return <>{children}</>;
  }
};
run(message, lang, { name: "Joe" }, formatters)

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried updating your example here to use nested rich text: https://codesandbox.io/p/sandbox/icu-to-json-demo-forked-2jzngr?file=%2Fsrc%2Ficu.tsx%3A19%2C1-20%2C1

Somehow the inner part is missing. Am I doing something wrong?

getFormatters(timeZone, formats, globalFormats)
);

let formattedMessage;
if (evaluated.length === 0) {
// Empty
formattedMessage = '';
} else if (evaluated.length === 1 && typeof evaluated[0] === 'string') {
// Plain text
formattedMessage = evaluated[0];
} else {
// Rich text
formattedMessage = evaluated;
}

// TODO: Add a test that verifies when we need this
if (formattedMessage == null) {
throw new Error(
process.env.NODE_ENV !== 'production'
Expand All @@ -294,11 +248,8 @@ function createBaseTranslatorImpl<
);
}

// Limit the function signature to return strings or React elements
return isValidElement(formattedMessage) ||
// Arrays of React elements
Array.isArray(formattedMessage) ||
typeof formattedMessage === 'string'
// @ts-expect-error Verify return type (see comment above)
return Array.isArray(formattedMessage)
? formattedMessage
: String(formattedMessage);
} catch (error) {
Expand Down
Loading