feat: Hash based id

[timofei-iatsenko]

Implementation for: #1360

For easier reviewing, i split this feature in the different commits. I recommend checking them one by one.

• implement in js macro and update tests
• implement in jsx macro and update tests
• cleanup runtime / extractors from "runtime" context
• changes in formats
• porting macro changes to SWC Plugin feat: implement hash based ids for js macro swc-plugin#19
• fallbackLocales, fallback to template, and fallback in case of empty translation refactor: getTranslations, prepare for hash-based ids #1443
• documentation

implementation details

1. Js macro now always generate call with message descriptor, no matter what input was

    i18n._({id: "mjnlP0", message: "Ola!"})

2. Id is generated as base64(sha256(value)).length(6). Unlike the formatjs where this placeholder is configurable, i explicitly didn't want to give flexibility here. Because it comes down to complexity of porting this to Rust.
3. i found very exhausting to updating tests with new ids, since we don't use jest snapsohots for them. So i created a small babel plugin used in tests which is stripping ids from the test cases where checking id is not a purpose of test.
4. i created a separate package message-id which helps with creating ids from messages. I need it in few places that's why it's extracted.
5. context has effect only on message id generation and in PO formatter (all during compile time). All usages in compile and runtime was removed. Context property also stripped in production mode.
6. po formatter works a bit tricky. We don't want to have our hash based id in the msgid field since it against the gettext ideology. But we also don't want to break compatibility with custom ids provided by users. For that purpose during serialization we're storing a flag generated-id in po file, which then used when deserializing to regenerate hash id.
   input:

    {
      id: <hash>,
      message: "Ola!",
      context: "My Contex",
    }

   output:

   #, generated-id
    msgctxt "my context"
    msgid "Ola!"
    msgstr ""

   We also can do vice-versa, store flag when explicit id provided, this is TBD. (and like this option more because in my mind there should more autogenerated id's than custom)

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Fixes # (issue)

Checklist

• I have read the CONTRIBUTING and CODE_OF_CONDUCT docs
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated
js-lingui	✅ Ready (Inspect)	Visit Preview	💬 Add your feedback	Feb 21, 2023 at 1:18PM (UTC)

[github-actions]

size-limit report 📦

Path	Size
./packages/core/build/esm/index.js	1.76 KB (0%)
./packages/detect-locale/build/esm/index.js	812 B (0%)
./packages/react/build/esm/index.js	1.79 KB (0%)
./packages/remote-loader/build/esm/index.js	7.29 KB (0%)

[codecov]

Codecov Report

Base: 72.27% // Head: 73.89% // Increases project coverage by +1.62% 🎉

Coverage data is based on head (e3882c8) compared to base (6b6adc9).
Patch coverage: 89.22% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             next    #1440      +/-   ##
==========================================
+ Coverage   72.27%   73.89%   +1.62%     
==========================================
  Files          74       75       +1     
  Lines        1897     1923      +26     
  Branches      510      513       +3     
==========================================
+ Hits         1371     1421      +50     
+ Misses        422      383      -39     
- Partials      104      119      +15     

Impacted Files	Coverage Δ
packages/macro/src/utils.ts	100.00% <ø> (ø)
packages/macro/test/js-arg.ts	100.00% <ø> (ø)
packages/macro/test/js-defineMessage.ts	100.00% <ø> (ø)
packages/macro/test/js-plural.ts	100.00% <ø> (ø)
packages/macro/test/js-select.ts	100.00% <ø> (ø)
packages/macro/test/js-selectOrdinal.ts	100.00% <ø> (ø)
packages/macro/test/js-t.ts	100.00% <ø> (ø)
packages/macro/test/jsx-plural.ts	100.00% <ø> (ø)
packages/macro/test/jsx-select.ts	100.00% <ø> (ø)
packages/macro/test/jsx-selectOrdinal.ts	100.00% <ø> (ø)
... and 26 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[JSteunou]

Thank you for leading this hard work @thekip this is a top notch long time awaiting feature :)

About the hash, would sha256 be a bit more time consuming than sha1 ?

[timofei-iatsenko]

About the hash, would sha256 be a bit more time consuming than sha1 ?

I have been thinking about what hash function to use for a while. The criteria were - short / with low collision rate / non cryptographical / implemented in Rust

I considered

• https://sites.google.com/site/murmurhash/
• adler32

Related SO topics:
https://stackoverflow.com/questions/4567089/hash-function-that-produces-short-hashes

But i couldn't choose and just took what formatjs is using as default, assuming they choose it by the reason.

I think for regular catalogs with few thousands of messages, the difference in speed between sha256 or sha1 would be negligible

[JSteunou]

I think for regular catalogs with few thousands of messages, the difference in speed between sha256 or sha1 would be negligible

That what it seems indeed https://github.com/hex7c0/nodejs-hash-performance 👍 ~same speed, less collision

[timofei-iatsenko]

@Martin005 @andrii-bodnar guys i want a feedback on these changes before I start porting it to the Rust. Updating test suite is extremely exhausting, so i don't want to do it more than one time.

[andrii-bodnar]

@thekip currently I have that one suggestion about the new package

also, I'm worrying about this PO flag - it looks like the PO Gettext specification has already a predefined list of allowed flags, isn't it? Or it allows any other flags also?

[timofei-iatsenko]

@andrii-bodnar new package is not related to Rust macro, so i can start porting.

Or it allows any other flags also?

Yes, there is a predefined list of flags which native gettext tools understand. But specification says more generously:

#, flag…

And also they say

The comment lines beginning with #, are special because they are not completely ignored by the programs as comments generally are. The comma separated list of flags is used by the msgfmt program to give the user some better diagnostic messages.

Means it's designed to affect message interpretation somehow, unlike to other comments which could be safely omitted.

[Martin005]

@thekip We have agreed with @andrii-bodnar that it would be best to postpone the flattening in extraction (drafted in #1431) for now.

[timofei-iatsenko]

@Martin005 actually to fix issue with flattening we can make js-lingui-id required in po files. And don't calculate id when reading from (message + context) but only take this hash. Something like this is made for po-gettext extractor. Where comments used to store original icu components.

#. js-lingui-id: <hash> 
msgctxt "..."
msgid "..."
msgstr ""

[timofei-iatsenko]

Guys, everything related to docs, just do it yourself. I'm just not able to do it all by myself.

[andrii-bodnar]

@thekip please just resolve the current suggestions and then we'll take care of its polishing