[WIP] Nodejs i18n module #133

lukaszewczak · 2018-07-09T04:41:37Z

Hi all,

After some break, I slowly start to work on #92 to create nodejs-i18 module, based on electron-i18 module. I set up this PR as WIP because I stuck on one problem, where maybe we all will be able to discuss it for some good ideas.

So I start to prepare script based on electron-module, but I want to build Object with locales grouped by node.js version, but for me It was the first time, when my node.js process was gone with error Allocation failed - JavaScript heap out of memory, so I think it will be good to create separate json file for specific node.js version with one main index.js module which exports function with parameter node.js version, which will return specific json file, but I had problem with saving this big javascript object to file, it should be probably done with streams, but I do not know how to write JS object with streams to file 😞, even if I will be able to prepare this file , it could be difficult to consume it by another project because of the json file size.

@zeke had suggestion to use leveldb, I think this can be a good direction to use some store like this one.

Here is a current sample of created json file, only for node.js version 10 and `en-US` locale, without parsed markdown files.

{
  "v10.x": {
    "locales": {
      "en-US": {
        "locale": "en-US",
        "languageName": "English",
        "languageNativeName": "English",
        "countryCode": "US",
        "countryName": "United States"
      }
    },
    "docs": {
      "en-US": [
        {
          "locale": "en-US",
          "slug": "STYLE_GUIDE",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "_toc",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "addons",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "all",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "assert",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "async_hooks",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "buffer",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "child_process",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "cli",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "cluster",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "console",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "crypto",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "debugger",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "deprecations",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "dgram",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "dns",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "documentation",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "domain",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "errors",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "esm",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "events",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "fs",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "globals",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "http",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "http2",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "https",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "index",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "inspector",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "intl",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "modules",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "n-api",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "net",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "os",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "path",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "perf_hooks",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "process",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "punycode",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "querystring",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "readline",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "repl",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "stream",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "string_decoder",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "synopsis",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "timers",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "tls",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "tracing",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "tty",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "url",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "util",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "v8",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "vm",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "zlib",
          "category": "api",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "CHANGELOG_ARCHIVE",
          "category": "changelogs",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "CHANGELOG_IOJS",
          "category": "changelogs",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "CHANGELOG_V010",
          "category": "changelogs",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "CHANGELOG_V012",
          "category": "changelogs",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "CHANGELOG_V10",
          "category": "changelogs",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "CHANGELOG_V4",
          "category": "changelogs",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "CHANGELOG_V5",
          "category": "changelogs",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "CHANGELOG_V6",
          "category": "changelogs",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "CHANGELOG_V7",
          "category": "changelogs",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "CHANGELOG_V8",
          "category": "changelogs",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "CHANGELOG_V9",
          "category": "changelogs",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "backporting-to-release-lines",
          "category": "guides",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "building-node-with-ninja",
          "category": "guides",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "coc",
          "category": "guides/contributing",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "issues",
          "category": "guides/contributing",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "pull-requests",
          "category": "guides/contributing",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "maintaining-V8",
          "category": "guides",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "maintaining-npm",
          "category": "guides",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "maintaining-the-build-files",
          "category": "guides",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "node-postmortem-support",
          "category": "guides",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "using-internal-errors",
          "category": "guides",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "writing-and-running-benchmarks",
          "category": "guides",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "writing-tests",
          "category": "guides",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "offboarding",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "onboarding-extras",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "onboarding",
          "sections": "here will be parsed markdown as html"
        },
        {
          "locale": "en-US",
          "slug": "releases",
          "sections": "here will be parsed markdown as html"
        }
      ]
    }
  }
}

zeke · 2018-07-09T19:19:40Z

This is a great start, @lukaszewczak!

sections is a thing that is unique to Electron for its language toggle feature on the website, so this should probably just be called html or something.

zeke · 2018-07-18T22:14:26Z

Hey @lukaszewczak, sorry to leave you hanging. I'm finding a little time to dig into this now.

Rather than the exported module being giant JSON file(s) that are too big to fit in memory, I think we should explore exporting a leveldb instance.

Upsides:

less memory-intensive build process
less memory-intensive require('nodejs-i18n') because you're not loading a big object into memory
leveldb has good compression on disk, so the exported module would be smaller than JSON files
streaming interface is fast and easy to consume

Downsides:

adds a dependency on level to the exported module. Also it's a native module which can cause headaches sometimes.
It's async so a little more cumbersome to use that non-async.
can't think of any other downsides.. :)

How it could work:

leveldb is a simple key-value data store, and I think keys can actually be objects. Every file is identifiable by three unique properties: node version, language, and filename. We could store entries using a key like this:

const key = {
  nodeVersion: 'v10.x',
  language: 'en-US',
  filename: 'doc/api/path.md'
}

If one wanted to fetch a specific record:

const i18n = require('nodejs/i18n')
const key = {
  nodeVersion: 'v10.x',
  language: 'en-US',
  filename: 'doc/api/path.md'
}
const file = await i18n.get(key)

And if one wanted to operate on all the records in a specific language and/or version:

const i18n = require('nodejs/i18n')
i18n.createReadStream()
  .on('data', ({key, value}) => {
    const {nodeVersion, language, filename} = key
    if (nodeVersion === 'v8.x' && language === 'en-US') {
      console.log(value)
    }
  })
  .on('end', () => {
    console.log('done')
  })

cc level-wiz @juliangruber 🎩 🐇 -- Does this make sense? Am I right about the keys? Is this a good idea? A bad idea?

zeke · 2018-07-18T22:23:10Z

In related news, I noticed that @rubys is working on nodejs/node#21490, an effort to replace marked to generate the current website's HTML. We should see if there are opportunities to collaborate / copy / work together on that.

zeke · 2018-07-19T00:09:36Z

I pushed up a new branch called level-module based on this branch. Noteworthy changes are here:

i18n/script/build-module.js

Lines 16 to 24 in c9ecda0

    
           for (let file of markdownFiles) { 
        
             file.nodeVersion = nodeVersion 
        
             file = await parseFile(file) 
        
             const key = { 
        
               nodeVersion: file.nodeVersion, 
        
               locale: file.locale, 
        
               path: file.path 
        
             } 
        
             await db.put(key, file)

lukaszewczak · 2018-07-19T06:20:21Z

Hi @zeke, Thank you for your work, which is great and definnetly a better approch. I'm sorry but I'm currently on holiday, this is why I wasn't active and I can back to this subject after July 28. Should I close this PR if you prepared branch based on level module?

I have one concern about this module specific to the usage of this module. If the new node.js website will have a dependency on this module, how often maintainers of the project will want to update dependency to have a new version of translations. I'm thinking about using some cloud for store...

zeke · 2018-07-19T16:22:32Z

If the new node.js website will have a dependency on this module, how often maintainers of the project will want to update dependency to have a new version of translations.

For Electron's website we have a process that runs on Heroku every ten minutes to update trusted dependencies: https://github.com/electron/electronjs.org/blob/a6ea42d6b66afff0ebbd695d13f145f5cc2666c0/script/update-deps.sh#L18-L27

lukaszewczak · 2018-08-31T07:34:04Z

Hi @zeke , I saw the discussion about that module on nodejs/website-redesign#76 (comment), its great that it can be used, can I help you somehow, with this npm module? Should I close this PR?

zeke · 2020-03-27T21:12:52Z

@lukaszewczak this is pretty out of date so I'm going to close this out. Let's move further discussion to #261

lukaszewczak added 4 commits July 2, 2018 20:44

[style] Format files based on standard code style

f1350b8

[chore] Update package.lock

14fc037

[feat] Add module for retrieve locales for specific node.js version

aa3b6ca

[feat][WIP] Add build-module script to prepare npm module

8d69484

zeke mentioned this pull request Aug 1, 2018

Consuming localized docs from nodejs-i18n module nodejs/website-redesign#76

Closed

zeke mentioned this pull request Aug 17, 2018

Node.js Foundation Website Redesign Strategic Initiative Meeting 2018-08-16 nodejs/website-redesign#79

Closed

This comment was marked as off-topic.

Sign in to view

tanhengyeow mentioned this pull request Nov 24, 2018

Add language around emeriti to docs 📝 #302 nodejs/community-committee#417

Merged

amorist mentioned this pull request Dec 4, 2018

isolate the module and docs #186

Closed

zeke mentioned this pull request Feb 7, 2019

website content to be translated #72

Closed

zeke mentioned this pull request Mar 17, 2020

Create LevelDB powered npm module #245

Closed

alexandrtovmach mentioned this pull request Mar 27, 2020

Discussion about NPM module #261

Closed

zeke closed this Mar 27, 2020

alexandrtovmach deleted the nodejs-i18n branch April 14, 2020 12:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Nodejs i18n module #133

[WIP] Nodejs i18n module #133

lukaszewczak commented Jul 9, 2018

zeke commented Jul 9, 2018

zeke commented Jul 18, 2018 •

edited

Loading

zeke commented Jul 18, 2018

zeke commented Jul 19, 2018

lukaszewczak commented Jul 19, 2018

zeke commented Jul 19, 2018

lukaszewczak commented Aug 31, 2018

This comment was marked as off-topic.

zeke commented Mar 27, 2020

[WIP] Nodejs i18n module #133

[WIP] Nodejs i18n module #133

Conversation

lukaszewczak commented Jul 9, 2018

zeke commented Jul 9, 2018

zeke commented Jul 18, 2018 • edited Loading

zeke commented Jul 18, 2018

zeke commented Jul 19, 2018

lukaszewczak commented Jul 19, 2018

zeke commented Jul 19, 2018

lukaszewczak commented Aug 31, 2018

This comment was marked as off-topic.

zeke commented Mar 27, 2020

zeke commented Jul 18, 2018 •

edited

Loading