Client spec v1.2: new --language flag, BCP47 to POSIX and other clarifications

[sbrl]

As per the discussion in #3161, this PR adds a new required --language CLI argument to the spec. Once clients have good support for it, we may consider updating the tldr page for tldr (though we should be careful to mention there that pages are returned in the system language by default).

Ping @pepa65 (tldr-bash-client, has language support already), @elecprog (elecprog/tldr, also has language support), @felixonmars (tldr-python-client), @agnivade (tldr-node-client) - please feel free to @mention anyone else who maintains a client that I've missed (I'm probably missing a bunch).

[sbrl]

Also, should we perhaps add in a clause that says that clients should mention what version of this spec they implement in their readmes?

[agnivade]

Ahh .. -l is already taken. Have you thought of choosing some other single letter to have a single letter flag ? I don't know .. maybe -e, -g ?

[sbrl]

Ah, good idea. Not sure what to choose that would make sense, actually.

[mebeim]

As far as I am concerned, having a language flag is not necessary. Most (if not all) command line programs I've seen that support localization use environment variables to detect the language. See for example man man. This is not a compulsory nor expected feature for a client, and it would also require to cache all pages for all languages (if the client uses caching), which would not be ideal and probably not what most of the users would want (for example, I personally wouldn't really like to waste disk space holding thousands of pages translated in languages that I don't understand).

We should ask clients to automatically detect language (as we already do), with an additional encouragement to make it configurable for the user (via configuration files or command-line options), but I would not suggest it as a must have feature.

[mebeim]

As per what I just said, I would suggest the following change (see below).

I would also suggest adding the following paragraph in the Language section:

If possible, it is recommended to not only rely on the environment, but also make language configurable. Clients SHOULD therefore offer options to configure or override the language using configuration files or command line arguments (like -L, --language suggested in the arguments section above).

[sbrl]

Great point, @mebeim. Ideally we want to encourage the user to use the environment variable, since that's a generally accepted standard.

Should we keep the --language after all then? 🤔

[mebeim]

@sbrl yes, like I said I think we should encourage to have such flag or a config file, but not make it a must have feature.

[stale]

Hi all! This thread has not had any recent activity. Are there any updates? Thanks!

[agnivade]

in progress

[principis]

• please feel free to @mention anyone else who maintains a client that I've missed (I'm probably missing a bunch).

@sbrl tldr-sharp had language support since 1.2.0 ;)
Good to see that it'll be included in the spec. :)

[mebeim]

As promised in #3183 (comment), I've integrated in this PR the changes regarding language and language tags (or, more precisely, locale tags), and I've changed the title accordingly. Since I believe that having fewer versions is better to avoid confusion, I also integrated some other changes.

I think this PR should be rebased in order to maintain the meaningfulness of the commits, but it could be also squashed: in such case please use the summary I'm adding below. Forgive the force-pushing, you can still see the commit history clearly from the relative tab.

Here's a summary of what happens in version 1.2 of the client specification:

• Addition of the recommended -L, --language command-line option.
• Shift from BCP-47 tags to POSIX locale names, with deprecation of previous spec versions.
• Overall reformatting of the document: use of --- and === to define section headers and more spacing, better spacing in tables using spaces instead of tabs.
• Rewording and reordering of the language section, also encouraging the use of configuration files for language.
• Rewording of the term "arguments" into "options`: the values described in the table are command-line options, "argument" is a more general term.
• Conversion of the "Other Consideration" section into "Caching" with additional clarifications: since the only thing being discussed under this section was caching.

I would suggest reading the whole document when reviewing, other than checking commits singularly, since changes are basically all around the document.

NOTE: this should be merged only after the changes discussed in #3183 are addressed. In other words, let's first apply the renaming of the folders and the change of the spec before updating the client spec as to not create confusion. As I already said in #3183, I will take care of this in a separate PR as soon as I can. Either way I'm waiting some time to see what you guys think about this before going further.

[agnivade]

Just gave some minor comments. Looks good otherwise.

and the change of the spec before updating the client spec as to not create confusion.

Which other spec did you mean ?

[mebeim]

@agnivade sorry, bad wording, not really a "spec", I was talking about CONTRIBUTING.md, we still talk about BCP-47 there. I will update it along with folder names in another PR.

tldr/CONTRIBUTING.md

Line 78 in 4445f03

Language specific directories must follow the pattern `pages.<language_tag>`, where `<language_tag>` is a [BCP 47](https://tools.ietf.org/html/bcp47) conforming tag in the form of `<language>[-<region>]`, where:

Fixed the typos.

[agnivade]

Got it. Thanks.

[principis]

Some thought I just had, what is more important, platform or language?
Not sure if this should be added to the spec.
But for example there is a page "X" and we have our default language nl_BE. LANGUAGE=fr:de:en and the platform is linux.
So what I'm doing now is checking every language for the Linux platform if it doesn't exist, look for the "common" platform. If that doesn't exist look for other platform and show a warning.

But maybe we should look for Linux and common first for each language before going to the next language?

[mebeim]

@principis I thought lookup should be like this:

$ LANGUAGE="it:en" tldr some-page

pages.it/linux/some-page.md -> does not exist
pages.it/common/some-page.md -> does not exist
pages/linux/some-page.md -> does not exist
pages/common/some-page.md -> FOUND!

Display pages/common/some-page.md

BUT that's a good question... now that you make me think about it I think it would make more sense for the platform to be more important.

This is a "problem" we have because common holds pages that can also be in other platforms. If that didn't happen, giving precedence to platform or to language wouldn't change anything.

[sbrl]

Hrm, that is indeed an issue. Looks like we've got 2 options:

• Language-first: Choose a page from another platform and prefer pages in the specified language
• Platform-first: Choose a page in another language in order to keep the platform the same.

I'd probably say platform-first is the better option:

pages.it/linux/some-page.md -> does not exist
pages/linux/some-page.md -> does not exist
pages.it/common/some-page.md -> does not exist
pages/common/some-page.md -> FOUND!

[mebeim]

@sbrl yes, that's what I was thinking. Platform first makes more sense. We cannot do much except having common pages only if the page is the same on all systems. It's up to the client developers to choose which of the two methods makes more sense.

[principis]

(So I wrote a long explanation but then had to leave and now it's gone...)
IMO platform is more important, I've currently implemented it like @sbrl mentioned. (With a fallback to English). But maybe this doesn't need to be in the spec? I did spend some time how to implement it correctly and consistent, so it could be added in the spec as a suggestion.

I think language-first really shouldn't be implemented (depends on how you do it). It could for example show a windows page on the linux platform because it's not in your first selected language, for example it, but in your second, en, there is a page for linux.

Platform-first doesn't have this problem.

[mebeim]

@principis yes I agree that platform first is really the only one which makes sense, and I also agree that it should be in the spec at least as a "SHOULD" or "RECOMMENDED" clause. I will add it to this PR as soon as I can. Language-first could cause problems for users in situations like the one you just explained.

[stale]

Hi all! This thread has not had any recent activity. Are there any updates? Thanks!

[sbrl]

@mebeim is working on this @probot.

[mebeim]

Welp, I almost forgot about this, thank you @probot 😅

I pushed the last commit applying @principis suggestions. After this, it should be pretty much ready to merge. Let me know what you guys think.

[mebeim]

Ping @agnivade, looks fine to merge for me. Any comments?

[mebeim]

Since this is a big PR and I'm basically co-author of the PR I'll leave to someone else to merge. It should be fine rebased rather than squashed. If you really want to squash please keep a bullet point list of the changes in the extended commit message!

[sbrl]

Apparently I can't approve my own PR, but this looks fine to me :D

[sbrl]

We should have some way of @mentioning all client authors at once.

[principis]

Welp, I almost forgot about this, thank you @probot sweat_smile

I pushed the last commit applying @principis suggestions. After this, it should be pretty much ready to merge. Let me know what you guys think.

(I really need to fix my notifications... Didn't see this.) it's too late but still, looks good, thanks!