Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Preparing to) move NSS docs off MDN #2669

Closed
chrisdavidmills opened this issue Feb 24, 2021 · 25 comments
Closed

(Preparing to) move NSS docs off MDN #2669

chrisdavidmills opened this issue Feb 24, 2021 · 25 comments
Assignees
Labels
Content:Other Any docs not covered by another "Content:" label needs triage Triage needed by staff and/or partners. Automatically applied when an issue is opened.

Comments

@chrisdavidmills
Copy link
Contributor

chrisdavidmills commented Feb 24, 2021

We need to help the Mozilla NSS team move their documentation off MDN. To be clear, we are talking about everything underneath https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS.

The goal is to get to the stage where all of the documentation under /NSS is available as static, standalone HTML files that are free from KumaScript macros, have links fixed up as much as is possible, and can be used in the new destination (which we are hoping will be the Firefox source docs Sphinx instance)

The tasks required are:

    • Fix all the flaws generated on the NSS docs (this is a list of errors found for each document, mainly broken links).
    • Most of these are automatically fixable, so we should run the commands to do this.
    • Scrape all of the docs under the /NSS directory from the MDN site, and get rid of all the MDN-specific scaffolding to just leave a minimal standalone HTML document for each one (this includes removing any navigation menus). Present them in a directory structure the same as the existing directory structure in the content repo.
    • Come up with a list of manual fixes that are needed for each document. Make these fixes for each document to get them ready for use in the new system (I was thinking that a team comprising of me, another MDN writer, and Niklas from the NSS team, could do this work)
    • Once the content is prepared, the NSS team would need to do final sorting of the links inside the resource:
    • Change all internal links that link to other NSS docs like this — /en-US/docs/Mozilla/Projects/NSS/xxx > /new/relative/path/xxx
    • Change all internal links that link to other docs on MDN not inside the NSS section like so — /en-US/docs/Mozilla/xxx > https://developer.mozilla.org/en-US/docs/Mozilla/xxx
    • On the MDN side, our engineers would need to put hard redirects in place from the old locations of the pages on MDN, to the new locations of the pages.
@hamishwillee
Copy link
Collaborator

@chrisdavidmills Decided to be helpful on point 2 - bunch of links fixed on #2960. A good task to cool my brain down on.

There are still quite a few broken links showing up. Specifically everything to the project NSPR and also a few archived docs like https://developer.mozilla.org/en-US/docs/Archive/Security/Introduction_to_SSL

FYI, not an automatic task :-(

@chrisdavidmills
Copy link
Contributor Author

@chrisdavidmills Decided to be helpful on point 2 - bunch of links fixed on #2960. A good task to cool my brain down on.

There are still quite a few broken links showing up. Specifically everything to the project NSPR and also a few archived docs like https://developer.mozilla.org/en-US/docs/Archive/Security/Introduction_to_SSL

FYI, not an automatic task :-(

Yeah, unfortunately not, by the looks of things. I'm assuming in doing this work you are just working through the folder structure alphabetically? Do you think you can create a simple list in here that shows which directories have had their broken URLs fixed so far, and by what PR? Just so others that might work on it can easily see at a glance what else there is to be done?

@hamishwillee
Copy link
Collaborator

Possibly, but if you can tell me how to fix the NSPR links and confirm that we can ignore the archive links that would be a smaller task. This pretty much fixes everything else that can be fixed due to links being "accidentally broken"

@hamishwillee
Copy link
Collaborator

hamishwillee commented Mar 15, 2021

@chrisdavidmills OK, so I did a bunch more fixes in #3134.
Below is a list of what remains to be fixed. Not completely exhaustive, but enough to get a good picture on high level things that remain - a bunch of them could be bulk fixed if we knew the right targets.

Note also that many of these are actually archived and just showing up as broken link. See mdn/yari#2675. This applies to much of the "/en-US/docs/Mozilla/Projects/NSPR" links

Broken Links
/en-US/docs/Mozilla/Projects/NSS/Reference/NSS_cryptographic_module/Data_types 👀 line 20:15
/en-US/docs/Mozilla/Projects/NSS/Reference/NSS_cryptographic_module/Non-FIPS_mode_of_operation 👀 line 21:15

/en-US/docs/Using_CXX_in_Mozilla_code line 121:106

/en-US/docs/Mozilla/Developer_guide/Build_Instructions/CVS_Tags line 106:198
INVALID (archived): /en-US/docs/Archive/Using_SSH_to_connect_to_CVS line 19:502
INVALID (archived): /en-US/docs/Archive/Mozilla/Creating_a_Mozilla_extension/Tinderbox line 75:76

INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_Public-Key_Cryptography line 30:19
INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_SSL line 32:19
INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR/Reference line 122:19
INVALID (archived): /en-US/docs/Archive/Mozilla/JavaScript_crypto line 129:19

INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR line 28:57 /en-US/docs/Mozilla/Projects/NSPR
INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR line 28:326
INVALID (archived): /en-US/docs/Mozilla/Security/x509_Certificates line 47:15

INVALID (archived): /en-US/docs/Mozilla/Security/x509_Certificates line 46:415
INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR line 72:1481

INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR line 14:338

INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR 👀 line 25:272
INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_Public-Key_Cryptography line 141:15
INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_SSL line 142:15
INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR/Reference line 147:109

##Lots broken under the /Projects/NSS/Reference - just subset listed

The point here to see that many of these are common

/en-US/NSC_CancelFunction line 27:19

/en-US/NSC_CloseAllSessions line 27:109

/en-US/NSC_DigestInit line 28:19

/en-US/NSC_DigestFinal line 30:105

/en-US/NSC_DigestInit line 28:19

/en-US/NSC_SetAttributeValue 👀 line 32:19

/en-US/NSS/CERTIssuerAndSN 👀 line 19:260

No flaws reported:

@chrisdavidmills
Copy link
Contributor Author

Thanks @hamishwillee , that's super helpful.

I will see if I can find some time to do a few of these. Also feel free to to keep picking bits off ;-)

@hamishwillee
Copy link
Collaborator

Great. Will do.
FYI, I found the NSPR link - it exists, but is archived: https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSPR

Quite a few of these might also be fixed by creating one or two little docs - e.g. aCK_SESSION_HANDLE is clearly a copy of https://www.cryptsoft.com/pkcs11doc/v100/pkcs11__all_8h.html#aCK_SESSION_HANDLE

@hamishwillee
Copy link
Collaborator

hamishwillee commented Mar 19, 2021

Yet more done in #3290.

This is what is left. I think that I'm at the end of what I can do

Broken Links
/en-US/docs/Mozilla/Projects/NSS/Reference/NSS_cryptographic_module/Data_types 👀 line 20:15
/en-US/docs/Using_CXX_in_Mozilla_code line 121:106

/en-US/docs/Mozilla/Developer_guide/Build_Instructions/CVS_Tags line 106:198
INVALID (archived): /en-US/docs/Archive/Using_SSH_to_connect_to_CVS line 19:502
INVALID (archived): /en-US/docs/Archive/Mozilla/Creating_a_Mozilla_extension/Tinderbox line 75:76

INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_Public-Key_Cryptography line 30:19
INVALID (archived): /en-US/docs/Archive/Security/Introduction_to_SSL line 32:19
INVALID (archived): /en-US/docs/Mozilla/Projects/NSPR/Reference line 122:19
INVALID (archived): /en-US/docs/Archive/Mozilla/JavaScript_crypto line 129:19

/en-US/NSC_CancelFunction line 27:19

/en-US/NSC_CloseAllSessions line 27:109

/en-US/NSC_DigestInit line 28:19

/en-US/NSC_DigestFinal line 30:105

/en-US/NSC_DigestInit line 28:19

/en-US/NSC_SetAttributeValue 👀 line 32:19

/en-US/NSS/CERTIssuerAndSN line 19:260

@chrisdavidmills
Copy link
Contributor Author

@escattone a friendly ping on this one Ryan. @hamishwillee has done a whole bunch of work to fix broken links in these docs, so the next stage is to scrape the rendered NSS docs off the site into a "pure" HTML form that the NSS team can use.

Also pinging @beurdouche for an update as to the latest progress on this.

@beurdouche
Copy link
Contributor

Thanks a lot, this is very helpful! :)

Regarding the links, if you can point the Non-FIPS mode to the page you suggested, it would be great
https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/PKCS11_Functions

I think that there is no need to worry about the rest of the links.
This list is useful and I can fix/delete/rewrite those after we move off of MDN 👍

@hamishwillee
Copy link
Collaborator

hamishwillee commented Mar 26, 2021

Hi @beurdouche , the link you refer to will be fixed by #3513.

I am not promising on my life that all links are fixed - but certainly the main docs have all been gone through and any flaws shown by the automatic tooling have been fixed up. The ones marked as Invalid above show up as broken if you build locally but work on the main site (i.e. the scaped version will work)

@chrisdavidmills
Copy link
Contributor Author

Hi @beurdouche ! It looks like we've finished doing link fixes on this content, so it is ready to scrape and get into a rendered out HTML format for you folks to use.

In which case, the next stage is to create a scraper to scrape the content off of MDN. We will probably create a simple script to do this, and then hand it over to you folks so you can do the conversion and tweak it as needed.

My question here is — what do you want the HTML to look like, exactly? Do you need a full page for each, including doctype, simple document head, plus body, or do you just need the body content for each? Or something else?

@dergoegge
Copy link
Contributor

Hi @chrisdavidmills, thanks for all the work so far! I can answer this for @beurdouche.

Just the content would be the best for us. We basically just need the files the way they are on github right now but with the macros expanded.
(Easiest example would be index.html)

@chrisdavidmills
Copy link
Contributor Author

Hi @chrisdavidmills, thanks for all the work so far! I can answer this for @beurdouche.

Just the content would be the best for us. We basically just need the files the way they are on github right now but with the macros expanded.
(Easiest example would be index.html)

Do you want them with the YAML front matter included?

@dergoegge
Copy link
Contributor

Good question. Currently the conversion script uses the YAML to get the title but I am guessing that the scraped HTML will have the titles in it? If that is the case then we don't need the YAML.

@chrisdavidmills
Copy link
Contributor Author

Good question. Currently the conversion script uses the YAML to get the title but I am guessing that the scraped HTML will have the titles in it? If that is the case then we don't need the YAML.

That's correct — the scraped page would include the <h1>, as wll as all the KS macros being rendered out.

So assuming that you don't have any usage for the tags, slug, etc, then that sounds ok.

My esteemed colleague @escattone and his team would be helping with the actual scraping code here, so I'll pass this over to him. Ryan, the one thought I still have here is that we'd probably need to scrape them and provide the scraped content in a folder structure that matches the existing folder structure in the source, to help @dergoegge put them in the same kind of structure that they are currently on MDN, in their new home. This would in turn help us to put redirects to the new locations in place much more easily. Sound reasonable?

I was thinking you could start by scraping a few docs, and sharing them with Niklas to check if tey are ok for his purposes.

@escattone
Copy link
Contributor

@chrisdavidmills All sounds reasonable! Let me get cracking on the scraper this week, share an example with @dergoegge, and we can iterate from there.

@chrisdavidmills
Copy link
Contributor Author

@escattone thanks, much appreciated.

@escattone
Copy link
Contributor

@dergoegge @beurdouche I created an initial set of "distilled" NSS docs in https://github.com/mdn/nss-docs. By "distilled", I mean I extracted <main> and <title> from the MDN version of each doc, and re-packaged in a fresh, minimal <html> package (so basically stripped out the MDN header, footer, and sidebar). Does that work for you all? I'm happy to adjust as you wish. Cheers.

@dergoegge
Copy link
Contributor

@escattone Could you change it so the files only contain the inner html of <article class="main-page-content"> on each page?
Besides that it looks good to me so far👍

@escattone
Copy link
Contributor

@dergoegge I've updated https://github.com/mdn/nss-docs to reflect your requested change. Hope that works for you.

@mathewhodson
Copy link
Contributor

It looks looks like the work to add the docs to the source tree is at https://bugzilla.mozilla.org/show_bug.cgi?id=1709817

@Rumyra Rumyra added needs triage Triage needed by staff and/or partners. Automatically applied when an issue is opened. Content:Other Any docs not covered by another "Content:" label labels Jun 7, 2021
@Rumyra Rumyra removed the needs triage Triage needed by staff and/or partners. Automatically applied when an issue is opened. label Jun 7, 2021
@escattone
Copy link
Contributor

Thanks for the update @mathewhodson.

@hamishwillee
Copy link
Collaborator

The NSS files have been deleted from MDN. Any reason we can't close this? Speak now, or I'll close next time I come around.

I appreciate there may be more "work" being done to publish by the NSS team, but that is out of scope for the people working on content,.

@escattone
Copy link
Contributor

@hamishwillee 👍 I agree. I think we can close this.

@hamishwillee
Copy link
Collaborator

hamishwillee commented Jan 31, 2022

@Rumyra @schalkneethling I am reopening because this has still not been removed and Ryan has left the building.

The plan here was to delete the NSS tree, which is broken and which the NSS team were taking over. We did so in #8215 but it was reverted because they were not ready. Ryan was liasing to find a time but not as far as I could see actually pushing.

I would like to propose deletion of this tree within the month "come what may" as they've have more than enough time - we can put the source in archive if desired. Is there someone we can notify in that team? If so, who?

EDIT: Note, closing. Seems this came up 4 days ago and is being addressed in #12471

@hamishwillee hamishwillee reopened this Jan 31, 2022
@github-actions github-actions bot added the needs triage Triage needed by staff and/or partners. Automatically applied when an issue is opened. label Jan 31, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 31, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Content:Other Any docs not covered by another "Content:" label needs triage Triage needed by staff and/or partners. Automatically applied when an issue is opened.
Projects
None yet
Development

No branches or pull requests

7 participants