[WIP] Migrate HTML to RST by joel-hamill · Pull Request #4536 · apache/kafka

joel-hamill · 2018-02-06T23:56:41Z

This PR migrates the HTML documentation content to restructuredText.

ewencp · 2018-02-07T00:20:35Z

@joel-hamill _build directory should not be included here. in fact we'll probably want a .gitignore entry for it.

WIP Add _build to gitignore remove build dir WIP

ewencp · 2018-02-26T22:46:34Z

+    introduction
+    quickstart
+    uses
+    documentation/index


having this nested structure like this is kinda weird.

@ewencp agreed. here's my idea: remove the 2nd Introduction, since it is a duplicate. and then won't the TOC actually look like:
HOME
INTRODUCTION
QUICKSTART
USE CASES
DOCUMENTATION

Kafka APIs

Configuration

Design

Implementation

Operations

Security

Kafka Connect

Kafka Streams

PERFORMANCE
POWERED BY
PROJECT INFO
ECOSYSTEM
CLIENTS
EVENTS
CONTACT US
APACHE

Currently the Intro is just a quick link into the documentation section. I'm +1 for removing the second introduction link.

ewencp · 2018-02-26T23:02:36Z

+=============
+
+.. contents::
+    :local:


Do we want to limit these local TOCs? At current depth this sometimes pushes the first content below the fold for many laptop users and I'm not sure how much value we get past the first level now that these are broken up topically.

@ewencp i don't think we should, unless the AK site will support left-hand TOC expansion (like CP docs). is that going to happen?

I was avoiding doing the left-hand TOC expansion for now since it requires more styling. It's absolutely possible to do though. In this case though, the limits I meant were just to do, e.g., 2 levels. The current version looks like this:

I was just questioning whether, e.g., the individual sections 3 levels deep under, e.g., "Developing a Simple Connector" are that valuable. Another way of putting it is that a section that deep in the page is individually useful, maybe we want to split this up further (which is perhaps related to my other comment about preemptively providing a connect/ directory even if we start out only with index.html).

ewencp · 2018-02-26T23:03:43Z

+============
+
+.. contents::
+    :local:


I think we should consider not using local TOC on pages like this. The only real point to this page is to be read linearly anyway and the TOC just gets in the way.

ewencp · 2018-02-26T23:26:14Z

@joel-hamill I pushed some build integration and styling stuff to make it work with the rest of the site. Remaining checklist that I'm aware of:

Tooling

Document what needs to be installed somewhere, provide requirements.txt at least since this would be painful to automate in gradle.
Ideally get MINOR: Extend release.py with a subcommand for staging docs into the kafka-site repo #3917 merged as testing any of this against kafka-site is incredibly painful without it. I've been using a copy of the script in that PR in order to iterate on testing this.
Handle autogenerated content like metrics. It looks like right now you may have just removed where these were included? If the html includes work well enough we might want to leave the full switch to rst as a follow up just so we can get the first steps done.
Figure out any redirects we now need or need modified, which may partly be in kafka-site's htaccess and may need to be more version-specific now
Replace the front-end replaced variables like fullDotVersion and scalaVersion. Given where we were using it before we'll need this to work in other contexts that sphinx doesn't normally support it. We also need to take care to handle some cases that are currently auto converted and show up with %7D and %7B instead of {{ and }}
Fix the makeLink links for streams. Currently they do no handle nested content because of the way they use relative URLs.
Ideally turn sphinx-build warnings into errors.

Content

Restore protocol.html. This was not directly referenced anywhere, but I think we still want to maintain it. (I think this is mostly content now; there's some import of autogenerated content, but I think I've resolved that process.)
Fix sphinx build warnings, especially the ones for missing images and invalid references. (If any of these aren't content-based, add items in the tooling section for the specific issues.)
Restore missing upgrade section (and possibly some other missing intro sections?). For example, there are links to streams upgrade notes missing.
Check links and table formatting for tables, especially in streams, e.g. notice that dsl-api has broken links and extra spaces in some cases because of line wrapping. We may need to investigate a better table solution for cases where we have very large blocks of text/long rst commands inside them. (This might end up needing a tooling solution; tables seem... painful for the large content we tend to house in them.)
Ensure there's no overlap/confusion in the new nested. For example, I see documentation/streams.rst, but documentation/streams/index.rst may not strictly collide, but if we want urls with /streams to not conflict with /streams/ (which commonly would resolve to /streams/index.html, including with our setup), we want to be careful about potential naming conflicts (and compatibility challenges) like this.
Top-level organization under /documentation/? Weird top-level of [quickstart, uses, documentation/index] toc that's thin and a weird docs intro compared to current approach of "dump everything immediately". Flatten with content under documentation/?
I think https://github.com/apache/kafka/blob/trunk/docs/streams/developer-guide/memory-mgmt.html and https://github.com/apache/kafka/blob/trunk/docs/streams/developer-guide/testing.html are missing equivalents. Right now since the structure of streams docs is basically the same we should be able to rely on the same file names instead of needing redirects.
Update to current content -- merge conflicts.

Styling

Worst case, we can live with some styling issues for a time, but we can at least track the issues here:

Do something about styling of the sphinx-generated tables. raw-included tables have the right classes, but auto-generated ones from sphinx have a different class on them that causes styling to be different (and ugly).

Some questions for you:

I ended up leaving the nav as it is today. This was mostly because of styling -- I don't think it's a huge deal to do it, I mostly had it, but we basically have to figure out how to map the elements generated by sphinx to the ones we already have styling for (or get the style to generate a different format which is probably just messing with some templates, but will take me awhile to figure out the details of). We can get nested levels of navs if we end up wanting them, but it was easier to leave as is for now since it mostly matches the second level anyway so we'd only be getting the value out of deeper nesting.
I stripped breadcrumbs and left them as a TODO since we don't have existing styling. Do we think these are useful and if so, do we need them now or can leave as future improvement.
Should we turn on strict builds (warnings are errors)? I see quite a few warnings and it seems easier to just enforce this now. Ideally it would then also be enforced as part of every PR (we'd have to check what commands run on Jenkins but it would basically be free). We'd obviously need to clean up the existing warnings before turning it on.
For pages like connect, do you want to preemptively put them into a directory with /index.html now? If we do that then we'll ideally end up with fewer redirects to maintain in the future. We'll still be able to link those ones without the index.html (e.g. http://kafka.apache.org/documentation/connect/).
How do we want to handle existing links to element IDs, especially ones like those used in the nav (but all of which may linked to from other sites), e.g. http://kafka.apache.org/documentation/#connect. Best I can think of is to get the full list based on the TOC, and then do redirects, either in JS or in the .htaccess.

guozhangwang · 2018-02-28T02:47:26Z

-  </div>
-
-    <div class="section" id="interactive-queries">
+


Do we still need these html files?

guozhangwang · 2018-02-28T02:49:07Z

+# The short X.Y version.
+version = '4.0'
+# The full version, including alpha/beta/rc tags.
+release = '4.0.0'


Some of the version macros like this are not needed in the AK site.

guozhangwang · 2018-02-28T02:50:50Z

+------------------------------------------------------------
+
+`Download <https://www.apache.org/dyn/closer.cgi?path=/kafka/%7B%7BfullDotVersion%7D%7D/kafka_%7B%7BscalaVersion%7D%7D-%7B%7BfullDotVersion%7D%7D.tgz>`__
+the {{fullDotVersion}} release and un-tar it. Note that there are


This macros like {{fullDotVersion}} and {{dotVersion}} are only for html. And should be modified in rst.

@ewencp ? ^

guozhangwang · 2018-02-28T02:51:04Z

+
+
+`Download <https://www.apache.org/dyn/closer.cgi?path=/kafka/%7B%7BfullDotVersion%7D%7D/kafka_2.11-%7B%7BfullDotVersion%7D%7D.tgz>`__
+the {{fullDotVersion}} release and un-tar it.


Ditto here.

guozhangwang · 2018-02-28T02:51:35Z

+        # add link prefix per link macro
+        if role == 'kafka-file':
+            linkPrefix = 'https://github.com/apache/kafka/blob/' + kafka_version + '/'
+        elif role == 'cp-javadoc':


Is this needed?

guozhangwang · 2018-02-28T02:52:45Z

+    <dependency>
+        <groupId>org.apache.kafka</groupId>
+        <artifactId>kafka-clients</artifactId>
+        <version>1.0.0-cp1</version>


cp1 should be removed.

guozhangwang · 2018-02-28T02:56:40Z

+        return [node], []
+    # add link macros
+    app.add_role('kafka-file', makeLink)
+    app.add_role('cp-javadoc', makeLink)\


remove this?

guozhangwang · 2018-02-28T02:58:20Z

+            mvn archetype:generate \
+                -DarchetypeGroupId=org.apache.kafka \
+                -DarchetypeArtifactId=streams-quickstart-java \
+                -DarchetypeVersion={{fullDotVersion}} \


guozhangwang · 2018-02-28T02:59:48Z

+   consumers), other brokers and tools, using either SSL or SASL. Kafka
+   supports the following SASL mechanisms:
+
+   -  SASL/GSSAPI (Kerberos) - starting at version 0.9.0.0


We should use macros for version numbers here.

@guozhangwang similar to https://github.com/apache/kafka/pull/4536/files#diff-85987f48f1258d9ee486e3191495582dR109 ?

I don't think this is correct, these are "starting at version" notes, so an absolute, fixed version is correct here.

guozhangwang · 2018-02-28T03:00:54Z

+| Group   | Artifact   | Versi | Description                          |
+| ID      | ID         | on    |                                      |
+=========+============+=======+======================================+
+| ``org.a | ``kafka-st | ``1.0 | (Required) Base library for Kafka    |


Version numbers be templated with macros.

…to migrate-html-to-rst

guozhangwang · 2018-03-01T00:09:54Z

+copyright = u'2018, Apache Software Foundation'
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the


Yup, we should replace the {{fullVersion}} etc macros with this. Just to keep in mind that in AK we also have other macros so we should also add them here, for example:

major_version_prefix = '0.11' doc_version = '0110'

Full set I can find is per https://github.com/apache/kafka/blob/trunk/docs/js/templateData.js#L19-L24. Afaict, these are all addressed. This actually doesn't address the doc_version you mentioned, but updated PR includes this as doc_url_version, which is only currently used for javadocs links. I actually don't see anywhere something like major_version_prefix is used -- if there's a case you're aware of can you point me towards it?

…support.

…encies to be version-agnostic

ewencp · 2018-03-28T04:21:32Z

@joel-hamill @guozhangwang Made a bunch of updates today to get us closer to functional/committable. Includes lots of sphinx customization to make it more usable for us.

Most importantly, have been updating my last comment with the TODO list with stuff I've found along the way: #4536 (comment). I've categorized stuff to help us split up work that can be tackled easily by different folks: tooling stuff is probably either me or somebody else familiar with sphinx (though really any developer); content could be handled by anyone -- @joel-hamill, streams developers, tools devs, etc --; and styling requires some CSS combined with understanding/running the docs build & kafka apache httpd setup.

I'm trying to add only things that I'd view as blockers to that list -- the stuff below the fold on that comment is stuff we decided we could follow up on later, even if it's technically a regression (generally just a styling/experience thing) from the current state since we'll be in a more reasonable, sustainable, state.

To summarize my view of current state:

Tooling: I think mostly in good shape. Redirects are still a bit scary, but I think it's the last thing to sort out. Tables and their formatting seem the other messiest thing that might bubble up. If there are bright ideas on how to handle that, let me know as we've already done a couple of custom extensions.
Content: Regressions in table formatting / links is biggest concern. Missing sections that need porting are next highest. Generally seems like it shouldn't be a problem, just some work porting/reorganizing content.
Styling: the least of my concerns. I think this should be easily fixable, just not important enough to investigate quite yet.

joel-hamill · 2018-03-29T16:25:16Z

@ewencp re: Content: Regressions in table formatting / links is biggest concern... - do you have a list of the specific items? should they be fixed in this PR, or a follow-on?

ewencp · 2018-03-29T16:47:52Z

@joel-hamill One example is https://github.com/apache/kafka/pull/4536/files#diff-ce1711dfc45980e53ccabbba7ef3ee4bR1160 where we have both the left column and right column wrapped in a way that either makes formatting weird (extra spaces) or breaks the embedded rst directives (the run, but due to the wrapping they generate incorrect links).

It seems like the built-in table functionality for rst doesn't handle "large" content well. Like the other customizations I added, we might need to consider an extension to make this more reasonable. Don't know if you have thoughts about good ways to handle tables like these. This is one case where HTML actually seems to shine -- they're not necessarily easy to understand quickly and visually from the html, but they are sane to write & maintain.

joel-hamill added 2 commits February 2, 2018 16:58

[WIP] Convert AK docs from .html to .rst

2ed1cc6

WIP

f3c1a1c

joel-hamill added 2 commits February 7, 2018 08:43

Add _build to gitignore

b4d007d

remove build dir

a7ad2ee

guozhangwang added the docs label Feb 8, 2018

[WIP] Convert AK docs from .html to .rst

dd929b5

WIP Add _build to gitignore remove build dir WIP

joel-hamill force-pushed the migrate-html-to-rst branch from f290a73 to dd929b5 Compare February 8, 2018 21:40

ewencp added 3 commits February 26, 2018 11:23

conf.py cleanup

04ad2df

Adjust site-docs jar to be built using new docs Makefile

3899694

Override docs theme styling

7fb6bd1

ewencp reviewed Feb 26, 2018

View reviewed changes

Merge changes from Ewen

3bccf9b

joel-hamill force-pushed the migrate-html-to-rst branch from f12505e to 3bccf9b Compare February 28, 2018 01:48

guozhangwang reviewed Feb 28, 2018

View reviewed changes

joel-hamill and others added 5 commits February 28, 2018 09:10

Remove dup topic

50c2dcf

Don't include Sphinx info

03dccc0

Remove legacy HTML

a565a4f

Merge branch 'migrate-html-to-rst' of github.com:joel-hamill/kafka in…

1a99070

…to migrate-html-to-rst

Rebase trunk

d37b4b5

guozhangwang reviewed Mar 1, 2018

View reviewed changes

ewencp added 6 commits March 26, 2018 21:49

Add requirements, virtualenv instructions, and release.py virtualenv …

6115285

…support.

Fix inclusion of generated docs tables

bee3ee9

Replace handlebars variables with sphinx substitution

c5ba77d

Replace javadoc links based on static URLs

1925b05

Fix makeLink roles for streams to work with non-relative layout

fc9198f

Add support for literals with substitutions and update streams depend…

1a8fe51

…encies to be version-agnostic

guozhangwang mentioned this pull request Apr 16, 2018

KAFKA-6670: Implement a Scala wrapper library for Kafka Streams #4756

Closed

3 tasks

Reorganize docs so we don't have multi-level toc for top-level docs

f0563a6

joel-hamill closed this Aug 4, 2021



		`Download <https://www.apache.org/dyn/closer.cgi?path=/kafka/%7B%7BfullDotVersion%7D%7D/kafka_2.11-%7B%7BfullDotVersion%7D%7D.tgz>`__
		the {{fullDotVersion}} release and un-tar it.

Uh oh!

Conversation

joel-hamill commented Feb 6, 2018

Uh oh!

ewencp commented Feb 7, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joel-hamill Feb 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ewencp commented Feb 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tooling

Content

Styling

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ewencp commented Mar 28, 2018

Uh oh!

joel-hamill commented Mar 29, 2018

Uh oh!

ewencp commented Mar 29, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

joel-hamill Feb 28, 2018 •

edited

Loading

ewencp commented Feb 26, 2018 •

edited

Loading