Docs: lint markdown files in site build #13977

manuzhang · 2025-09-02T15:56:18Z

In this PR, I add a step check_markdown_files in make build. The step scans markdown files in following paths with Pymarkdown Linter with pymarkdown scan.

site/docs/docs/nightly/docs/*.md (symlinked to versioned docs in docs/docs/*.md)
site/docs/*.md (site docs)
README.md

I selectively enabled the following rules through config file markdownlint.yml, which I believe are good to apply and can be applied automatically with pymarkdown fix command (see below).

list-indent
no-trailing-spaces
no-hard-tabs
no-multiple-blanks
no-multiple-space-atx
no-multiple-space-closed-atx
heading-start-left
no-multiple-space-blockquote
list-marker-space
hr-style
no-space-in-emphasis
no-space-in-code
no-space-in-links
proper-names
single-trailing-newline
code-fence-style

Issues found during scan can be fixed with dev/lint.sh --fix which actually runs pymarkdown fix to apply the rules.

manuzhang · 2025-09-13T09:41:45Z

docs/docs/aws.md

 - limitations under the License.
 -->
- 
+


MD009: Trailing spaces

docs/docs/aws.md

manuzhang · 2025-09-13T09:43:55Z

docs/docs/aws.md


 ### Amazon Data Firehose
-You can use [Firehose](https://docs.aws.amazon.com/firehose/latest/dev/apache-iceberg-destination.html) to directly deliver streaming data to Apache Iceberg Tables in Amazon S3. With this feature, you can route records from a single stream into different Apache Iceberg Tables, and automatically apply insert, update, and delete operations to records in the Apache Iceberg Tables. This feature requires using the AWS Glue Data Catalog.
+You can use [Firehose](https://docs.aws.amazon.com/firehose/latest/dev/apache-iceberg-destination.html) to directly deliver streaming data to Apache Iceberg Tables in Amazon S3. With this feature, you can route records from a single stream into different Apache Iceberg Tables, and automatically apply insert, update, and delete operations to records in the Apache Iceberg Tables. This feature requires using the AWS Glue Data Catalog.


MD047: Each file should end with a single newline character.

manuzhang · 2025-09-13T09:55:00Z

docs/docs/evolution.md

-2.  Dropping a column or field does not change the values in any other column.
-3.  Updating a column or field does not change values in any other column.
-4.  Changing the order of columns or fields in a struct does not change the values associated with a column or field name.
+1. Added columns never read existing values from another column.


MD030: Spaces after list markers

manuzhang · 2025-09-13T09:58:50Z

docs/docs/flink-ddl.md

 ## DDL commands

-###  `CREATE Catalog`
+### `CREATE Catalog`


MD019: Multiple spaces are present after hash character on Atx Heading.

manuzhang · 2025-09-15T01:50:52Z

@kevinjqliu May I get some early review from you?

kevinjqliu

Thanks for working on this! Looks like we have to make a lot of changes.

WDYT about incrementally fixing the .md files to make this easier to review? We can add the site/ changes first and lint more files in sequent PRs.

site/dev/common.sh

site/docs/community.md

site/docs/releases.md

kevinjqliu · 2025-09-22T15:57:38Z

#14154

this might be helpful for development :)

manuzhang · 2025-10-13T04:01:02Z

docs/docs/branching.md

-1	a	1.0
-2	b	2.0
-3	c	3.0
+1   a   1.0


MD010: Hard Tabs

manuzhang · 2025-10-13T06:05:50Z

format/gcm-stream-spec.md

- `Magic` is four bytes 0x41, 0x47, 0x53, 0x31 ("AGS1", short for: AES GCM Stream, version 1)
- `BlockLength` is four bytes (little endian) integer keeping the length of the equal-size split blocks before encryption. The length is specified in bytes.
- `CipherBlockᵢ` is the i-th enciphered block in the file, with the structure defined below.
+* `Magic` is four bytes 0x41, 0x47, 0x53, 0x31 ("AGS1", short for: AES GCM Stream, version 1)


MD004: Inconsistent Unordered List Start style.

manuzhang · 2025-10-14T09:43:05Z

@kevinjqliu @nastra This is ready for another round of review now. PTAL. Thanks!

site/dev/common.sh

nastra

LGTM but I think we should also set up a CI action so that this runs as part of the build. Also please document how to check/fix markdown files in this section here:

iceberg/README.md

Lines 54 to 59 in fc88614

    
           Iceberg is built using Gradle with Java 11, 17, or 21. 
        
           * To invoke a build and run tests: `./gradlew build` 
        
           * To skip tests: `./gradlew build -x test -x integrationTest` 
        
           * To fix code style for default versions: `./gradlew spotlessApply` 
        
           * To fix code style for all versions of Spark/Hive/Flink:`./gradlew spotlessApply -DallModules`

manuzhang · 2025-10-15T15:59:09Z

@nastra We already have a Docs Build CI which will run check_markdown_files. I've updated the instruction in the site/README.md

nastra · 2025-10-16T08:15:50Z

@nastra We already have a Docs Build CI which will run check_markdown_files.

@manuzhang I don't see this being called anywhere. Could you please point to the file that runs this when CI runs?

site/dev/setup_env.sh

manuzhang · 2025-10-20T03:24:59Z

@kevinjqliu @Fokko could you please take another look?

kevinjqliu

LGTM, i rendered the site locally and spot checked most of the markdown files.

small nit about restructuring the check_markdown_files and fix_markdown_files command. Instead of calling check_markdown_files in site/dev/setup_env.sh, i would prefer to only lint when make lint is called.

we can add make lint to CI too

iceberg/.github/workflows/docs-ci.yml

Lines 28 to 38 in 68e555b

    
           jobs: 
        
             build-docs: 
        
               runs-on: ubuntu-latest 
        
               steps: 
        
                 - uses: actions/checkout@v4 
        
                 - uses: actions/setup-python@v5 
        
                   with: 
        
                     python-version: 3.x 
        
                 - name: Build Iceberg documentation 
        
                   run: make build 
        
                   working-directory: ./site

site/dev/lint.sh

site/dev/setup_env.sh

manuzhang · 2025-10-27T07:31:37Z

@kevinjqliu @nastra Now dev/lint.sh has two modes.

The default mode dev/lint.sh can be run alone, and in dev/build.sh and dev/serve.sh to check markdown files for style issues. The fix mode dev/lint.sh --fix can be run to fix those files.

kevinjqliu

LGTM

have a minor nit but we can address as a follow up

kevinjqliu · 2025-10-27T16:18:10Z

site/dev/common.sh

+    echo "Markdown style issues found. Please run './dev/lint.sh --fix' to fix them."
+    exit 1


nit: was there a make lint command before? i think thats pretty helpful so that i can call like so

make lint make lint --fix

That doesn't work since --fix will be parsed as Make options. On the other hand, I think dev/lint.sh and dev/lint.sh --fix are already easy to call.

kevinjqliu · 2025-10-27T16:59:42Z

doing some sanity checks, i added trailing whitespaces to site/docs/status.md and docs/docs/api.md
running ./dev/lint.sh and ./dev/lint.sh --fix doesnt affect the changed files.

i'd expect the whitespaces to be fixed

EDIT: adding back the changes in docs/docs/api.md and running ./dev/lint.sh worked

kevinjqliu · 2025-10-27T19:58:23Z

I spot checked many of the markdown flies locally. LGTM

kevinjqliu · 2025-10-27T19:58:47Z

Thank you @manuzhang for adding this and thanks @nastra for the review

github-actions bot added the docs label Sep 2, 2025

manuzhang force-pushed the lint_doc_files branch from 06484cf to f88ac9c Compare September 2, 2025 16:00

github-actions bot added the INFRA label Sep 2, 2025

manuzhang force-pushed the lint_doc_files branch from f88ac9c to 15c353d Compare September 2, 2025 16:04

manuzhang force-pushed the lint_doc_files branch from 15c353d to 99421e9 Compare September 12, 2025 15:34

github-actions bot added the Specification Issues that may introduce spec changes. label Sep 12, 2025

manuzhang force-pushed the lint_doc_files branch from 99421e9 to 256ff71 Compare September 12, 2025 15:39

github-actions bot removed the INFRA label Sep 12, 2025

manuzhang commented Sep 13, 2025

View reviewed changes

docs/docs/aws.md

- limitations under the License.

-->

Copy link

Member Author

manuzhang Sep 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MD009: Trailing spaces

manuzhang commented Sep 13, 2025

View reviewed changes

docs/docs/aws.md Outdated Show resolved Hide resolved

manuzhang commented Sep 13, 2025

View reviewed changes

manuzhang force-pushed the lint_doc_files branch 2 times, most recently from f326676 to b6b97fd Compare September 13, 2025 14:42

manuzhang marked this pull request as ready for review September 14, 2025 15:53

manuzhang force-pushed the lint_doc_files branch from b6b97fd to 05622c1 Compare September 14, 2025 15:54

manuzhang requested a review from kevinjqliu September 15, 2025 01:50

kevinjqliu reviewed Sep 15, 2025

View reviewed changes

site/dev/common.sh Outdated Show resolved Hide resolved

kevinjqliu reviewed Sep 19, 2025

View reviewed changes

site/dev/common.sh Outdated Show resolved Hide resolved

nastra reviewed Sep 22, 2025

View reviewed changes

site/docs/community.md Outdated Show resolved Hide resolved

nastra reviewed Sep 22, 2025

View reviewed changes

site/docs/releases.md Outdated Show resolved Hide resolved

manuzhang force-pushed the lint_doc_files branch from 05622c1 to 28b8e7a Compare September 29, 2025 16:04

manuzhang commented Oct 13, 2025

View reviewed changes

docs/docs/branching.md

1 a 1.0

2 b 2.0

3 c 3.0

1 a 1.0

Copy link

Member Author

manuzhang Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MD010: Hard Tabs

manuzhang force-pushed the lint_doc_files branch from 28b8e7a to c84704d Compare October 13, 2025 05:58

manuzhang commented Oct 13, 2025

View reviewed changes

manuzhang force-pushed the lint_doc_files branch 2 times, most recently from 54288c0 to 715ea1c Compare October 13, 2025 06:25

manuzhang force-pushed the lint_doc_files branch from 715ea1c to db6d69a Compare October 13, 2025 06:37

manuzhang requested review from kevinjqliu and nastra October 13, 2025 06:52

nastra reviewed Oct 14, 2025

View reviewed changes

site/dev/common.sh Outdated Show resolved Hide resolved

nastra approved these changes Oct 14, 2025

View reviewed changes

manuzhang force-pushed the lint_doc_files branch from db6d69a to a35c2fa Compare October 15, 2025 15:55

manuzhang commented Oct 16, 2025

View reviewed changes

site/dev/setup_env.sh Outdated Show resolved Hide resolved

nastra approved these changes Oct 16, 2025

View reviewed changes

nastra requested a review from Fokko October 16, 2025 09:52

manuzhang force-pushed the lint_doc_files branch from a35c2fa to 2e6094c Compare October 20, 2025 03:10

kevinjqliu approved these changes Oct 26, 2025

View reviewed changes

site/dev/lint.sh Show resolved Hide resolved

site/dev/setup_env.sh Outdated Show resolved Hide resolved

site/dev/setup_env.sh Outdated Show resolved Hide resolved

site/dev/setup_env.sh Outdated Show resolved Hide resolved

Docs: lint markdown files in site build

3a4c37f

manuzhang force-pushed the lint_doc_files branch from 2e6094c to 3a4c37f Compare October 27, 2025 07:27

kevinjqliu approved these changes Oct 27, 2025

View reviewed changes

kevinjqliu merged commit 6a0d4a0 into apache:main Oct 27, 2025
3 checks passed

kevinjqliu mentioned this pull request Oct 27, 2025

site: use virtualenv and add make lint #14428

Merged

-	a	1.0
-	b	2.0
-	c	3.0
+a   1.0

	Iceberg is built using Gradle with Java 11, 17, or 21.

	* To invoke a build and run tests: `./gradlew build`
	* To skip tests: `./gradlew build -x test -x integrationTest`
	* To fix code style for default versions: `./gradlew spotlessApply`
	* To fix code style for all versions of Spark/Hive/Flink:`./gradlew spotlessApply -DallModules`

	jobs:
	build-docs:
	runs-on: ubuntu-latest
	steps:
	- uses: actions/checkout@v4
	- uses: actions/setup-python@v5
	with:
	python-version: 3.x
	- name: Build Iceberg documentation
	run: make build
	working-directory: ./site

		echo "Markdown style issues found. Please run './dev/lint.sh --fix' to fix them."
		exit 1

Docs: lint markdown files in site build #13977

Docs: lint markdown files in site build #13977

Uh oh!

Conversation

manuzhang commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manuzhang commented Sep 15, 2025

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevinjqliu commented Sep 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manuzhang commented Oct 14, 2025

Uh oh!

Uh oh!

nastra left a comment

Choose a reason for hiding this comment

Uh oh!

manuzhang commented Oct 15, 2025

Uh oh!

nastra commented Oct 16, 2025

Uh oh!

Uh oh!

manuzhang commented Oct 20, 2025

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

manuzhang commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinjqliu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinjqliu commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinjqliu commented Oct 27, 2025

Uh oh!

Uh oh!

kevinjqliu commented Oct 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

manuzhang commented Sep 2, 2025 •

edited

Loading

manuzhang commented Oct 27, 2025 •

edited

Loading

kevinjqliu commented Oct 27, 2025 •

edited

Loading