Config automation #1428
Config automation #1428
Conversation
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
📝 WalkthroughWalkthroughThis PR introduces infrastructure for automating property documentation generation. It adds a GitHub Actions workflow triggered on repository tags that compares version semantics, generates property documentation using doc-tools, and opens pull requests with the generated content. Supporting changes include modifying configuration files (antora.yml and local-antora-playbook.yml), converting inline property documentation to modular include directives across broker, object-storage, and topic property pages, adding autogenerated documentation partials for broker properties, topic properties, deprecated properties, and property mappings, and updating version-specific property changes in a JSON tracking file. Sequence DiagramsequenceDiagram
participant GitHub as GitHub Event
participant Workflow as update-property-docs<br/>Workflow
participant AWS as AWS Secrets<br/>Manager
participant DocTools as doc-tools
participant CreatePR as create-pull-request
participant Repo as Repository
GitHub->>Workflow: workflow_dispatch or<br/>repository_dispatch triggered
Workflow->>AWS: Fetch AWS credentials
AWS-->>Workflow: Credentials retrieved
Workflow->>Repo: Checkout & setup Node.js
Workflow->>Workflow: Extract tag from input<br/>or dispatch payload
Workflow->>Workflow: Compare tag to latest<br/>in antora.yml using semver
alt Tag is newer
rect rgb(200, 240, 200)
note right of Workflow: Generate Docs Path
Workflow->>DocTools: Generate property docs<br/>with specified options
DocTools-->>Workflow: Property docs generated
Workflow->>CreatePR: Create PR with generated<br/>docs & commit message
CreatePR-->>Repo: PR opened
end
else Tag not newer
rect rgb(240, 200, 200)
note right of Workflow: Skip Path
Workflow->>Workflow: Log skip notice
end
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes
Possibly related PRs
Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (2 inconclusive)
✅ Passed checks (2 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…docs into config-automation
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
.github/workflows/update-property-docs.yml (2)
62-83: Version comparison logic is sound but could benefit from input validation.The semver comparison correctly:
- Strips the leading 'v' prefix from both versions
- Uses
sort -Vfor semantic version ordering- Requires the new version to be strictly greater than the current version
However, the logic doesn't validate that the stripped version strings are valid semver. If malformed tags are provided,
sort -Vmay produce unexpected results. Consider adding a validation step before comparison.Consider adding validation:
# Example: validate semver format before comparison if ! [[ "$CUR_NUM" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9]+)?$ ]]; then echo "❌ Invalid semver format: $CUR_NUM" >&2 exit 1 fi
98-108: PR creation is well-configured with appropriate automation labels.The pull request creation step uses:
- Consistent commit message and branch naming with the tag
- Appropriate auto-docs label for filtering and tracking
- Clear PR title and body
One suggestion: include a link to the internal wiki with review instructions (as mentioned in DOC-1690 objectives). Update the body to include guidance.
Consider enhancing the PR body with review instructions:
body: | This PR auto-generates updated Redpanda property documentation for **${{ steps.tag.outputs.tag }}**. **Review instructions:** See [Property Documentation Review Guide](https://redpandadata.atlassian.net/wiki/spaces/DOC/pages/1396244485/Review+Redpanda+configuration+properties)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Jira integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (11)
.github/workflows/update-property-docs.yml(1 hunks)antora.yml(2 hunks)local-antora-playbook.yml(1 hunks)modules/reference/pages/properties/broker-properties.adoc(2 hunks)modules/reference/pages/properties/object-storage-properties.adoc(1 hunks)modules/reference/pages/properties/topic-properties.adoc(1 hunks)modules/reference/partials/deprecated/deprecated-properties.adoc(1 hunks)modules/reference/partials/properties/broker-properties.adoc(1 hunks)modules/reference/partials/properties/topic-properties.adoc(1 hunks)modules/reference/partials/topic-property-mappings.adoc(1 hunks)modules/reference/property-changes-v25.2.1-to-v25.2.10.json(1 hunks)
🧰 Additional context used
🧠 Learnings (5)
📚 Learning: 2025-07-16T19:33:20.420Z
Learnt from: Feediver1
Repo: redpanda-data/docs PR: 1153
File: modules/reference/pages/properties/topic-properties.adoc:45-50
Timestamp: 2025-07-16T19:33:20.420Z
Learning: In the Redpanda documentation, topic property cross-references like <<max.compaction.lag.ms>> and <<min.compaction.lag.ms>> require corresponding property definition sections with anchors like [[maxcompactionlagms]] and [[mincompactionlagms]] to prevent broken links.
Applied to files:
modules/reference/partials/deprecated/deprecated-properties.adocmodules/reference/partials/topic-property-mappings.adocmodules/reference/partials/properties/broker-properties.adocmodules/reference/pages/properties/topic-properties.adocmodules/reference/pages/properties/object-storage-properties.adocmodules/reference/pages/properties/broker-properties.adocmodules/reference/partials/properties/topic-properties.adoc
📚 Learning: 2025-07-14T19:28:43.296Z
Learnt from: Feediver1
Repo: redpanda-data/docs PR: 1153
File: antora.yml:3-5
Timestamp: 2025-07-14T19:28:43.296Z
Learning: In Redpanda docs, during beta releases, the version metadata may intentionally show inconsistencies where the header displays the beta version (e.g., 25.2 Beta) while internal attributes like full-version, latest-redpanda-tag, operator-beta-tag still reference the stable version (e.g., 25.1). This is resolved during the GA merge process when all version references are synchronized.
Applied to files:
antora.yml
📚 Learning: 2025-05-07T01:06:00.937Z
Learnt from: kbatuigas
Repo: redpanda-data/docs PR: 1113
File: modules/manage/partials/iceberg/use-iceberg-catalogs.adoc:100-107
Timestamp: 2025-05-07T01:06:00.937Z
Learning: In AsciiDoc documentation for Redpanda, the syntax `+` and `--` around content blocks within a `[tabs]` section are valid AsciiDoc formatting elements for tabbed content. The `+` after a tab name (like `rpk::`) indicates that the following block belongs to that tab, and the `--` markers enclose the content for that tab. These are not diff artifacts and should not be removed.
Applied to files:
antora.yml
📚 Learning: 2025-08-19T19:26:57.364Z
Learnt from: paulohtb6
Repo: redpanda-data/docs PR: 1324
File: modules/deploy/partials/high-availability.adoc:537-537
Timestamp: 2025-08-19T19:26:57.364Z
Learning: In AsciiDoc documentation, page aliases (`:page-aliases:`) should never be removed as they serve as old redirects to maintain backward compatibility when pages are moved or renamed. They preserve SEO and prevent broken links for users.
Applied to files:
antora.yml
📚 Learning: 2025-08-08T00:44:07.530Z
Learnt from: paulohtb6
Repo: redpanda-data/docs PR: 1298
File: modules/deploy/pages/deployment-option/self-hosted/manual/production/production-readiness.adoc:405-406
Timestamp: 2025-08-08T00:44:07.530Z
Learning: In AsciiDoc documentation, properties defined as level 3 headings (===) automatically generate anchors from the heading text. Manual [[anchor]] definitions are only needed for property names with special characters (like dots) or when customizing anchor names differs from the heading text.
Applied to files:
modules/reference/partials/properties/topic-properties.adoc
🔇 Additional comments (17)
local-antora-playbook.yml (1)
20-20: Config additions align with local build requirements.The branch filter for
cloud-docsandstart_pathsfordocs-sitefollow standard Antora conventions. Ensure that anyone running local builds is aware thatconfig-automationbranch for cloud-docs must exist for the playbook to work.Also applies to: 23-23
modules/reference/partials/deprecated/deprecated-properties.adoc (1)
1-106: Autogenerated deprecated properties list is well-structured.The organization by property type (Broker vs Cluster) and simple bullet-point format is appropriate for a reference list. The autogenerated header and override instructions provide clear guidance.
Note: Ensure the doc-tools generation continues to keep this file synchronized with the actual deprecated properties in the codebase, as this is now a versioned artifact.
modules/reference/property-changes-v25.2.1-to-v25.2.10.json (1)
1-50: Version tracking JSON is properly structured.The metadata and property change details are well-formatted, with summary counts matching the details array. The three properties documented show reasonable changes (type conversions and default updates).
modules/reference/partials/properties/topic-properties.adoc (2)
1-50: Topic properties documentation is well-organized and comprehensive.The file provides excellent coverage with:
- Logical categorization by functional area (retention, compaction, segments, schema registry, etc.)
- Consistent metadata for each property (type, accepted values, defaults, nullability)
- Helpful cross-references to related cluster properties
- Conditional content for cloud vs self-managed environments
- Related topics sections with relevant documentation links
571-586: Exclusion tags confirmed at specified locations.Verification confirms both properties are indeed tagged with
exclude-from-docs:
redpanda.remote.allowgapsat lines 571–586redpanda.virtual.cluster.idat lines 747–762The review comment's factual claims are accurate. However, determining whether this exclusion is intentional requires domain knowledge and design context beyond code inspection. Manual verification is needed to confirm these properties should remain excluded from published documentation.
antora.yml (3)
15-15: Latest Redpanda tag updated to v25.2.10.The
latest-redpanda-tagis updated from its previous value tov25.2.10, which aligns with the PR's objective of generating property docs for this version. This tag is used by the update-property-docs workflow for version comparison.
29-32: Version metadata and badge configuration updated.Supported versions for RHEL, Ubuntu are reformatted (removing unnecessary quotes), and the deprecated badge is updated with proper AsciiDoc xref syntax. These maintenance updates follow best practices.
6-6: Navigation file verified and exists.The
modules/ROOT/nav.adocfile referenced on line 6 is present in the repository. The documentation build will proceed without issues from this addition..github/workflows/update-property-docs.yml (3)
1-112: GitHub Actions workflow implements core automation requirements effectively.The workflow successfully:
- Triggers on both manual and repository dispatch events
- Securely handles AWS credentials and GitHub token
- Compares semantic versions to determine if doc generation is needed
- Conditionally generates docs and opens PRs to avoid unnecessary operations
- Uses consistent labeling and naming for tracking
The implementation aligns with the PR objectives for automating property documentation generation. Address the minor concerns about secret path formatting and property overrides file verification before merging.
32-34: No issues found—secret path format is correct.The leading comma in
,sdlc/prod/github/actions_bot_tokenon line 33 is intentional. The format ",secretId" is a blank alias that uses parsed-JSON keys or default naming, which aligns withparse-json-secrets: trueon line 34. This follows the documented behavior of theaws-actions/aws-secretsmanager-get-secretsaction.
85-96: All verification checks passed — the code is correct.
- ✓
docs-data/property-overrides.jsonexists and contains valid JSON- ✓
doc-toolsis properly installed via@redpanda-data/docs-extensions-and-macros(version ^4.0.0), made available to the workflow throughnpm ci- ✓
GITHUB_TOKENenvironment variable is correctly set from the AWS Secrets Manager secret (ACTIONS_BOT_TOKEN)The
--cloud-supportflag is used in the active workflow with no evidence of issues, and the--overridesoption is documented in the autogenerated property files, confirming the tool's support for these flags.modules/reference/partials/properties/broker-properties.adoc (1)
421-491: Data inconsistency detected in autogenerated properties documentation.The file uses category tags (category-redpanda, category-pandaproxy) to organize properties for targeted inclusion, resulting in intentional duplication. However,
cloud_storage_inventory_hash_path_directoryhas conflicting metadata between its two instances:
- Line 421:
*Nullable:* No- Line 458:
*Nullable:* YesSince this file is autogenerated (as noted in the header), verify:
- Whether the doc generation tool is producing inconsistent output
- Whether content differences exist in the other 8 duplicated properties beyond the Nullable field
- Whether the source configuration data has conflicts
modules/reference/partials/topic-property-mappings.adoc (1)
1-48: Fix broken topic property anchors in the auto-generated mappings table.All topic property cross-references in this file are broken. The anchors are formatted without dots/special characters (
<<cleanuppolicy>>) but the actual property headings intopic-properties.adocpreserve them (=== cleanup.policy), causing AsciiDoc to auto-generate anchors that don't match (e.g.,cleanup.policyinstead ofcleanuppolicy).The cluster property references work correctly because they consistently use underscores throughout. Topic property anchors must either:
- Add explicit
[[cleanuppolicy]]style anchors before each property heading intopic-properties.adoc, or- Update the mappings table to reference the auto-generated anchors that preserve dots (e.g.,
<<cleanup.policy>>instead of<<cleanuppolicy>>)This appears to be a bug in the doc-tools CLI auto-generation logic for topic properties.
⛔ Skipped due to learnings
Learnt from: Feediver1 Repo: redpanda-data/docs PR: 1153 File: modules/reference/pages/properties/topic-properties.adoc:45-50 Timestamp: 2025-07-16T19:33:20.420Z Learning: In the Redpanda documentation, topic property cross-references like <<max.compaction.lag.ms>> and <<min.compaction.lag.ms>> require corresponding property definition sections with anchors like [[maxcompactionlagms]] and [[mincompactionlagms]] to prevent broken links.Learnt from: JakeSCahill Repo: redpanda-data/docs PR: 1304 File: modules/troubleshoot/partials/errors-and-solutions.adoc:44-44 Timestamp: 2025-08-13T16:39:20.038Z Learning: In AsciiDoc documentation with Antora, explicit anchors like [[anchor-name]] take precedence over automatically generated anchors from headings. When troubleshooting cross-references, check for both explicit anchors and auto-generated ones from heading text (lowercase with spaces replaced by hyphens).Learnt from: paulohtb6 Repo: redpanda-data/docs PR: 1298 File: modules/deploy/pages/deployment-option/self-hosted/manual/production/production-readiness.adoc:405-406 Timestamp: 2025-08-08T00:44:07.530Z Learning: In AsciiDoc documentation, properties defined as level 3 headings (===) automatically generate anchors from the heading text. Manual [[anchor]] definitions are only needed for property names with special characters (like dots) or when customizing anchor names differs from the heading text.modules/reference/pages/properties/broker-properties.adoc (3)
49-67: Clarify ownership and maintenance of the HTTP-Based Authentication section.This new manual documentation section appears alongside autogenerated include directives. If this content is hand-written and not generated via the automation workflow, it risks diverging from autogenerated content and may become stale. Confirm whether:
- This section is maintained manually, or
- It should be included in the autogenerated partial with a specific tag (e.g.,
tags=http-based-auth)If manual, document the maintenance expectation to prevent drift during future automation runs.
86-92: Verify Schema Registry example YAML alignment with partial content.The Schema Registry example block (lines 86-92) has been modified and will appear alongside the autogenerated partial include on line 97. Ensure the example remains accurate and complementary to the properties listed in the partial. If the example is meant to be comprehensive, verify that all properties shown exist in the
category-schema-registrytagged section.
47-47: The original review comment is incorrect—all required category tags are present and properly configured.The initial verification script had a flawed search pattern. Actual inspection confirms:
- category-redpanda, category-schema-registry, and category-pandaproxy-client tags all exist in the partial file with proper
tag::andend::delimiters- Anchors are auto-generated from section headers (standard AsciiDoc behavior):
=== property_namecreates[[property_name]]anchors- Cross-references in the partial (e.g.,
<<admin_api_tls>>) resolve correctly to these auto-generated anchors- Manual content in the main file (HTTP-Based Authentication section) includes an explicit anchor
[[http_based_auth_method]]The include directives at lines 47, 97, and 131 will function as intended.
Likely an incorrect or invalid review comment.
modules/reference/pages/properties/object-storage-properties.adoc (1)
15-15: No issues found – the partial is properly structured with auto-generated anchors.The partial file exists at
modules/reference/partials/properties/object-storage-properties.adocand contains 3,148 lines of autogenerated content. Each property is defined as a level 3 heading (=== property_name), which automatically generates anchors in Antora/Asciidoctor. Cross-references within the partial (e.g.,<<cloud_storage_secret_key>>) correctly reference these auto-generated anchors. The include directive uses correct Antora syntax, and the tag filters are properly balanced (5 deprecated tags that will be excluded).
| // This content is autogenerated. Do not edit manually. To override descriptions, use the doc-tools CLI with the --overrides option: https://redpandadata.atlassian.net/wiki/spaces/DOC/pages/1396244485/Review+Redpanda+configuration+properties | ||
| // tag::category-redpanda[] | ||
| === admin | ||
|
|
||
| Network address for the glossterm:Admin API[] server. | ||
|
|
||
| ifndef::env-cloud[] | ||
| *Restored during xref:manage:whole-cluster-restore.adoc[Whole Cluster Restore]:* Yes | ||
| endif::[] | ||
|
|
||
| // tag::self-managed-only[] | ||
| *Visibility:* `user` | ||
| // end::self-managed-only[] | ||
|
|
||
| *Type:* array | ||
|
|
||
| ifdef::env-cloud[] | ||
| *Default:* Available in the Redpanda Cloud Console | ||
| endif::[] | ||
| ifndef::env-cloud[] | ||
| *Default:* `[{address: "127.0.0.1", port: 9644}]` | ||
| endif::[] | ||
|
|
||
| *Nullable:* No | ||
|
|
||
|
|
||
| .Example | ||
| [,yaml] | ||
| ---- | ||
| redpanda: | ||
| admin: | ||
| - name: <admin-api-name> | ||
| address: <external-broker-hostname> | ||
| port: <admin-api-port> | ||
| ---- | ||
|
|
||
| Replace the following placeholders with your values: | ||
|
|
||
| * `<admin-api-name>`: Name for the Admin API listener (TLS configuration is handled separately in the <<admin_api_tls,`admin_api_tls`>> broker property) | ||
| * `<external-broker-hostname>`: The externally accessible hostname or IP address that clients use to connect to this broker | ||
| * `<admin-api-port>`: The port number for the Admin API endpoint | ||
|
|
||
|
|
||
| --- | ||
| // end::category-redpanda[] | ||
|
|
||
| // tag::category-redpanda[] | ||
| === admin_api_doc_dir | ||
|
|
||
| Path to the API specifications for the Admin API. |
There was a problem hiding this comment.
Autogenerated file structure is comprehensive but requires deduplication.
The properties documentation is well-formatted with consistent metadata (type, defaults, visibility, examples, related topics). However, the duplicate entries must be resolved before merging. Beyond the duplicates, the file quality looks good:
- Clear category tagging aids organization
- Proper AsciiDoc formatting
- Helpful examples and cross-references
- Proper conditional formatting for cloud vs self-managed environments
Address the duplicate property definitions noted above before merging this PR.
🤖 Prompt for AI Agents
modules/reference/partials/properties/broker-properties.adoc lines 1-50: there
are duplicate property definitions (the "admin" property appears more than once)
which must be deduplicated before merging; locate both definitions, choose a
single canonical "admin" block that preserves full metadata (type, default,
nullable, visibility, example, and conditional cloud/self-managed sections),
merge any differing fields into that canonical block, remove the redundant
duplicate(s), keep the separate "admin_api_doc_dir" section intact, and verify
the autogenerated source or doc-tools overrides are updated so the duplicate
does not reappear.
| latest-redpanda-tag: v25.2.10 | ||
| latest-console-tag: v2.8.5 | ||
| latest-release-commit: 68c64fd | ||
| latest-operator-version: v2.3.8-24.3.6 | ||
| operator-beta-tag: v25.1.1-beta1 | ||
| helm-beta-tag: v25.1.1-beta1 |
Description
Resolves https://redpandadata.atlassian.net/browse/DOC-1690
This pull request introduces automated generation and updating of Redpanda property documentation and related reference files, along with some configuration and metadata updates. The main focus is on improving the accuracy and maintainability of property docs by using a GitHub Actions workflow and updating version references and property mappings.
Automated documentation and reference updates:
.github/workflows/update-property-docs.yml) to automatically generate property documentation, create pull requests for updates, and manage AWS credentials/secrets for secure automation.modules/reference/partials/deprecated/deprecated-properties.adoclists all deprecated configuration properties, andmodules/reference/partials/topic-property-mappings.adocmaps topic properties to their corresponding cluster properties. These files are now maintained via automation and should not be edited manually. [1] [2]modules/reference/property-changes-v25.2.1-to-v25.2.10.json) summarizing property changes between versions, including changed defaults and types.Configuration and metadata updates:
antora.ymlto reference the latest Redpanda release tag (v25.2.10), standardized YAML formatting, and improved home page and navigation metadata for clarity and consistency. [1] [2]local-antora-playbook.ymlto include new branches and start paths for documentation sources, ensuring the latest docs and labs content are included in local builds.Page previews
Checks