Skip to content

Conversation

@jojochuang
Copy link
Contributor

HDDS-13050. Update StartFromDockerHub.md.

What changes were proposed in this pull request?

Generated-by: Copilot Agent (Preview), prompt:

1. The section "Ozone in a Single Container" is no longer valid with Ozone 2.0 and above. Remove it.
2. Replace the section "Local multi-container cluster" with the following:
Obtain the Docker Compose Configuration First, obtain Ozone's sample Docker Compose configuration:
Download the latest Docker Compose configuration file
curl -O https://raw.githubusercontent.com/apache/ozone-docker/refs/heads/latest/docker-compose.yaml
Start the Cluster Start your Ozone cluster with three Datanodes using the following command:
docker compose up -d --scale datanode=3
This command will:
Automatically pull required images from Docker Hub Create a multi-node cluster with the core Ozone services Start all components in detached mode Verify the Deployment Check the status of your Ozone cluster components:
docker compose ps
You should see output similar to this:
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS docker-datanode-1 apache/ozone:1.4.1-rocky "/usr/local/bin/dumb…" datanode 14 seconds ago Up 13 seconds 0.0.0.0:32958->9864/tcp, :::32958->9864/tcp docker-datanode-2 apache/ozone:1.4.1-rocky "/usr/local/bin/dumb…" datanode 14 seconds ago Up 13 seconds 0.0.0.0:32957->9864/tcp, :::32957->9864/tcp docker-datanode-3 apache/ozone:1.4.1-rocky "/usr/local/bin/dumb…" datanode 14 seconds ago Up 12 seconds 0.0.0.0:32959->9864/tcp, :::32959->9864/tcp docker-om-1 apache/ozone:1.4.1-rocky "/usr/local/bin/dumb…" om 14 seconds ago Up 13 seconds 0.0.0.0:9874->9874/tcp, :::9874->9874/tcp docker-recon-1 apache/ozone:1.4.1-rocky "/usr/local/bin/dumb…" recon 14 seconds ago Up 13 seconds 0.0.0.0:9888->9888/tcp, :::9888->9888/tcp docker-s3g-1 apache/ozone:1.4.1-rocky "/usr/local/bin/dumb…" s3g 14 seconds ago Up 13 seconds 0.0.0.0:9878->9878/tcp, :::9878->9878/tcp docker-scm-1 apache/ozone:1.4.1-rocky "/usr/local/bin/dumb…" scm 14 seconds ago Up 13 seconds 0.0.0.0:9876->9876/tcp, :::9876->9876/tcp
Check the Ozone version
docker compose exec om bash ozone version
Access the Ozone Recon server, which provides monitoring and management capabilities by navigating to the Recon server home page.
Configuration You can customize your Ozone deployment by modifying the configuration parameters in the docker-compose.yaml file:
Common Configurations: Located under the x-common-config section Service-Specific Settings: Found under the environment section of individual services As an example, to update the port on which Recon listens to, append the following configuration:
x-common-config: ... OZONE-SITE.XML_ozone.recon.http-address: 0.0.0.0:9090
Refer to the Configuring Ozone For Production page for more configuration guidelines.
Next Steps Now that your Ozone cluster is up and running, you can enter any container and explore the environment.
docker compose exec om bash

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13050

How was this patch tested?

User doc

@jojochuang jojochuang requested review from Copilot and peterxcli June 8, 2025 20:39
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the StartFromDockerHub.md guide to remove outdated single-container instructions and introduce a Docker Compose–based multi-container setup for Ozone 2.0 and above.

  • Removed the obsolete “Ozone in a Single Container” section
  • Added steps to obtain and use the sample docker-compose.yaml for a three-node cluster
  • Included verification commands, configuration tips, and S3 client setup examples
Comments suppressed due to low confidence (2)

hadoop-hdds/docs/content/start/StartFromDockerHub.md:60

  • This example block isn’t using the proper Markdown fence. Preface the output with a standalone bash line, then list each container line below, and close with .
```docker-datanode-1   apache/ozone:2.0.0   "/usr/local/bin/dumb…"   datanode   14 seconds ago   Up 13 seconds   0.0.0.0:32958->9864/tcp, :::32958->9864/tcp

hadoop-hdds/docs/content/start/StartFromDockerHub.md:39

  • [nitpick] Inconsistent casing: consider using DataNodes (camel case) to match other occurrences of DataNode in the document.
Start your Ozone cluster with three Datanodes using the following command:

```bash
docker run -p 9878:9878 -p 9876:9876 apache/ozone
# Download the latest Docker Compose configuration file
curl -O https://raw.githubusercontent.com/apache/ozone-docker/refs/heads/latest/docker-compose.yaml
Copy link

Copilot AI Jun 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The URL may not resolve correctly using refs/heads. Use the standard raw GitHub path, e.g., https://raw.githubusercontent.com/apache/ozone-docker/latest/docker-compose.yaml.

Suggested change
curl -O https://raw.githubusercontent.com/apache/ozone-docker/refs/heads/latest/docker-compose.yaml
curl -O https://raw.githubusercontent.com/apache/ozone-docker/latest/docker-compose.yaml

Copilot uses AI. Check for mistakes.
Copy link
Contributor

@chungen0126 chungen0126 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jojochuang for working on this. Left some comments.

@jojochuang jojochuang added the documentation Improvements or additions to documentation label Jun 9, 2025
jojochuang and others added 9 commits June 10, 2025 21:47
Change-Id: I109ba57650014c7f8ca2f4274fa2effaba97de27
Change-Id: Id6c055e69d9690d531e1cfe143afad4e55a49f52
Change-Id: I2b14fdcf67d2c72c1726856e12b9ec114d76b422
Change-Id: I5880971332b53971571f07b046effe4f14e95c7f
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Change-Id: I4049ecc815211d76b6919b1f95a15c6c1e8d5733
Change-Id: I578bf8c534e9a45480db24516b25f1350aa69b94
@jojochuang
Copy link
Contributor Author

Thanks @chungen0126. Updated.

Copy link
Contributor

@chungen0126 chungen0126 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM

Copy link
Member

@peterxcli peterxcli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jojochuang for this documentation! LGTM!

@peterxcli peterxcli merged commit 8c32363 into apache:master Jun 11, 2025
14 checks passed
@peterxcli
Copy link
Member

Thanks @jojochuang for this patch, and @chungen0126 for reviewing!

@peterxcli
Copy link
Member

peterxcli commented Jun 11, 2025

@adoroszlai I was thinking if we need to remove those ai co-authors from merge commit message?
image

@adoroszlai
Copy link
Contributor

I was thinking if we need to remove those ai co-authors from merge commit message?

I would treat it similar to human contributions. If a significant portion of the patch is generated by AI tools, then yes. If it's just a few minor review items, I wouldn't include it.

However, simple Co-authored-by may not be enough to capture AI contribution in the first case.

See also:
https://www.apache.org/legal/generative-tooling.html

jojochuang added a commit to jojochuang/ozone that referenced this pull request Jun 13, 2025
Co-authored-by: Copilot <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
sadanand48 pushed a commit to sadanand48/hadoop-ozone that referenced this pull request Jun 16, 2025
Co-authored-by: Copilot <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
aswinshakil added a commit to aswinshakil/ozone that referenced this pull request Jun 20, 2025
…239-container-reconciliation

Commits: 62

da53b5b HDDS-13299. Fix failures related to delete (apache#8665)
8c1b439 HDDS-13296. Integration check always passes due to missing output (apache#8662)
7329859 HDDS-13023. Container checksum is missing after container import (apache#8459)
a0af93e HDDS-13292. Change `<? extends KeyValue>` to `<KeyValue>` in test (apache#8657)
f3050cf HDDS-13276. Use KEY_ONLY/VALUE_ONLY iterator in SCM/Datanode. (apache#8638)
e9c0a45 HDDS-13262. Simplify key name validation (apache#8619)
f713e57 HDDS-12482. Avoid using CommonConfigurationKeys (apache#8647)
b574709 HDDS-12924. datanode used space calculation optimization (apache#8365)
de683aa HDDS-13263. Refactor DB Checkpoint Utilities. (apache#8620)
97262aa HDDS-13256. Updated OM Snapshot Grafana Dashboard to reflect metric updates from HDDS-13181. (apache#8639)
9d2b415 HDDS-13234. Expired secret key can abort leader OM startup. (apache#8601)
d9049a2 HDDS-13220. Change Recon 'Negative usedBytes' message loglevel to DEBUG (apache#8648)
6df3077 HDDS-9223. Use protobuf for SnapshotDiffJobCodec (apache#8503)
a7fc290 HDDS-13236. Change Table methods not to throw IOException. (apache#8645)
9958f5b HDDS-13287. Upgrade commons-beanutils to 1.11.0 due to CVE-2025-48734 (apache#8646)
48aefea HDDS-13277. [Docs] Native C/C++ Ozone clients (apache#8630)
052d912 HDDS-13037. Let container create command support STANDALONE , RATIS and EC containers (apache#8559)
90ed60b HDDS-13279. Skip verifying Apache Ranger binaries in CI (apache#8633)
9bc53b2 HDDS-11513. All deletion configurations should be configurable without restart (apache#8003)
ac511ac HDDS-13259. Deletion Progress - Grafana Dashboard (apache#8617)
3370f42 HDDS-13246. Change `<? extend KeyValue>` to `<KeyValue>` in hadoop-hdds (apache#8631)
7af8c44 HDDS-11454. Ranger integration for Docker Compose environment (apache#8575)
5a3e4e7 HDDS-13273. Bump awssdk to 2.31.63 (apache#8626)
77138b8 HDDS-13254. Change table iterator to optionally read key or value. (apache#8621)
ce288b6 HDDS-13265. Simplify the page Access Ozone using HTTPFS REST API (apache#8629)
36fe888 HDDS-13275. Improve CheckNative implementation (apache#8628)
d38484e HDDS-13274. Bump sqlite-jdbc to 3.50.1.0 (apache#8627)
3f3ec43 HDDS-13266. `ozone debug checknative` to show OpenSSL lib (apache#8623)
8983a63 HDDS-13272. Bump junit to 5.13.1 (apache#8625)
a927113 HDDS-13271. [Docs] Minor text updates, reference links. (apache#8624)
7e77058 HDDS-13112. [Docs] OM Bootstrap can also happen when follower falls behind too much. (apache#8600)
fd13300 HDDS-10775. Support bucket ownership verification (apache#8558)
3ecf345 HDDS-13207. [Docs] Third party systems compatible with Ozone S3. (apache#8584)
ad5a507 HDDS-13035. SnapshotDeletingService should hold write locks while purging deleted snapshots (apache#8554)
38a9186 HDDS-12637. Increase max buffer size for tar entry read/write (apache#8618)
f31c264 HDDS-13045. Implement Immediate Triggering of Heartbeat when Volume Full (apache#8590)
0701d6a HDDS-13248. Remove `ozone debug replicas verify` option --output-dir (apache#8612)
ca1afe8 HDDS-13257. Remove separate split for shell integration tests (apache#8616)
5d6fe94 HDDS-13216. Standardize Container[Replica]NotFoundException messages (apache#8599)
1e47217 HDDS-13168. Fix error response format in CheckUploadContentTypeFilter (apache#8614)
6d4d423 HDDS-13181. Added metrics for internal Snapshot Operations. (apache#8606)
4a461b2 HDDS-10490. Intermittent NPE in TestSnapshotDiffManager#testLoadJobsOnStartUp (apache#8596)
bf29f7f HDDS-13235. The equals/hashCode methods in anonymous KeyValue classes may not work. (apache#8607)
6ff3ad6 HDDS-12873. Improve ContainerData statistics synchronization. (apache#8305)
09d3b27 HDDS-13244. TestSnapshotDeletingServiceIntegrationTest should close snapshots after deleting them (apache#8611)
931bc2d HDDS-13243. copy-rename-maven-plugin version is missing (apache#8605)
3b5985c HDDS-13244. Disable TestSnapshotDeletingServiceIntegrationTest
6bf009c HDDS-12927. metrics and log to indicate datanode crossing disk limits (apache#8573)
752da2b HDDS-12760. Intermittent Timeout in testImportedContainerIsClosed (apache#8349)
8c32363 HDDS-13050. Update StartFromDockerHub.md. (apache#8586)
ba1887c HDDS-13241. Fix some potential resource leaks (apache#8602)
bbaf71e HDDS-13130. Rename all instances of Disk Usage to Namespace usage (apache#8571)
0628386 HDDS-13142. Correct SCMPerformanceMetrics for delete operation. (apache#8592)
516bc96 HDDS-13148. [Docs] Update Transparent Data Encryption doc. (apache#8530)
5787135 HDDS-13229. [Doc] Fix incorrect CLI argument order in OM upgrade docs (apache#8598)
ba95074 HDDS-13107. Support limiting output of `ozone admin datanode list` (apache#8595)
e7f5544 HDDS-13171. Replace pipelineID if nodes are changed (apache#8562)
3c9d4d8 HDDS-13103. Correct transaction metrics in SCMBlockDeletingService. (apache#8516)
f62eb8a HDDS-13160. Remove SnapshotDirectoryCleaningService and refactor AbstractDeletingService (apache#8547)
b46e6b2 HDDS-13150. Fixed SnapshotLimitCheck when failures occur. (apache#8532)
203c1d3 HDDS-13206. Update documentation for Apache Ranger (apache#8583)
2072ef0 HDDS-13214. populate-cache fails due to unused dependency (apache#8594)

Conflicts:
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingTask.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants