HDDS-13165. [Docs] Python client developer guide.

[jojochuang]

What changes were proposed in this pull request?

HDDS-13165. [Docs] Python client developer guide.

Please describe your PR in detail:

• Added interface/Python.md: overall Python client access introduction.
• Recipe: Access Ozone using PyArrow (Docker Quickstart)
• Recipe: Access Ozone using Boto3 (Docker Quickstart)
• Recipe: Access Ozone using HTTPFS REST API (Docker + Python Requests)

For interface/Python.md, the draft was generated using ChatGPT 4o using the prompt:

Create a user document in Markdown format for Python developers who want to access Apache Ozone. This document will be part of the Ozone Client Interfaces page: https://ozone.apache.org/docs/edge/interface.html.

📌 *Audience*: Python developers familiar with Python integration and Ozone. Skip the introduction.

📌 *Structure*:

Setup and Prerequisites:
Required libraries (PyArrow, Boto3, WebHDFS)
Required configurations (e.g., HADOOP_CONF_DIR, Ozone URIs, credentials, authentication)
Access Method 1: PyArrow with libhdfs
Setup steps (including any system paths or environment variables)
Python code sample (validate for correctness)
Access Method 2: Boto3 with Ozone S3 Gateway
Setup steps (including Ozone S3 endpoint format, bucket naming conventions, credentials)
Python code sample (validate for correctness)
Access Method 3: WebHDFS/HttpFS or REST API
Setup steps (including endpoint URL, authentication)
Python code sample (using requests or webhdfs)
Access from PySpark
Configuration settings in Spark (fs.ozone. settings)
Python code sample for reading/writing data to Ozone
Troubleshooting Tips
Common issues (e.g., authentication failures, connection errors)
Suggested debugging techniques
References and Further Resources
Links to official Ozone documentation, PyArrow, Boto3, WebHDFS, PySpark
📌 *Markdown Format*:

Use proper headers (##, ###) for each section.
Include Python syntax highlighting in code blocks (```python).
Use clear formatting and spacing for readability.
Include warnings or notes where appropriate (e.g., > *Note:*).
If applicable, include a simple diagram showing connection flows.
📌 *Quality Checks*:

Validate all code samples for correctness.
Ensure the document is clear and concise.
Focus only on actionable instructions and setup information.
Generate the complete Markdown document in response. Include a Hugo header. Include Apache License header

The PyArrow recipe draft was generated using ChatGPT 4o prompt:

I personally verified the following steps using Ozone's Docker image. Please rewrite in a user tutorial format.

PyArrow to access Ozone

# Download the latest Docker Compose configuration file
curl -O https://raw.githubusercontent.com/apache/ozone-docker/refs/heads/latest/docker-compose.yaml

docker compose up -d --scale datanode=3


connect to the SCM container:

docker exec -it weichiu-scm-1 bash

ozone sh volume create volume
ozone sh bucket create volume/bucket

pip install pyarrow


curl -L "https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.0/hadoop-3.4.0-aarch64.tar.gz" | tar -xz --wildcards 'hadoop-3.4.0/lib/native/libhdfs.*’
or
curl -L "https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.0/hadoop-3.4.0.tar.gz" | tar -xz --wildcards 'hadoop-3.4.0/lib/native/libhdfs.*’

export ARROW_LIBHDFS_DIR=hadoop-3.4.0/lib/native/
export CLASSPATH=$(ozone classpath ozone-tools)

Add to /etc/hadoop/core-site.xml

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>ofs://om:9862</value>
                <description>Where HDFS NameNode can be found on the network</description>
        </property>
</configuration>


Code:

#!/usr/bin/python
import pyarrow.fs as pafs

# Create Hadoop FileSystem object
fs = pafs.HadoopFileSystem("default", 9864)

fs.create_dir("volume/bucket/aaa")

path = "volume/bucket/file1"
with fs.open_output_stream(path) as stream:
        stream.write(b'data')

The Boto3 recipe draft was generated using ChatGPT 4o prompt:

Following the similar PyArrow using Ozone Docker image tutorial, create a similar one for boto3 using the following instructions:

ozone sh bucket create s3v/bucket



Code

#!/usr/bin/python
import boto3

# Create a local file to upload
with open("localfile.txt", "w") as f:
    f.write("Hello from Ozone via Boto3!\n")

# Configure Boto3 client
s3 = boto3.client(
's3',
endpoint_url='http://weichiu-s3g-1:9878',
aws_access_key_id='ozone-access-key',
aws_secret_access_key='ozone-secret-key'
)

# List buckets
response = s3.list_buckets()
print(response['Buckets'])

# Upload a file
s3.upload_file('localfile.txt', 'bucket', 'file.txt')

# Download a file
s3.download_file('bucket', 'file.txt', 'downloaded.txt’)

The httpfs receipe draft was generated using ChatGPT 4o, prompt:

Use the following instructions to create a tutorial of accessing Ozone using HttpFS REST API via requests library

Ozone httpfs using Python requests



# Download the latest Docker Compose configuration file
curl -O https://raw.githubusercontent.com/apache/ozone-docker/refs/heads/latest/docker-compose.yaml
add to docker-compose.yaml:   CORE-SITE.XML_fs.defaultFS: "ofs://om"
   CORE-SITE.XML_hadoop.proxyuser.hadoop.hosts: "*"
   CORE-SITE.XML_hadoop.proxyuser.hadoop.groups: "*"

docker compose up -d --scale datanode=3


connect to the SCM container:



ozone sh volume create vol1 
ozone sh bucket create vol1/bucket1

pip install requests

#!/usr/bin/python
import requests

# Ozone HTTPFS endpoint and file path
host = "http://weichiu-httpfs-1:14000"
volume = “vol1"
bucket = "bucket1"
filename = "hello.txt"
path = f"/webhdfs/v1/{volume}/{bucket}/{filename}"
user = "ozone"  # can be any value in simple auth mode

# Step 1: Initiate file creation (responds with 307 redirect)
params_create = {
    "op": "CREATE",
    "overwrite": "true",
    "user.name": user
}

print("Creating file...")
resp_create = requests.put(host + path, params=params_create, allow_redirects=False)

if resp_create.status_code != 307:
    print(f"Unexpected response: {resp_create.status_code}")
    print(resp_create.text)
    exit(1)

redirect_url = resp_create.headers['Location']
print(f"Redirected to: {redirect_url}")

# Step 2: Write data to the redirected location with correct headers
headers = {"Content-Type": "application/octet-stream"}
content = b"Hello from Ozone HTTPFS!\n"

resp_upload = requests.put(redirect_url, data=content, headers=headers)
if resp_upload.status_code != 201:
    print(f"Upload failed: {resp_upload.status_code}")
    print(resp_upload.text)
    exit(1)
print("File created successfully.")

# Step 3: Read the file back
params_open = {
    "op": "OPEN",
    "user.name": user
}

print("Reading file...")
resp_read = requests.get(host + path, params=params_open, allow_redirects=True)
if resp_read.ok:
    print("File contents:")
    print(resp_read.text)
else:
    print(f"Read failed: {resp_read.status_code}")
    print(resp_read.text)

Note: initially the draft had PySpark content. Due to the length of the content, I decided to leave it out. Will work on it in a follow-up task.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13165?filter=-1

How was this patch tested?

After Gemini/ChatGPT generated the user doc draft, I manually followed the code samples and verified the steps.

[Copilot]

Pull Request Overview

This PR adds comprehensive documentation for Python client access to Apache Ozone. It includes three new tutorial documents—one each for HTTPFS via Python Requests, PyArrow, and Boto3—and an updated interface document summarizing the available Python access methods.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
docs/content/recipe/PythonRequestsOzoneHttpFS.md	New tutorial for accessing Ozone via HTTPFS REST API using requests
docs/content/recipe/PyArrowTutorial.md	New tutorial for accessing Ozone via PyArrow in a Docker environment
docs/content/recipe/Boto3Tutorial.md	New tutorial for accessing Ozone via Boto3 using the S3 Gateway
docs/content/interface/Python.md	Updated overview of Python client interfaces with configuration and code examples

[peterxcli]

Thanks @jojochuang for this docs

[jojochuang]

Open up for review.

Please also check out the review comments previously made by Gemini: jojochuang#566

[jojochuang]

add @adoroszlai for the parts regarding httpfs

[peterxcli]

Thanks @jojochuang for working on this.