[Data] Add enhanced support for Unity Catalog

[soffer-anyscale]

Description

This PR adds support for reading Unity Catalog Delta tables in Ray Data with automatic credential vending. This enables secure, temporary access to Delta Lake tables stored in Databricks Unity Catalog without requiring users to manage cloud credentials manually.

What's Added

• ray.data.read_unity_catalog() - Updated public API for reading Unity Catalog Delta tables
• UnityCatalogConnector - Handles Unity Catalog REST API integration and credential vending
• Multi-cloud support - Works with AWS S3, Azure Data Lake Storage, and Google Cloud Storage
• Automatic credential management - Obtains temporary, least-privilege credentials via Unity Catalog API
• Delta Lake integration - Properly configures PyArrow filesystem for Delta tables with session tokens

Key Features

✅ Production-ready credential vending API - Uses stable, public Unity Catalog APIs
✅ Secure by default - Temporary credentials with automatic cleanup
✅ Multi-cloud - AWS (S3), Azure (Blob Storage), and GCP (Cloud Storage)
✅ Delta Lake optimized - Handles session tokens and PyArrow filesystem configuration
✅ Comprehensive error handling - Helpful messages for common issues (deletion vectors, permissions, etc.)
✅ Full logging support - Debug and info logging throughout

Usage Example

import ray

# Read a Unity Catalog Delta table
ds = ray.data.read_unity_catalog(
    table="main.sales.transactions",
    url="https://dbc-XXXXXXX-XXXX.cloud.databricks.com",
    token="dapi...",
    region="us-west-2"  # Optional, for AWS
)

# Use standard Ray Data operations
ds = ds.filter(lambda row: row["amount"] > 100)
ds.show(5)

Implementation Notes

This is a simplified, focused implementation that:

• Supports Unity Catalog tables only (no volumes - that's in private preview)
• Assumes Delta Lake format (most common Unity Catalog use case)
• Uses production-ready APIs only (no private preview features)
• Provides ~600 lines of clean, reviewable code

The full implementation with volumes and multi-format support is available in the data_uc_volumes branch and can be added in a future PR once this foundation is reviewed.

Testing

• ✅ All ruff lint checks pass
• ✅ Code formatted per Ray standards
• ✅ Tested with real Unity Catalog Delta tables on AWS S3
• ✅ Proper PyArrow filesystem configuration verified
• ✅ Credential vending flow validated

Related issues

Related to Unity Catalog and Delta Lake support requests in Ray Data.

Additional information

Architecture

The implementation follows the connector pattern rather than a Datasource subclass because Unity Catalog is a metadata/credential layer, not a data format. The connector:

1. Fetches table metadata from Unity Catalog REST API
2. Obtains temporary credentials via credential vending API
3. Configures cloud-specific environment variables
4. Delegates to ray.data.read_delta() with proper filesystem configuration

Delta Lake Special Handling

Delta Lake on AWS requires explicit PyArrow S3FileSystem configuration with session tokens (environment variables alone are insufficient). This implementation correctly creates and passes the filesystem object to the deltalake library.

Cloud Provider Support

Provider	Credential Type	Implementation
AWS S3	Temporary IAM credentials	PyArrow S3FileSystem with session token
Azure Blob	SAS tokens	Environment variables (AZURE_STORAGE_SAS_TOKEN)
GCP Cloud Storage	OAuth tokens / Service account	Environment variables (GCP_OAUTH_TOKEN, GOOGLE_APPLICATION_CREDENTIALS)

Error Handling

Comprehensive error messages for common issues:

• Deletion Vectors: Guidance on upgrading deltalake library or disabling the feature
• Column Mapping: Compatibility information and solutions
• Permissions: Clear list of required Unity Catalog permissions
• Credential issues: Detailed troubleshooting steps

Future Enhancements

Potential follow-up PRs:

• Unity Catalog volumes support (when out of private preview)
• Multi-format support (Parquet, CSV, JSON, images, etc.)
• Custom datasource integration
• Advanced Delta Lake features (time travel, partition filters)

Dependencies

• Requires deltalake package for Delta Lake support
• Uses standard Ray Data APIs (read_delta, read_datasource)
• Integrates with existing PyArrow filesystem infrastructure

Documentation

• Full docstrings with examples
• Type hints throughout
• Inline comments with references to external documentation
• Comprehensive error messages with actionable guidance

[gemini-code-assist]

Code Review

This pull request provides a significant enhancement to the Unity Catalog integration by refactoring the connector into a more robust and maintainable implementation. The use of dataclasses to model API responses, detailed logging, and comprehensive error handling for common Delta Lake issues are excellent improvements. The code is clean and well-documented. I have a couple of suggestions to further improve robustness, particularly around handling required parameters and failing fast to provide clearer error messages to the user.