Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] HdfsStorage and LocalStorage's TableLocation should be full URI with scheme #121

Open
2 of 8 tasks
HotSushi opened this issue Jun 7, 2024 · 0 comments
Open
2 of 8 tasks
Labels
feat New feature or request

Comments

@HotSushi
Copy link
Collaborator

HotSushi commented Jun 7, 2024

Willingness to contribute

{"label"=>"Yes. I can contribute a fix for this bug independently.", "value"=>"independent"}

Feature Request Proposal

Currently HDFSStorage and LocalStorage set TableLocation without scheme, ie: /data/openhouse/db/tbl-UUID instead of full URI hdfs:///data/openhouse/db/tbl-UUID.

This was done to ensure backward compatibility, replication and migration.

Iceberg spec mandates a full-URI (containing scheme) as part of its metadata. source: https://iceberg.apache.org/spec/?ref=blog.min.io#manifests

As part of this bugFix, HDFSStorage and LocalStorage should start using fully-qualified URI.

Motivation

What is the use case for this feature?

Making this change will generalize tableLocation behavior for various Storages such as s3, hdfs, local etc.

What component(s) does this feature affect?

  • Table Service: This is the RESTful catalog service that stores table metadata. :services:tables
  • Jobs Service: This is the job orchestrator that submits data services for table maintenance. :services:jobs
  • Data Services: This is the jobs that performs table maintenance. apps:spark
  • Iceberg internal catalog: This is the internal Iceberg catalog for OpenHouse Catalog Service. :iceberg:openhouse
  • Spark Client Integration: This is the Apache Spark integration for OpenHouse catalog. :integration:spark
  • Documentation: This is the documentation for OpenHouse. docs
  • Local Docker: This is the local Docker environment for OpenHouse. infra/recipes/docker-compose
  • Other: Please specify the component.

Details

No response

@HotSushi HotSushi added the feat New feature or request label Jun 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feat New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant