Replies: 2 comments 14 replies
-
|
Great document! 👍
|
Beta Was this translation helpful? Give feedback.
-
|
It's quite like Hyper's: |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your input. I will take this into account as I begin my PR for adding the VISION document.
Either "Reliable Behavior" or "Solid Foundation," I propose a tenet that convery OpenDAL should be stable, reliable, solid, trustworthy, and carefully built. It's a missing piece among existing tenets that no one else addresses. It may overlap with existing ones, but they focus on different aspects. It also can't be covered by 'Well-defined Behavior'. Maybe we can find better words, but for now, I haven't come up with any. Anyway, the VISION and its tenets don't mean they won't change; we can improve them as we find better words to convey the things we want to build.
I don't think it's part of the VISION. It's clear that we are going to build a layer for all storage. All other tenets are merely guidelines that lead us there. It's somewhat like a telescope. If we set the scale to the largest, we just see a big "layer." But if we zoom in a bit, we can read "an open layer," "a solid layer." If we come much closer, we will see "powered by Rust," "memory safety," and so on. |
Beta Was this translation helpful? Give feedback.
-
The core of my concern is that such a tenet is too big and hollow to provide guidance meaning and be constructive: who will ever want to uncarefully build unsolid, untrustworthy software...? Isn't it some self-evident that don't need to be told? |
Beta Was this translation helpful? Give feedback.
-
|
I think we've fully expressed our ideas. It should be better to gain others' opinions in the community.. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I have started the PR for adding this proposal in our repo: #5309 Welcome to suggest better terms that can repalce |
Beta Was this translation helpful? Give feedback.
-
|
I see in hyper Vision they talk about Non-goals. I think it may also help for OpenDAL VISION. The currently doc already talked about some |
Beta Was this translation helpful? Give feedback.
-
|
According to current
|
Beta Was this translation helpful? Give feedback.
-
Yep. I didn't include it in this proposal because I haven't thought deeply about it. Perhaps you could start a new thread for this? I'm willing to join.
Each specific feature should be discussed individually. I'm concerned about adding them prematurely, as it may eliminate potential opportunities. In fact, we ended up using some of the GCS XML API in OpenDAL because its JSON API doesn't handle concurrent writing well. One thing that comes to mind is that OpenDAL likely won't support control plane operations such as managing lifecycle, ACLs, and policies.
We definitely want to clarify that OpenDAL will not support "structured data access." |
Beta Was this translation helpful? Give feedback.
-
|
Hi, @xxchan. I'll add the vision and tenets to the repository first if you think the absence of a |
Beta Was this translation helpful? Give feedback.
-
Following up on #5296, which presents a vision but lacks tenets and use cases, making it difficult to reach an agreement. This discussion aims to address that and establish a new foundation for continuing our conversation.
Charter
One Layer, All Storage.
Tenets
Tenets are guiding principles. They guide how decisions are made for the whole project. Ideally, we do all of them all the time. In some cases, though, we may be forced to decide between slightly penalizing one goal or another. In that case, we tend to support those goals that come earlier in the list over those that come later (but every case is different).
0. Open Community
OpenDAL is an open storage library.
OpenDAL is an ASF project governed by the OpenDAL PMC. At ASF, we believe in "Community Over Code" and adhere to the Apache Way. We aim to develop OpenDAL to meet the needs of our community. We do not maintain private versions or include features that aren't useful to others.
For example, OpenDAL prefers to have clear and readable code, as this allows more people in the community to join the development.
1. Reliable Behavior
OpenDAL is a reliable storage library.
Its reliable behavior ensures that users can trust OpenDAL to perform operations on real-world storage services. Wherever possible, OpenDAL enforces this reliable behavior.
For example, OpenDAL performs additional error checks for AWS S3 complete multipart operations, as S3 may return an error in response with a 200 status code, even though this may add extra costs that conflict with "Fast Access.”
2. Fast Access
OpenDAL is a fast storage library.
Its fast access ensures that OpenDAL implements storage support with zero overhead. Users can integrate with OpenDAL without concerns about additional costs. OpenDAL should be as fast as, or faster than, the SDK for any given storage service.
For example, OpenDAL uses Capability to describe the capabilities of different services and adopts native features of those services whenever possible.
3. Object Storage First
OpenDAL is an object storage first library.
OpenDAL does support all storage services, but it is usually optimized for modern storage services. At the time of writing, we can say OpenDAL is object storage first. When designing features, OpenDAL tends to prioritize optimization for object storage.
For example, OpenDAL's Buffer design is primarily optimized for HTTP-based services, helping to reduce extra allocation, in-memory copying, and memory usage.
4. Extensible Architecture
OpenDAL is an extensible storage library.
OpenDAL can be extended to different languages, various backends, and multiple layers. Each of them is independent and does not depend on the others. Users can combine different layers such as metrics, logging, tracing, and retry, and extend their own languages, backends, and layers.
For example, OpenDAL's core never relies on the behavior or dependency of a single layer. Users can stack as many layers as they want on a given operator.
Use Cases
Who are the users of opendal? How would they use opendal?
Infrastructure Builders (databend, risingwave, greptimed, iceberg)
Use Cases:
Primary Concerns:
Application Developers (sccache, vector, rustic)
Use Cases:
Primary Concerns:
Platform Developers (pants, zino, shuttle)
Use Cases:
Primary Concerns:
This documention is inpisred a lot by hyper’s VISION document.
Beta Was this translation helpful? Give feedback.
All reactions