Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A more Featureful Approach to Storage APIs #265

Merged
merged 8 commits into from
Oct 14, 2021
Merged

Commits on Oct 10, 2021

  1. storage: start new API based on feature detection.

    This will eventually rival and replace linking.BlockWriteOpener
    and linking.BlockReadOpener, being more primitive than them,
    and easier to implement without importing any go-ipld-prime types.
    (The eventual migration story should be smooth, though;
    we'll probably support both styles at once for a while.)
    
    It also explicitly does not acknolwedge the interfaces from
    the github.com/ipfs/go-datastore module.  We're seeking several
    key improvements over those, including:
    
    - slimmer core; more feature-detection
    
    - only standard library types needed to implement the API
    
    - context everywhere
    
    - clear support for streaming operation
    
    - separation between read and write directions of operation
    
    - a consistent strategy for feature-detection and optional fastpaths
    
    - key type should be much simpler (and not do strange internal
      manipulations that cause useless allocations!)
    
    We'll offer wrappers that bridge back to the go-datastore style API,
    because there's lots of things implemented today using it.
    However, these objectives for a new clean API are not reasonably
    feasible to reach by incrementally evolving that older interface,
    so we have to start anew.
    
    In this commit: we have a first draft of the most essential interfaces,
    as well as always-available functions at package scope which will
    "do the right thing" in the most efficient way possible, by using
    feature detection internally on your behalf, for ease-of-use.
    warpfork committed Oct 10, 2021
    Configuration menu
    Copy the full SHA
    85fcf24 View commit details
    Browse the repository at this point in the history
  2. add binary access to the interface for Link.

    This is a pretty big change if anyone else out there
    is implementing new link types.
    
    If you're using the linking/cid package, the change is already done.
    
    I'm going to consider this a minimally breaking change because
    I don't think we've got a lot of diverse Link implementations
    out in the wild.
    
    (And even if so: this should be a pretty easy addition to make.)
    
    I'm anticipating using this as part of a good layering of APIs
    regarding storage.  (We should be able to use binary, dense things
    in any storage substrate that allows it.  Some common storage systems,
    like e.g. flatfs, don't work with binary.  But any escaping having to
    do with that should be associated with the storage system -- not
    pushed to any other layer.  Having binary access here helps do that.)
    warpfork committed Oct 10, 2021
    Configuration menu
    Copy the full SHA
    3b4df1e View commit details
    Browse the repository at this point in the history
  3. storage: move memstore.

    Fixes import cycles that I would otherwise be about to experience
    starting in the next commit.
    warpfork committed Oct 10, 2021
    Configuration menu
    Copy the full SHA
    0fdf939 View commit details
    Browse the repository at this point in the history
  4. linking: new method to help configure LinkSystem to use interfaces de…

    …fined by the storage package.
    
    This means we now have a one-liner call that can:
    
    - rig up a storage system...
    
    - which can have been written while using zero types from this repo...
    
    - and it'll DTRT.
    
    - All while it's also become much easier to write storage implementations,
      as well as extend them gracefully, because of the new storage package APIs.
    
    This seems good.
    warpfork committed Oct 10, 2021
    Configuration menu
    Copy the full SHA
    5711767 View commit details
    Browse the repository at this point in the history
  5. storage/dsadapter: a small module which bridges go-datastore forward …

    …into the new storage API.
    warpfork committed Oct 10, 2021
    Configuration menu
    Copy the full SHA
    522500c View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2021

  1. storage: Peek API talks about close explicitly.

    Opens the way to a storage implementation that makes reuse of buffers.
    warpfork committed Oct 14, 2021
    Configuration menu
    Copy the full SHA
    c56158d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    55a4896 View commit details
    Browse the repository at this point in the history
  3. storage/memstore: update to match the new interfaces.

    Most usages change from a one-liner to another one-liner.
    
    Some tests get a bit more involved, because they were actually
    using the callback structure to hook introspections up.
    warpfork committed Oct 14, 2021
    Configuration menu
    Copy the full SHA
    273362c View commit details
    Browse the repository at this point in the history