obstore

The simplest, highest-throughput ¹ Python interface to S3, GCS, Azure Storage, & other S3-compliant APIs, powered by Rust.

• Sync and async API with full type hinting.
• Streaming downloads with configurable chunking.
• Streaming uploads from async or sync iterators.
• Streaming list, with no need to paginate.
• Automatic multipart uploads for large file objects.
• Support for conditional put ("put if not exists"), as well as custom tags and attributes.
• Optionally return list results in Apache Arrow format, which is faster and more memory-efficient than materializing Python dicts.
• File-like object API and fsspec integration.
• Easy to install with no required Python dependencies.
• The underlying Rust library is production quality and used in large scale production systems, such as the Rust package registry crates.io.
• Zero-copy data exchange between Rust and Python via the buffer protocol.

Installation

To install obstore using pip:

pip install obstore

Obstore is on conda-forge and can be installed using conda, mamba, or pixi. To install obstore using conda:

conda install -c conda-forge obstore

Documentation

Full documentation is available on the website.

Head to Getting Started to dig in.

Footnotes

1. Benchmarking is currently ongoing, but early results indicate 9x higher throughput than aioboto3 and fsspec. ↩