Parfun is a lightweight library providing helpers to make it easy to write and run a Python function in parallel and distributed systems.
The main feature of the library is its @parfun
decorator that transparently executes standard Python functions
following the map-reduce pattern:
from parfun import parfun
from parfun.combine.collection import list_concat
from parfun.partition.api import per_argument
from parfun.partition.collection import list_by_chunk
@parfun(
split=per_argument(
values=list_by_chunk
),
combine_with=list_concat,
)
def list_pow(values: List[float], factor: float) -> List[float]:
return [v**factor for v in values]
- Provides significant speedups to existing Python functions
- Does not require any deep knowledge of parallel or distributed computing systems
- Automatically estimates the optimal sub-task splitting (the partition size)
- Automatically handles data transmission, caching and synchronization.
- Supports various distributed computing backends, including Python's multiprocessing, Scaler or Dask.
Parfun efficiently parallelizes short-duration functions.
When running a short 0.28-second ML function on an AMD Epyc 7313 16-Cores Processor, Parfun provides an impressive 7.4x speedup. Source code for this experiment here.
The built-in Sphinx documentation contains detailed usage instructions, implementation details, and an exhaustive API reference.
Use the doc
Make target to build the HTML documentation from the source code:
make doc
The documentation's main page can then ben found at docs/build/html/index.html
.
Take a look at our documentation's quickstart tutorial to get more examples and a deeper overview of the library.
Your contributions are at the core of making this a true open source project. Any contributions you make are greatly appreciated.
We welcome you to:
- Fix typos or touch up documentation
- Share your opinions on existing issues
- Help expand and improve our library by opening a new issue
Please review our community contribution guidelines and functional contribution guidelines to get started 👍.
We are committed to making open source an enjoyable and respectful experience for our community. See
CODE_OF_CONDUCT
for more information.
This project is distributed under the Apache-2.0 License. See
LICENSE
for more information.
If you have a query or require support with this project, raise an issue. Otherwise, reach out to [email protected].