feat: async @daft.func

[universalmind303]

Changes Made

adds basic support & tests for async @daft.func

@daft.func
async def my_udf(text)->str:
    return text.upper()

df = daft.from_pydict({
    "text":["hello", "world"]
})

print(df.select(my_udf(df["text"])).collect())

Related Issues

Checklist

• Documented in API Docs (if applicable)
• Documented in User Guide (if applicable)
• If adding a new documentation page, doc is added to docs/mkdocs.yml navigation
• Documentation builds and is formatted properly (tag @/ccmao1130 for docs review)

[greptile-apps]

Greptile Summary

This PR introduces async function support for Daft's @daft.func decorator, allowing users to write asynchronous user-defined functions (UDFs) that can perform I/O-bound operations efficiently. The implementation adds three main components:

1. Detection of async functions: The Rust layer now uses asyncio.iscoroutinefunction() to identify when a decorated function is async and routes it to a specialized handler.

2. Concurrent async execution: A new Python function call_async_batch_with_evaluated_exprs() executes all async UDF calls concurrently using asyncio.gather(), providing significant performance benefits for I/O-bound operations compared to sequential execution.

3. Modified execution path: Async functions follow a different code path than synchronous functions, collecting all evaluated arguments upfront and processing them together rather than using the chunked parallel processing approach used for sync functions.

The feature integrates cleanly with Daft's existing UDF system - users can simply add async to their function definition and await async operations inside, while the framework handles the complexity of concurrent execution. This is particularly valuable for UDFs that need to make HTTP requests, query databases, or perform other async I/O operations. A comprehensive test validates that async UDFs produce identical results to their synchronous counterparts while benefiting from concurrent execution.

Confidence score: 3/5

• This PR introduces complex async functionality that could have subtle runtime issues
• Score reflects concerns about event loop management, error handling, and argument processing edge cases
• Pay close attention to daft/udf/row_wise.py and src/daft-dsl/src/python_udf.rs for potential async-related issues

_{3 files reviewed, 2 comments}

_{Edit Code Review Bot Settings | Greptile}

[universalmind303]

this seemed like a pretty reasonable initial approach to me. I think as an optimization, we could create a global event loop that we reuse instead of potentially creating a new loop every time an async udf is called

[srilman]

Makes sense to me, I was also thinking of using the Tokio event loop and making eval_expression_list async to avoid blocking entirely.

[codecov]

Codecov Report

❌ Patch coverage is 70.76923% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.26%. Comparing base (7d76af5) to head (78b9f4d).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
daft/udf/row_wise.py	13.63%	19 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4908      +/-   ##
==========================================
+ Coverage   77.84%   79.26%   +1.41%     
==========================================
  Files         906      906              
  Lines      127107   126048    -1059     
==========================================
+ Hits        98952    99913     +961     
+ Misses      28155    26135    -2020     

Files with missing lines	Coverage Δ
src/daft-dsl/src/python_udf.rs	86.98% <100.00%> (+4.57%)	⬆️
daft/udf/row_wise.py	60.00% <13.63%> (-22.93%)	⬇️

... and 44 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[universalmind303]

precommit autoformatter updated this.

[srilman]

LGTM, thanks @universalmind303!

[srilman]

Nit: Change _inputs back to inputs since its used

[srilman]

Makes sense to me, I was also thinking of using the Tokio event loop and making eval_expression_list async to avoid blocking entirely.

[codspeed-hq]

CodSpeed Performance Report

Merging #4908 will degrade performances by 99.29%

_{Comparing cory/async-udf (78b9f4d) with main (7d76af5)}

Summary

❌ 1 regressions
✅ 23 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	BASE	HEAD	Change
❌	test_show[1 Small File]	12.5 ms	1,751.3 ms	-99.29%

[jaychia]

Plz add this into docs! This seems super cool and we should 100% be shouting it out where possible.

@kevinzwang to advise where best to advertise in the new docs