[FEAT]: write lance

[universalmind303]

No description provided.

[codecov]

Codecov Report

Attention: Patch coverage is 77.27273% with 5 lines in your changes missing coverage. Please review.

Please upload report for BASE (main@a2bb7f8). Learn more about missing BASE report.
Report is 9 commits behind head on main.

❗ Current head 158291c differs from pull request most recent head e8d0c62

Please upload reports for the commit e8d0c62 to get more accurate results.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2421   +/-   ##
=======================================
  Coverage        ?   63.14%           
=======================================
  Files           ?      939           
  Lines           ?   105349           
  Branches        ?        0           
=======================================
  Hits            ?    66519           
  Misses          ?    38830           
  Partials        ?        0           

Files	Coverage Δ
daft/dataframe/dataframe.py	89.12% <77.27%> (ø)

[universalmind303]

@samster25 any idea why this is failing only for 3.8 ray

https://github.com/Eventual-Inc/Daft/actions/runs/9685879782/job/26727018529?pr=2421

[samster25]

@samster25 any idea why this is failing only for 3.8 ray

https://github.com/Eventual-Inc/Daft/actions/runs/9685879782/job/26727018529?pr=2421

It looks like it's failing for all 3.8 CI jobs that use pyarrow 8+ (where the lance tests actually get triggered)

write_fragments() got an unexpected keyword argument 'storage_options'

[universalmind303]

@samster25 any idea why this is failing only for 3.8 ray
https://github.com/Eventual-Inc/Daft/actions/runs/9685879782/job/26727018529?pr=2421

It looks like it's failing for all 3.8 CI jobs that use pyarrow 8+ (where the lance tests actually get triggered)

write_fragments() got an unexpected keyword argument 'storage_options'

is ray using an older version of lance or something? because storage_options is definitely a valid argument

https://lancedb.github.io/lance/api/python/lance.html#lance.fragment.write_fragments

https://github.com/lancedb/lance/blob/acf750a9a702c2bb322fed8ef7dc4fcaffdb9c71/python/python/lance/fragment.py#L511

[samster25]

@universalmind303 I dug into the bug a bit and was able to repro on a linux machine under python 3.8. It looks like the version (pylance-0.10.15) installed under 3.8 just doesn't have storage options as an argument. Maybe they dropped support for 3.8?

To fix this, we should require a minimum version of lance where they introduced storage options.

cc: @wjones127

[universalmind303]

It looks like lance requires >=3.9

https://github.com/lancedb/lance/blob/acf750a9a702c2bb322fed8ef7dc4fcaffdb9c71/python/pyproject.toml#L9C1-L9C26

[samster25]

Looks great! Let's also make sure this is visible in the api docs as well as the integrations like this: https://www.getdaft.io/projects/docs/en/latest/user_guide/integrations/delta_lake.html

[universalmind303]

Note: zed uses pyright by default without the ability to disable it, so i needed to add this to not see 100+ irrelevant type errors.