feat(flotilla): Use max sources config to determine files per partition

[colin-ho]

Changes Made

Allow the number of files per partition in swordfish to be configurable by config. For now.

Ideally the scheduler or optimizer will be able to size them better

Related Issues

Checklist

• Documented in API Docs (if applicable)
• Documented in User Guide (if applicable)
• If adding a new documentation page, doc is added to docs/mkdocs.yml navigation
• Documentation builds and is formatted properly (tag @/ccmao1130 for docs review)

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.

Project coverage is 77.32%. Comparing base (5f7fefa) to head (37399dd).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
.../daft-distributed/src/pipeline_node/scan_source.rs	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4535      +/-   ##
==========================================
+ Coverage   73.82%   77.32%   +3.49%     
==========================================
  Files         845      847       +2     
  Lines      118896   116720    -2176     
==========================================
+ Hits        87776    90254    +2478     
+ Misses      31120    26466    -4654     

Files with missing lines	Coverage Δ
.../daft-distributed/src/pipeline_node/scan_source.rs	16.89% <0.00%> (+16.89%)	⬆️

... and 57 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.