Client.upload_file send to both Workers and Scheduler

[milesgranger]

Closes #7797

• Tests added / passed

• Passes pre-commit run --all-files

• Moves the previous Worker.upload_file into Server.upload_file, giving access to both workers and the scheduler to a local_directory.

• Impl of UploadFile for the Scheduler

• Updates Client.upload_file to upload files to both scheduler and workers.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       26 files  ±  0         26 suites  ±0   16h 7m 41s ⏱️ + 57m 33s
  3 559 tests +  5    3 449 ✔️ +  4     105 💤 ±0  5 ❌ +1 
45 041 runs  +63  42 910 ✔️ +61  2 126 💤 +1  5 ❌ +1 

For more details on these failures, see this check.

Results for commit 830f4e0. ± Comparison against base commit 57ae3e7.

♻️ This comment has been updated with latest results.

[milesgranger]

@jrbourbeau this is ready for review :)

[jrbourbeau]

Thanks @milesgranger! This is looking good

The current main branch has upload_file related breakages, but no test coverage to account for those breakages. Is this something you could add here?

[milesgranger]

When you have time @jrbourbeau I think this is ready for a final look.

[milesgranger]

@jacobtomlinson I think James is out most of this week, would you be able to give this another look?

[jacobtomlinson]

This looks great thanks @milesgranger. It looks like you've addressed everything. If there is anything else from @jrbourbeau I'm sure we can address it in a follow up.

I might put out a warning tweet about renaming the temporary directly but other than that this should have minimal impact.

[jacobtomlinson]

https://twitter.com/dask_dev/status/1655871198301913089