Task submission from local scheduler client is blocking

[stephanie-wang]

Describe the problem

The local scheduler client submits tasks by creating the task spec and then writing the task spec synchronously to the local scheduler socket. This can block, if the socket buffer is full or if the local scheduler does not read from the socket quickly enough. When the local scheduler client has many tasks to submit, this can add a lot of delay, since there is a high chance that one of them will block on writing to the socket. Then, the client can't create the next task until the write completes and the issue repeats.

Potential solutions/mitigations include:

• Increasing the local scheduler socket write buffer size
• Writing to the socket asynchronously (requires running an event loop and/or possibly writing in a separate thread)
• A batch interface for task submission that allows multiple tasks to be submitted at once