Skip to content

Conversation

robtandy
Copy link

Summary

  • Add partition controls - Users can now configure how many partitions to use and partitions per task for distributed queries
    • partitions_per_task dropdown only appears if distributed is true. When it does it defaults to partitions / 2 rounded down, and it allows choices from 1 -> the current value of partitions.
  • Added dropdowns for partition selection that update properly when values change

Screenshot

image

@gabotechs
Copy link
Collaborator

It looks like the Rust lambda will be too big for the vercel runtime now. Compiling for a Vercel lambda gives the following error:

Building [email protected]:api/main.rs
LambdaSizeExceededError: The lambda function size (50.07mb) exceeds the maximum size limit (50mb).
    at executeBuild (/Users/gabriel.musatmestre/temp/datafusion-fiddle/node_modules/.pnpm/[email protected][email protected]/node_modules/vercel/dist/index.js:166494:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
  code: 'MAX_LAMBDA_SIZE_EXCEEDED',
  meta: { size: 52497350, maxLambdaSize: 52428800 }
}

Not really a problem with this PR, the space was very tight already...

We probably should not merge this PR until this is solved, otherwise we will be blocking every future deployment of the app.

@gabotechs
Copy link
Collaborator

I think we should probably move out of vercel in favor of plain AWS lambdas

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants