Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Task]: RunInference - send failures to dead letter queue #24209

Closed
damccorm opened this issue Nov 16, 2022 · 5 comments · Fixed by #26261
Closed

[Task]: RunInference - send failures to dead letter queue #24209

damccorm opened this issue Nov 16, 2022 · 5 comments · Fixed by #26261
Assignees
Labels
done & done Issue has been reviewed after it was closed for verification, followups, etc. ml P2 python run-inference task

Comments

@damccorm
Copy link
Contributor

What needs to happen?

Right now, if RunInference fails a batch inference, it fails the whole transform. For batch pipelines, this means failing the pipeline on non-retryable failures (which represent most inference failures), for streaming it means infinite retries and a stuck pipeline.

We should handle failures by passing them to the next step as part of the PredictionResult object instead so that users can perform custom error handling. We should also document this behavior in the PyDoc and on our website.

Issue Priority

Priority: 2

Issue Component

Component: run-inference

@damccorm
Copy link
Contributor Author

damccorm commented Nov 16, 2022

@BjornPrime this is a good one to pick up if you have space in the future (if you finish your current work or are blocked)

@BjornPrime
Copy link
Contributor

.take-issue

@AnandInguva
Copy link
Contributor

@damccorm what is the action item on this one?

Should I pick this up as an effort to model updates? or we can add it once the model updates is out

@damccorm
Copy link
Contributor Author

I would address it separately, but if you would like to do so it would be helpful

@AnandInguva
Copy link
Contributor

.take-issue

@damccorm damccorm assigned damccorm and unassigned AnandInguva Apr 13, 2023
@github-actions github-actions bot added this to the 2.48.0 Release milestone Apr 14, 2023
@damccorm damccorm added the done & done Issue has been reviewed after it was closed for verification, followups, etc. label Apr 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
done & done Issue has been reviewed after it was closed for verification, followups, etc. ml P2 python run-inference task
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants