Skip to content

Support query retry for HBO#23147

Merged
abhinavmuk04 merged 1 commit intoprestodb:masterfrom
abhinavmuk04:retryforhbo
Aug 2, 2024
Merged

Support query retry for HBO#23147
abhinavmuk04 merged 1 commit intoprestodb:masterfrom
abhinavmuk04:retryforhbo

Conversation

@abhinavmuk04
Copy link
Contributor

@abhinavmuk04 abhinavmuk04 commented Jul 8, 2024

Description

Implement retry query support for HBO in Presto, which retries failed queries and tries to retry if the query plan has changed. If the plan has changed it will re-execute, otherwise after the plan comparison we immediately fail it back

Motivation and Context

Many times queries fail because of cases where estimates are inaccurate and various out-of-memory, errors can occur. With this change enabled, users will be able to have an automatic retry which will determine if a retry of the query can prevent their query from failing entirely

Impact

Now when a query fails and a user reruns it, ending up in the query succeeding, with this session property enabled, we can speed up this process to ensure it automatically retries to see if HBO can help the query

Test Plan

Various created unit tests deeply testing different parts of the new and amended code's functionality. Also E2E tested feature within verified cluster

Please message me or Feilong for example of this change working!

Contributor checklist

  • Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.

Release Notes

* Add support for query retry when History-Based Optimization can help a failed query,
   with the ``retry-query-with-history-based-optimization`` session property :pr:`23147`

@tdcmeehan
Copy link
Contributor

I understand this is draft, but what is the design or idea behind this change?

@abhinavmuk04
Copy link
Contributor Author

I understand this is draft, but what is the design or idea behind this change?

The goal is to implement a simple version of retry query support for HBO in Presto, which retries failed queries and checks if the query plan has changed before re-executing. Currently, just building out basic portions.

@abhinavmuk04 abhinavmuk04 marked this pull request as ready for review July 24, 2024 21:09
@abhinavmuk04 abhinavmuk04 requested a review from presto-oss July 24, 2024 21:09
Copy link
Contributor

@feilong-liu feilong-liu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the plan comparison logic needs some end to end test

Copy link
Contributor

@agrawaldevesh agrawaldevesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest updating the test plan in the PR description to include exactly how you verified e2e that a retry is happening for both cases where a retry should happen and where it shouldn't. Please give examples of queries.

@abhinavmuk04 abhinavmuk04 requested a review from feilong-liu July 29, 2024 17:46
@abhinavmuk04 abhinavmuk04 requested a review from feilong-liu July 30, 2024 19:18
Copy link
Contributor

@feilong-liu feilong-liu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also shadow production traffic with the feature turned on?

@abhinavmuk04 abhinavmuk04 requested a review from feilong-liu July 31, 2024 23:49
@abhinavmuk04 abhinavmuk04 merged commit 106cc4b into prestodb:master Aug 2, 2024
@abhinavmuk04 abhinavmuk04 deleted the retryforhbo branch August 2, 2024 21:49
@tdcmeehan tdcmeehan mentioned this pull request Aug 23, 2024
34 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants