fix(query): Avoid race in pipeline delegator #2608
Conversation
This fixes the race in pipeline delegator by making a copy of the query. But there is an override mechanism via options wherein we can override copying query for certain referrers. These would be the subscription and subscriptions_executor referrers since they are too sensitive to copying and they would be disallowed from running parallel queries.
| @@ -26,6 +29,20 @@ | |||
| ] | |||
|
|
|||
|
|
|||
| def _is_query_copying_disallowed(current_referrer: str) -> bool: | |||
| """ | |||
| Check if the given referrer is disallowed from making a copy of the query. | |||
There was a problem hiding this comment.
maybe you could add why a given referrer would be disallowed from making a copy of the query? I'm not sure it's fully obvious to me why subscriptions and subscriptions executor are too sensitive for using deepcopy
|
Too sensitive is referring to the increased query latency we saw on the original deploy of that change, right? |
Codecov Report
@@ Coverage Diff @@
## master #2608 +/- ##
=======================================
Coverage 92.75% 92.75%
=======================================
Files 606 606
Lines 28195 28213 +18
=======================================
+ Hits 26152 26170 +18
Misses 2043 2043
Continue to review full report at Codecov.
|
This fixes the race in pipeline delegator by making a copy of the query.
But there is an override mechanism via options wherein we can override
copying query for certain referrers. These would be the
subscription and subscriptions_executor referrers since they are
too sensitive to copying and they would be disallowed from running
parallel queries.