This repository has been archived by the owner on Apr 12, 2024. It is now read-only.
add stream API #42
Open
add stream API #42
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add stream API to stream data for a large number of results, this can be controlled by chunk_size query parameter, the server will sleep for 1 second when it has yieled `chunk_size` data. The server needs to sleep for 1 second since it needs time to gather the records (chunk_size). Signed-off-by: Priyank Singh <[email protected]>
Signed-off-by: Priyank Singh <[email protected]>
Signed-off-by: Priyank Singh <[email protected]>
Signed-off-by: Priyank Singh <[email protected]>
Signed-off-by: Priyank Singh <[email protected]>
Signed-off-by: Priyank Singh <[email protected]>
preyunk
commented
Feb 1, 2024
bridgeql/django/bridge.py
Outdated
| @@ -3,14 +3,19 @@ | |||
| # SPDX-License-Identifier: BSD-2-Clause | |||
|
|
|||
| import json | |||
| import time | |||
There was a problem hiding this comment.
// TODO remove this import
piyusql
suggested changes
Feb 1, 2024
bridgeql/django/bridge.py
Outdated
| @@ -33,6 +38,29 @@ def create_django_model(request, db_name, app_label, model_name): | |||
| return JSONResponse(res, status=e.status_code) | |||
|
|
|||
|
|
|||
| @method_decorator(require_http_methods(['GET']), name='dispatch') | |||
| class StreamView(View): | |||
| STREAM_SEPERATOR = '$$' | |||
There was a problem hiding this comment.
A simple comma will also work to make every dict a row.
| @@ -292,35 +294,55 @@ def query_has_properties(self): | |||
| return bool(set(self.params.fields).intersection( | |||
| self.model_config.get_properties())) | |||
|
|
|||
| def _get_model_fields(self, row): | |||
There was a problem hiding this comment.
change the method name to _get_model_row_dict to make it more readable.
bridgeql/django/models.py
Outdated
| @@ -269,7 +269,9 @@ def _apply_opts(self): | |||
| self.qset = self.qset[:self.params.limit] | |||
| elif isinstance(value, list): | |||
| # handle values case where property is passed in fields | |||
| if qset_opt == 'values' and self.query_has_properties(): | |||
| if qset_opt == 'values' and stream: | |||
| return self.yield_fields() | |||
There was a problem hiding this comment.
Let's reform this to make a unified approach b/w return and non-return for each method, in this case this is being a special case here to return generator.
One more point regrading count query which is being evaluated after values operator, if count option is passed, stream will beof no use and should be set to False.
|
|
||
| def queryset(self): | ||
| def queryset(self, stream=False): | ||
| # construct Q object from dictionary | ||
| query = Query(self.params.filter) | ||
| if self.params.db_name: | ||
| self.qset = self.model_config.model.objects.using( | ||
| self.params.db_name).filter(query.Q) | ||
| else: | ||
| self.qset = self.model_config.model.objects.filter(query.Q) |
There was a problem hiding this comment.
handle the stream part using self.yqset
use stream as a query parameter in read, this gives the flexibility to ignore stream if count or some other paramter is present Signed-off-by: Priyank Singh <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add stream API to stream data for a large number of results, this can be controlled by chunk_size query parameter, the server will sleep for 1 second when it has yieled
chunk_sizedata. The server needs to sleep for 1 second since it needs time to gather the records (chunk_size).