Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to paginate through a dataset query in parallel #289

Open
davecap opened this issue Dec 17, 2018 · 0 comments
Open

Ability to paginate through a dataset query in parallel #289

davecap opened this issue Dec 17, 2018 · 0 comments

Comments

@davecap
Copy link
Member

davecap commented Dec 17, 2018

Paginating through datasets can be slow especially for large datasets. Adding a parallel query capability through requests-futures or similar would be useful since you can opt-in to the behavior if you have the dependency installed.

The implementation would need to handle the following (the same way the current client code handles them):

  • Parsing the JSON response
  • Handling error responses
  • Handling 429 rate limits

Ideally it will be transparent to the end-user. They can just iterate through a query as usual but behind the scenes it will pull multiple pages in parallel, with a configurable number of "workers".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant