Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel data queueing for OLIApi #1303

Closed
wants to merge 25 commits into from

Conversation

avdudchenko
Copy link
Contributor

@avdudchenko avdudchenko commented Feb 17, 2024

Fixes/Resolves:

This adds capability to run parallel request to OLIApi via aiohttp.
(replace this with the issue # fixed or resolved, if no issue exists then a brief statement of what this PR does)

Summary/Motivation:

The OLI API is adding burst capability to enable larger and faster data through put. This enables user to leverages this capability through aiohttp implementation that asynchronously queues OLI Api for data.
To cleanly support this, the OLI api client is updated to process a list of requests in addition to single calls. Going forward any survey calls should generate a list of requests dicitonaries and send it to OLI API client, this has been implemented in flash.

Changes proposed in this PR:

-add aiohttp support
-add support for processing list of requests by the client it self.

Legal Acknowledgement

By contributing to this software project, I agree to the following terms and conditions for my contribution:

  1. I agree my contributions are submitted under the license terms described in the LICENSE.txt file at the top level of this directory.
  2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

@veccp veccp mentioned this pull request Feb 17, 2024
8 tasks
Copy link

codecov bot commented Feb 19, 2024

Codecov Report

Attention: Patch coverage is 7.91075% with 454 lines in your changes missing coverage. Please review.

Project coverage is 93.85%. Comparing base (a3eb077) to head (e516536).
Report is 145 commits behind head on main.

Files with missing lines Patch % Lines
watertap/tools/oli_api/client.py 6.55% 285 Missing ⚠️
watertap/tools/oli_api/flash.py 5.68% 166 Missing ⚠️
...s/oli_api/util/watertap_to_oli_helper_functions.py 81.81% 2 Missing ⚠️
watertap/tools/oli_api/conftest.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1303      +/-   ##
==========================================
- Coverage   94.41%   93.85%   -0.56%     
==========================================
  Files         370      370              
  Lines       37753    37993     +240     
==========================================
+ Hits        35645    35660      +15     
- Misses       2108     2333     +225     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ksbeattie ksbeattie added the Priority:Normal Normal Priority Issue or PR label Feb 22, 2024
@veccp veccp mentioned this pull request Feb 26, 2024
@ksbeattie ksbeattie added the Priority:High High Priority Issue or PR label Mar 7, 2024
@veccp veccp mentioned this pull request Mar 14, 2024
12 tasks
@adam-a-a adam-a-a added oli and removed Priority:Normal Normal Priority Issue or PR labels Jul 16, 2024
@ksbeattie ksbeattie added Backlog Will get to it... someday and removed Backlog Will get to it... someday labels Jul 18, 2024
@ksbeattie ksbeattie added Priority:Normal Normal Priority Issue or PR and removed Priority:High High Priority Issue or PR labels Aug 29, 2024
@TimBartholomew
Copy link
Contributor

@adam-a-a @avdudchenko we need to discuss this and whether this is still a priority, and if it is still a priority then we should identify when we want to finish it

@avdudchenko
Copy link
Contributor Author

Closing this for now - should revisit once OLI API issues are fixed with data management.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
oli Priority:Normal Normal Priority Issue or PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants