Skip to content

Automate the test plan to run via script based on test plan #143

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
sunank200 opened this issue Nov 17, 2023 · 10 comments · Fixed by #210
Closed

Automate the test plan to run via script based on test plan #143

sunank200 opened this issue Nov 17, 2023 · 10 comments · Fixed by #210
Assignees
@vatsrahul1001
Milestone
Phase 2.5 - Enhan...

Comments

@sunank200
Copy link
Collaborator

sunank200 commented Nov 17, 2023 •
edited
Loading

Currently, the test plan is being run manually and evaluated.

  • Automate running test plan via script as manually running it is error-prone.
  • Run the script to answer questions from APIs.
  • Run the script to retrieve the answers directly from Weaviate.
  • Populate the results to the cloud bucket for evaluation.
  • Evaluation is manual currently

Depends on: #142
@sunank200 sunank200 added this to the 0.1.0 milestone Nov 17, 2023
@sunank200
Copy link
Collaborator Author

@mpgreg can you link this issue to the PR you have created? @vatsrahul1001 can add more details as required, please?

@mpgreg
Copy link
Contributor

mpgreg commented Nov 20, 2023

Some feedback.

  • Should consider NOT providing 'answer' to vectordb search. The point of the search is to see which links are retrieved. This will save cost too. Use near_text instead of with_ask.
  • Remove duplicates from references.

@sunank200
Copy link
Collaborator Author

@vatsrahul1001 @mpgreg can you create the PR for this, please?

@vatsrahul1001
Copy link
Collaborator

vatsrahul1001 commented Nov 22, 2023 •
edited
Loading

@sunank200 as discussed this can be done after 28th milestone, also @mpgreg can you raise the PR of your script whenever you have bandwidth afterwards we can close this

@phanikumv
Copy link
Collaborator

Waiting for @mpgreg to create the PR with the script that he developed

@sunank200 sunank200 removed this from the 0.1.0 milestone Nov 27, 2023
@vatsrahul1001
Copy link
Collaborator

@mpgreg we need to plan this out, let us know if there is any update on this?

@sunank200
Copy link
Collaborator Author

@mpgreg would you be able to share the code URL for this if you already have in your fork?

@phanikumv phanikumv assigned mpgreg and unassigned vatsrahul1001 Dec 7, 2023
@phanikumv
Copy link
Collaborator

Waiting for an ETA on this @mpgreg

@sunank200
Copy link
Collaborator Author

sunank200 commented Dec 13, 2023 •
edited
Loading

@mpgreg This is important for testing phase 2.5 changes. Any update on this, else we might need to change the phase 2.5 release deadlines.

mpgreg added a commit to mpgreg/ask-astro-upstream that referenced this issue Dec 14, 2023
@mpgreg
fixes astronomer#143
c9a0957
@phanikumv
Copy link
Collaborator

@mpgreg can you link the PR to this issue please?

@sunank200 sunank200 mentioned this issue Dec 18, 2023
Merged
@phanikumv phanikumv assigned vatsrahul1001 and unassigned mpgreg Dec 27, 2023
vatsrahul1001 pushed a commit that referenced this issue Jan 10, 2024
@mpgreg @pankajastro
@sunank200 @Lee-W
Automate Ask Astro QA (#210)
35b9823 
This PR addresses automation of the QA process for Ask Astro, User can
provide a list of questionnaires as a
[gsheet](https://docs.google.com/spreadsheets/d/1zJKnPQDJdLo0tP3e8lxhh2b_RY9A_YRcnl0pM1G7uA4/edit)
and run the `test_retrieval` DAG, Result
[sheet](https://docs.google.com/spreadsheets/d/19p3rwHp9CXZLxaXZ2Bf7rFTH2-rd4lVxqxlXQXwJm78/edit#gid=0)
will be generated by the DAG at respective
[env](https://drive.google.com/drive/u/1/folders/1JkxOsztKUPC19ky77c_IFAqV4eCSYFMx)
folder

closes: #143

---------

Co-authored-by: Pankaj Singh <[email protected]>
Co-authored-by: Ankit Chaurasia <[email protected]>
Co-authored-by: Wei Lee <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vatsrahul1001 vatsrahul1001

Labels
None yet
Projects
None yet
Milestone
Phase 2.5 - Enhanced Community Release
Development

Successfully merging a pull request may close this issue.

Automate Ask Astro QA mpgreg/ask-astro-upstream
4 participants
@sunank200 @mpgreg @vatsrahul1001 @phanikumv