Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat/multi llm enhancement #412

Merged
merged 62 commits into from
Jul 8, 2024
Merged

feat/multi llm enhancement #412

merged 62 commits into from
Jul 8, 2024

Conversation

jagadeeswaran-zipstack
Copy link
Contributor

@jagadeeswaran-zipstack jagadeeswaran-zipstack commented Jun 18, 2024

What

Multi llm enhancement consists of the following changes:

  1. Restrict the number of LLM profiles that can be added to a Prompt Studio project to four.
  2. Run the particular LLM profile or All LLM Profile for the current or chosen project document.
  3. We show indexed data from vector DB for each profile.
  4. Coverage and Combined output will be displayed for each profile.
  5. Info button implemented per profile that shows details of the profile.

Why

  • Currently, it is possible to add multiple LLM profiles in Prompt Studio, but the current UX/UI does not allow users to easily compare output from multiple LLMs easily. To this end, this specification proposes a way to make it easy for users to configure and see responses from LLMs in parallel, letting them compare different LLMs.

How

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

  • Yes, this PR can break existing features. The code changes have been made in quite a few existing modules, increasing the chances of regression.

Database Migrations

  • Added a new field in the model for the context.

Env Config

  • BACKEND ENV
    INDEXING_FLAG_TTL=1800

Relevant Docs

  • NA

Related Issues or PRs

  • NA

Dependencies Versions

  • NA

Notes on Testing

  • NA

Screenshots

image
image
image
image

Checklist

I have read and understood the Contribution Guidelines.

@Deepak-Kesavan
Copy link
Contributor

Deepak-Kesavan commented Jun 20, 2024

@jagadeeswaran-zipstack Make sure to update the branch with latest main and resolve frontend build issues. Also there were some pre-commit issues.

@jagadeeswaran-zipstack jagadeeswaran-zipstack requested review from a team June 20, 2024 04:00
Copy link
Contributor

@chandrasekharan-zipstack chandrasekharan-zipstack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - left some NIT comments. Make sure the polling values for FE is reviewed by @hari-kuriakose

backend/prompt_studio/prompt_studio_core/constants.py Outdated Show resolved Hide resolved
frontend/src/helpers/GetStaticData.js Outdated Show resolved Hide resolved
Copy link
Contributor

@muhammad-ali-e muhammad-ali-e left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. However, I have a major concern about the long polling. It would be better to either move it to the front end or implement a webhook method.
cc: @jagadeeswaran-zipstack @hari-kuriakose

@tahierhussain tahierhussain self-requested a review July 5, 2024 10:17
Copy link
Contributor

@tahierhussain tahierhussain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested a few improvements. Will approve the PR once they're addressed.

Copy link

github-actions bot commented Jul 8, 2024

filepath function $$\textcolor{#23d18b}{\tt{passed}}$$ SUBTOTAL
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_logs}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_cleanup}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_cleanup\_skip}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_client\_init}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_image\_exists}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_image}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_get\_container\_run\_config\_without\_mount}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{worker/src/unstract/worker/clients/test\_docker.py}}$$ $$\textcolor{#23d18b}{\tt{test\_run\_container}}$$ $$\textcolor{#23d18b}{\tt{1}}$$ $$\textcolor{#23d18b}{\tt{1}}$$
$$\textcolor{#23d18b}{\tt{TOTAL}}$$ $$\textcolor{#23d18b}{\tt{9}}$$ $$\textcolor{#23d18b}{\tt{9}}$$

Copy link

sonarcloud bot commented Jul 8, 2024

Copy link
Contributor

@tahierhussain tahierhussain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tahierhussain tahierhussain merged commit 4efcb1a into main Jul 8, 2024
5 checks passed
@tahierhussain tahierhussain deleted the FEAT/multi-llm-enhancement branch July 8, 2024 05:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants