Interacting Telemetry [2/n]: Hook up mutable AIConfig events to datadog logger #1220
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rossdanlm
force-pushed
the
pr1220
branch
3 times, most recently
from
65b08fd
to
5fc667a
Compare
rossdanlm
requested review from
saqadri,
rholinshead,
suyoglastmileai,
Ankush-lastmile and
jonathanlastmileai
as code owners
rholinshead
approved these changes
Feb 13, 2024
| @@ -452,6 +456,7 @@ function AIConfigEditorBase({ | |||
| id: promptId, | |||
| modelName: newModel, | |||
| }); | |||
| logEventHandler?.("UPDATE_PROMPT_MODEL"); | |||
There was a problem hiding this comment.
Can we also include the model name for better insight into what models are being used?:
logEventHandler?.("UPDATE_PROMPT_MODEL", {model: newModel});
| | "SET_NAME" | ||
| | "SET_DESCRIPTION" |
There was a problem hiding this comment.
nit, flip for alphabetical
…og logger This builds upon the PR from https://github.com/lastmile-ai/gradio-workbook/pull/184 where we defined the `session_id_manual` param for each session. Now we're just attaching this to all events that have the ability to change the AIConfig (we already had the run command, add prompt, download and share buttons). Note that for all the text-based ones, these fire every time we do text input chnage, so it will disproportionally spam the loggers, but we can adjust sample rate for those as a P1 in the future if needed. I heard overall that storage is not a concern for us at the moment, so feel it's fine to do this ## Test Plan Go to the `aiconfig/python/src/aiconfig/editor/client` directory and run this command ``` rm -rf node_modules && yarn && yarn build ``` Then from the `aiconfig` top-level directory, run this ``` aiconfig_path=./cookbooks/Gradio/huggingface.aiconfig.json parsers_path=./cookbooks/Gradio/aiconfig_model_registry.py aiconfig edit --aiconfig-path=$aiconfig_path --server-port=8080 --server-mode=debug_servers --parsers-module-path=$parsers_path ``` Comment out the if-else blocks from https://github.com/lastmile-ai/aiconfig/blob/601eaf64b8b1af5d7172457da2433d6bfdad1d09/python/src/aiconfig/editor/client/src/LocalEditor.tsx#L59-L62 <img width="1089" alt="Screenshot 2024-02-12 at 22 43 29" src="https://github.com/lastmile-ai/aiconfig/assets/151060367/f64ba580-03d0-4318-a334-e58aebb66882">
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Interacting Telemetry [2/n]: Hook up mutable AIConfig events to datadog logger
This builds upon the PR from https://github.com/lastmile-ai/gradio-workbook/pull/184 where we defined the
session_id_manualparam for each session.Now we're just attaching this to all events that have the ability to change the AIConfig (we already had the run command, add prompt, download and share buttons). Note that for all the text-based ones, these fire every time we do text input chnage, so it will disproportionally spam the loggers, but we can adjust sample rate for those as a P1 in the future if needed. I heard overall that storage is not a concern for us at the moment, so feel it's fine to do this
Test Plan
Go to the
aiconfig/python/src/aiconfig/editor/clientdirectory and run this commandThen from the
aiconfigtop-level directory, run thisComment out the if-else blocks from
aiconfig/python/src/aiconfig/editor/client/src/LocalEditor.tsx
Lines 59 to 62 in 601eaf6