Add W&B Weave tracing #135

ayulockin · 2025-04-17T09:31:15Z

Description

Closes

This PR adds support for exporting the OTEL spans to Weights and Biases Weave platform for better interactivity and off-the-shelf LLMOps life-cycle.

As a user all you need to do is update the config.yaml with the following:

general:
  use_uvloop: true
  telemetry:
    tracing:
      weave:
        _type: weave
        entity: <your-wandb-entity>
        project: my-awesome-genai-app
        log_otel_only: true

The log_otel_only is a special config which allows you to:

If true, export OTEL spans generated by AgentIQ to Weave.
If false, do weave.init(..) under the hood to initialize a Weave client. Doing so will allow you to trace the underlying frameworks, llm vendors, etc., as we have auto patched integrations with them. This give a more granular trace captured automatically.

An example OTEL trace timeline exported to Weave:

View more traces in this project: https://wandb.ai/ayut/simple/weave/traces

This is a draft PR:

Refactor the integration when feat(weave): Refactor otel parsing to standardize fields wandb/weave#4143 is shipped.
Add documentation and example.

cc: @dnandakumar-nv

By Submitting this PR I confirm:

I am familiar with the Contributing Guidelines.
We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
- Any contribution which contains commits that are not Signed-Off will not be accepted.
When the PR is ready for review, new or existing tests cover these changes.
When the PR is ready for review, the documentation is up to date with these changes.

copy-pr-bot · 2025-04-17T09:31:20Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

src/aiq/observability/register.py

mdemoret-nv · 2025-04-23T00:08:36Z

@ayulockin Thanks for your PR. For outside contributors, we require the DCO signoff. You can find more info here.

mdemoret-nv · 2025-04-23T00:14:32Z

/ok to test 95fced6

ayulockin · 2025-04-23T10:20:02Z

Hey @mdemoret-nv thank you for reviewing and approving the PR.

I however, I have made some updates to the integration which allows for even more granular tracing. This is possible because, weave already has monkey patching based integration with many frameworks -- so we are now leveraging these existing traces to create a unified view of the entire application stack.

By directly connecting AIQ's intermediate steps to Weave calls and detecting existing framework traces, we can now see the complete flow from LangChain (or other frameworks) all the way through AIQ components in a single coherent trace. This enables better debugging and performance analysis without having to correlate separate traces manually.

Here's an example trace:

https://wandb.ai/ayut/simple/weave/calls/01966217-e242-7442-b5ce-b2ae9b3abb1b?descendentCallId=01966217-e242-7442-b5ce-b2ae9b3abb1b&hideTraceTree=0

mdemoret-nv · 2025-04-23T14:50:25Z

/ok to test 1834607

mdemoret-nv · 2025-04-23T14:51:29Z

Thanks for the additional updates. Let me take a look. I want to test this out with a weave instance which I am still trying to get access to.

ayulockin · 2025-04-23T16:30:37Z

Here's the updated config:

general:
  use_uvloop: true
  telemetry:
    tracing:
      weave:
        _type: weave
        entity: <wandb/weave entity> # add your entity
        project: <wandb/weave project name> # add your project name

pyproject.toml

mdemoret-nv · 2025-04-25T00:20:50Z

/ok to test c989642

…119) ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: #119 Signed-off-by: ayulockin <[email protected]>

Signed-off-by: ayulockin <[email protected]>

Intended usage: ```bash cat ci/release/pr_code_freeze_template.md | envsubst | \ gh pr create --base main --head ${SOURCE_BRANCH} \ --title "[RELEASE] ${REPO_NAME} v${RELEASE_VERSION}" \ --body-file - \ --label "! - Release" \ --web ``` ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: #123 Signed-off-by: ayulockin <[email protected]>

…e cluster (#109) Sample Usage: 1. Start the workflow service on any cluster via `aiq serve`- ``` aiq serve --config_file=examples/simple/configs/config.yml ``` 2. Send evaluation request to it using a separate (or same) eval_config file - That will return immediately with a job id. The job is run as a background task ``` curl --request POST --url http://localhost:8000/evaluate --header 'Content-Type: application/json' --data '{ "config_file": "examples/simple/configs/eval_config.yml" }' {"job_id":"551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d","status":"submitted"} ``` 3. You can then query the status by using the job-id or my asking for the status of the last (created) job - ``` curl --request GET --url http://localhost:8000/evaluate/job/last |jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 242 100 242 0 0 204 0 0:00:01 0:00:01 --:--:-- 204 { "job_id": "551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d", "status": "running", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": "None", "created_at": "2025-04-10T03:56:43.931778", "updated_at": "2025-04-10T03:56:43.933886" } ``` ``` curl --request GET --url http://localhost:8000/evaluate/job/551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d |jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 283 100 283 0 0 74 0 0:00:03 0:00:03 --:--:-- 74 { "job_id": "551d181a-3e3c-4a1c-95b4-a4dde2f8aa3d", "status": "success", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": ".tmp/aiq/examples/simple/workflow_output.json", "created_at": "2025-04-10T03:56:43.931778", "updated_at": "2025-04-10T03:57:11.722188" } ``` 4. You can get a complete list of jobs by running - ``` curl --request GET --url http://localhost:8000/evaluate/jobs | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 569 100 569 0 0 47535 0 --:--:-- --:--:-- --:--:-- 56900 [ { "job_id": "a62086d3-3373-485f-95b1-80e0477fac6b", "status": "success", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": ".tmp/aiq/examples/simple/workflow_output.json", "created_at": "2025-04-10T04:43:29.741201", "updated_at": "2025-04-10T04:43:58.451151" }, { "job_id": "a5a11719-cd8d-468d-b1c2-9855ab2e9927", "status": "success", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": ".tmp/aiq/examples/simple/workflow_output.json", "created_at": "2025-04-10T04:43:35.432887", "updated_at": "2025-04-10T04:44:04.429978" } ] ``` This endpoint allows you to filter by status - ``` curl http://localGET --url http://localhost:8000/evaluate/jobs?status=running |jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 244 100 244 0 0 3870 0 --:--:-- --:--:-- --:--:-- 3935 [ { "job_id": "1cff39fe-bedb-4089-9034-50377abc8f6b", "status": "running", "config_file": "examples/simple/configs/eval_config.yml", "error": null, "output_path": "None", "created_at": "2025-04-10T04:45:06.708431", "updated_at": "2025-04-10T04:45:06.710928" } ] ``` Closes: #100 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Eric Evans II (https://github.com/ericevans-nv) - David Gardner (https://github.com/dagardner-nv) URL: #109 Signed-off-by: ayulockin <[email protected]>

Signed-off-by: ayulockin <[email protected]>

Closes ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Eric Evans II (https://github.com/ericevans-nv) URL: #126 Signed-off-by: ayulockin <[email protected]>

Feature: Function Tracking Decorator This PR adds a straightforward decorator, `@track_function`, that makes it easy to log details about function calls. It works on synchronous and asynchronous functions, generators (sync and async), and class methods—all with a consistent interface. Key Points: - Versatile Support: Works with regular functions, async functions, generators, and even class methods. - Optional Metadata: You can add extra context by passing a dictionary to the decorator. Metadata keys are validated to ensure they’re strings. - Data Serialization: Automatically serializes input arguments and outputs (or each yield) into a JSON-friendly format. It even handles Pydantic models by calling model_dump() if needed. - Intermediate Event Logging: The decorator sends tracking events to a reactive stream in three stages: -- SPAN_START: When the function starts. -- SPAN_CHUNK: For each yielded item (for generators). -- SPAN_END: When the function returns (or ends in generators). Usage Example: ```python from aiq.profiler.decorators.function_tracking import track_function @track_function(metadata={"action": "compute", "source": "api"}) def add_numbers(a, b): return a + b ``` Just apply `@track_function` to any function (or method) where you need a bit of extra insight into the inputs and outputs. Subscribe to the reactive stream to see the published events and get a better sense of the function’s flow. ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Dhruv Nandakumar (https://github.com/dnandakumar-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #98 Signed-off-by: ayulockin <[email protected]>

Fix a typo in `aiq.profiler.decorators` ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Dhruv Nandakumar (https://github.com/dnandakumar-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) - Will Killian (https://github.com/willkill07) URL: #132 Signed-off-by: ayulockin <[email protected]>

This pull request makes the keyword arguments for the evaluation output customizer scripts optional, allowing users to omit them. Key changes include: Updating EvalCustomScriptConfig in evaluate.py to set a default empty dictionary for kwargs. Changing the workflow_to_csv.py script’s argument parser to use default values for input and output paths. Revising the documentation to reflect the new usage and relative paths. ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: #139 Signed-off-by: ayulockin <[email protected]>

#142) Closes ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Eric Evans II (https://github.com/ericevans-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) - Yuchen Zhang (https://github.com/yczhang-nv) URL: #142 Signed-off-by: ayulockin <[email protected]>

@AnuradhaKaruppiah

Closes Issue #78 ## Changes Currently AgentIQ allows you to override options in the config file for the aiq run command, and now with this change we can similarly run the eval command with the override options. cc: @AnuradhaKaruppiah ## Test ``` aiq eval --config_file examples/simple/configs/eval_config.yml \ --override llms.nim_llm.temperature 0.7 \ --override llms.nim_llm.model_name meta/llama-3.3-70b-instruct ``` <details> <summary>Response</summary> ``` 2025-04-14 22:59:35,964 - aiq.cli.cli_utils.config_override - INFO - Successfully set override for llms.nim_llm.temperature with value: 0.7 with type <class 'float'>) 2025-04-14 22:59:35,964 - aiq.cli.cli_utils.config_override - INFO - Successfully set override for llms.nim_llm.model_name with value: meta/llama-3.3-70b-instruct with type <class 'str'>) 2025-04-14 22:59:35,968 - aiq.cli.cli_utils.config_override - INFO - Configuration after overrides: embedders: nv-embedqa-e5-v5: _type: nim model_name: nvidia/nv-embedqa-e5-v5 eval: evaluators: rag_accuracy: _type: ragas llm_name: nim_rag_eval_llm metric: AnswerAccuracy rag_groundedness: _type: ragas llm_name: nim_rag_eval_llm metric: ResponseGroundedness rag_relevance: _type: ragas llm_name: nim_rag_eval_llm metric: ContextRelevance trajectory_accuracy: _type: trajectory llm_name: nim_trajectory_eval_llm general: dataset: _type: json file_path: examples/simple/data/langsmith.json output: cleanup: true dir: ./.tmp/aiq/examples/simple/ profiler: bottleneck_analysis: enable_nested_stack: true compute_llm_metrics: true concurrency_spike_analysis: enable: true spike_threshold: 7 csv_exclude_io_text: true prompt_caching_prefixes: enable: true min_frequency: 0.1 token_uniqueness_forecast: true workflow_runtime_forecast: true functions: current_datetime: _type: current_datetime general: use_uvloop: true llms: nim_llm: _type: nim model_name: meta/llama-3.3-70b-instruct temperature: 0.7 nim_rag_eval_llm: _type: nim max_tokens: 2 model_name: meta/llama-3.3-70b-instruct temperature: 1.0e-07 top_p: 0.0001 nim_trajectory_eval_llm: _type: nim max_tokens: 1024 model_name: meta/llama-3.1-70b-instruct temperature: 0.0 workflow: _type: react_agent llm_name: nim_llm max_retries: 3 retry_parsing_errors: true tool_names: - current_datetime verbose: true 2025-04-14 22:59:36,035 - aiq.eval.evaluate - INFO - Starting evaluation run with config file: examples/simple/configs/eval_config.yml 2025-04-14 22:59:36,043 - aiq.eval.evaluate - INFO - Cleaning up output directory .tmp/aiq/examples/simple 2025-04-14 22:59:36,184 - aiq.profiler.decorators - INFO - Langchain callback handler registered 2025-04-14 22:59:36,470 - aiq.agent.react_agent.agent - INFO - Filling the prompt variables "tools" and "tool_names", using the tools provided in the config. 2025-04-14 22:59:36,470 - aiq.agent.react_agent.agent - INFO - Adding the tools' input schema to the tools' description 2025-04-14 22:59:36,470 - aiq.agent.react_agent.agent - INFO - Initialized ReAct Agent Graph 2025-04-14 22:59:36,473 - aiq.agent.react_agent.agent - INFO - ReAct Graph built and compiled successfully Running workflow: 0%| | 0/3 [00:00<?, ?it/s]2025-04- ....................... The agent's thoughts are: Thought: Since I don't have the specific tool to search for Langsmith documentation and tutorials, I'll try to provide a general answer based on my knowledge. Langsmith is a platform that allows users to create and test conversational interfaces. To prototype with Langsmith, you can start by creating a new project and defining the conversational flow using their visual interface. You can then add intents, entities, and responses to create a functional conversational interface. Langsmith also provides features like testing and analytics to help you refine your prototype. Final Answer: To prototype with Langsmith, create a new project, define the conversational flow, add intents, entities, and responses, and use testing and analytics features to refine your prototype. 2025-04-14 22:59:42,047 - aiq.observability.async_otel_listener - INFO - Intermediate step stream completed. No more events will arrive. Running workflow: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:04<00:00, 1.66s/it] Evaluating Ragas nv_accuracy: 0%| | 0/3 [00:00<?, ?it/s2025-04-14 22:59:43,516 - aiq.eval.trajectory_evaluator.evaluate - INFO - Running trajectory evaluation with 3 records | 0/3 [00:00<?, ?it/s] Evaluating Ragas nv_context_relevance: 100%|██████████████████████████████████████████████████████████████████████████████| 3/3 [00:01<00:00, 1.72it/s] Evaluating Ragas nv_response_groundedness: 100%|██████████████████████████████████████████████████████████████████████████| 3/3 [00:02<00:00, 1.06it/s] Evaluating Ragas nv_accuracy: 100%|███████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:07<00:00, 2.50s/it] Evaluating Trajectory: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:06<00:00, 2.07s/it] 2025-04-14 22:59:49,774 - aiq.profiler.profile_runner - INFO - Wrote combined data to: .tmp/aiq/examples/simple/all_requests_profiler_traces.json 2025-04-14 22:59:49,815 - aiq.profiler.profile_runner - INFO - Wrote merged standardized DataFrame to .tmp/aiq/examples/simple/standardized_data_all.csv 2025-04-14 22:59:49,835 - aiq.profiler.profile_runner - INFO - Wrote inference optimization results to: .tmp/aiq/examples/simple/inference_optimization.json 2025-04-14 22:59:50,271 - aiq.profiler.profile_runner - INFO - Nested stack analysis complete 2025-04-14 22:59:50,281 - aiq.profiler.profile_runner - INFO - Concurrency spike analysis complete 2025-04-14 22:59:50,281 - aiq.profiler.profile_runner - INFO - Wrote workflow profiling report to: .tmp/aiq/examples/simple/workflow_profiling_report.txt 2025-04-14 22:59:50,281 - aiq.profiler.profile_runner - INFO - Wrote workflow profiling metrics to: .tmp/aiq/examples/simple/workflow_profiling_metrics.json 2025-04-14 22:59:50,283 - aiq.eval.evaluate - INFO - Workflow output written to 2025-04-14 22:59:50,283 - aiq.eval.utils.output_uploader - INFO - No S3 config provided; skipping upload. ``` </details> ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Hritik Raj (https://github.com/Hritik003) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: #129 Signed-off-by: ayulockin <[email protected]>

This PR adds an implementation of a ReWOO agent. - A ReWOO agent works as follows: 1. The `planner` node generates the entire chain of tools ("plan") in one go. 2. The `executor` node processes the "plan" by iterating through each step and invoking the specified tools. Each tool’s output replaces the corresponding placeholder in the plan, and that updated plan is passed to the subsequent tool. 3. The `solver` node generates the final results based on the complete "plan" and the tool outputs from the previous steps. - Compared to the ReAct agent, the ReWOO pattern eliminates the need to include the system prompt and all previous steps for every reasoning iteration, thereby reducing token usage and boosting performance. - This PR also introduces a minor refactor to the `BaseAgent` interface to accommodate the ReWOO pattern, which utilizes three nodes instead of the two nodes used by ReAct. Closes https://jirasw.nvidia.com/browse/AIQ-849 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Yuchen Zhang (https://github.com/yczhang-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #75 Signed-off-by: ayulockin <[email protected]>

* Add a simple test for the `ModelTrainer.train` method * Relocate some fixtures from `tests/aiq/eval/conftest.py` to the top-level `conftest.py` to be used by other tests. * Fix some docstrings for the two profiler forecasting models. Closes #104 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Dhruv Nandakumar (https://github.com/dnandakumar-nv) URL: #107 Signed-off-by: ayulockin <[email protected]>

Add a confirmation check from CLI before deletion of workflow. Closes - #114 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Snehlata (https://github.com/atalhens) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #137 Signed-off-by: ayulockin <[email protected]>

Signed-off-by: ayulockin <[email protected]>

Improved agent logging for consistency, readability, and information. - Re-formatted the logging information of the agents (and added some colorized font to enhance readability) - Switched some logging from `info` to `debug` level - Removed some duplicated logging - Updated console output examples in `README` The original logging: <img width="1032" alt="Screenshot 2025-04-23 at 20 41 02" src="https://github.com/user-attachments/assets/f98b58fe-35f4-4246-994d-70ce7e784246" /> The updated logging: <img width="804" alt="Screenshot 2025-04-23 at 20 40 26" src="https://github.com/user-attachments/assets/9711f0cb-04c6-4ac8-9f80-c6a839e1a5bb" /> Closes [AIQ-933](https://jirasw.nvidia.com/browse/AIQ-933?filter=-1) ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Yuchen Zhang (https://github.com/yczhang-nv) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) - David Gardner (https://github.com/dagardner-nv) - Michael Demoret (https://github.com/mdemoret-nv) - https://github.com/gzitzlsb-nv - Snehlata (https://github.com/atalhens) - Andrey Voroshilov (https://github.com/avoroshilov) - Bruno Alvisio (https://github.com/balvisio) - Eric Evans II (https://github.com/ericevans-nv) - Dhruv Nandakumar (https://github.com/dnandakumar-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: #136 Signed-off-by: ayulockin <[email protected]>

Adds a check for empty tools being used in agent configs. If we don't find any tools, raises an exception with the name of the llm in question to make debugging easier. Sample output: ``` aiq run --config_file workflow.yaml --input foo.msg [...] ValueError: No tools specified for ReAct Agent 'email_extraction_llm' Error: No tools specified for ReAct Agent 'email_extraction_llm' ``` Closes #145 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Jesse Kornblum (https://github.com/jkornblum-nv) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: #146 Signed-off-by: ayulockin <[email protected]>

Signed-off-by: Yuchen Zhang <[email protected]> Signed-off-by: ayulockin <[email protected]>

Signed-off-by: ayulockin <[email protected]>

Signed-off-by: Ayush Thakur <[email protected]>

mdemoret-nv · 2025-04-28T23:40:15Z

/ok to test 6fb02b1

Signed-off-by: Ayush Thakur <[email protected]>

ayulockin · 2025-04-29T12:34:36Z

/ok to test 5c43402

ayulockin · 2025-04-29T13:38:24Z

I think I messed up rebasing. I am closing this PR to quickly open a new one.

@mdemoret-nv

Closes This PR adds support tracing of AIQ intemediate steps and the underlying framework (Langchain, CrewAI, etc.) using Weights and Biases [Weave](https://weave-docs.wandb.ai) platform for better interactivity and off-the-shelf LLMOps life-cycle. As a user all you need to do is update the `config.yaml` with the following: ``` general: use_uvloop: true telemetry: tracing: weave: _type: weave entity: <your-wandb-entity> project: my-awesome-genai-app ``` An example trace timeline logged to Weave: <img width="1728" alt="image" src="https://github.com/user-attachments/assets/382d4926-b115-4514-a453-c58cf550a457" /> View more traces in this project: https://wandb.ai/ayut/simple_calculator/weave/traces?view=traces_default cc: @mdemoret-nv This PR was opened in favor of #135 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Ayush Thakur (https://github.com/ayulockin) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Dhruv Nandakumar (https://github.com/dnandakumar-nv) URL: #170

@mdemoret-nv

Closes This PR adds support tracing of AIQ intemediate steps and the underlying framework (Langchain, CrewAI, etc.) using Weights and Biases [Weave](https://weave-docs.wandb.ai) platform for better interactivity and off-the-shelf LLMOps life-cycle. As a user all you need to do is update the `config.yaml` with the following: ``` general: use_uvloop: true telemetry: tracing: weave: _type: weave entity: <your-wandb-entity> project: my-awesome-genai-app ``` An example trace timeline logged to Weave: <img width="1728" alt="image" src="https://github.com/user-attachments/assets/382d4926-b115-4514-a453-c58cf550a457" /> View more traces in this project: https://wandb.ai/ayut/simple_calculator/weave/traces?view=traces_default cc: @mdemoret-nv This PR was opened in favor of NVIDIA#135 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Ayush Thakur (https://github.com/ayulockin) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Dhruv Nandakumar (https://github.com/dnandakumar-nv) URL: NVIDIA#170 Signed-off-by: Yuchen Zhang <[email protected]>

@mdemoret-nv

Closes This PR adds support tracing of AIQ intemediate steps and the underlying framework (Langchain, CrewAI, etc.) using Weights and Biases [Weave](https://weave-docs.wandb.ai) platform for better interactivity and off-the-shelf LLMOps life-cycle. As a user all you need to do is update the `config.yaml` with the following: ``` general: use_uvloop: true telemetry: tracing: weave: _type: weave entity: <your-wandb-entity> project: my-awesome-genai-app ``` An example trace timeline logged to Weave: <img width="1728" alt="image" src="https://github.com/user-attachments/assets/382d4926-b115-4514-a453-c58cf550a457" /> View more traces in this project: https://wandb.ai/ayut/simple_calculator/weave/traces?view=traces_default cc: @mdemoret-nv This PR was opened in favor of NVIDIA#135 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Ayush Thakur (https://github.com/ayulockin) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Dhruv Nandakumar (https://github.com/dnandakumar-nv) URL: NVIDIA#170 Signed-off-by: Yuchen Zhang <[email protected]>

@mdemoret-nv

Closes This PR adds support tracing of AIQ intemediate steps and the underlying framework (Langchain, CrewAI, etc.) using Weights and Biases [Weave](https://weave-docs.wandb.ai) platform for better interactivity and off-the-shelf LLMOps life-cycle. As a user all you need to do is update the `config.yaml` with the following: ``` general: use_uvloop: true telemetry: tracing: weave: _type: weave entity: <your-wandb-entity> project: my-awesome-genai-app ``` An example trace timeline logged to Weave: <img width="1728" alt="image" src="https://github.com/user-attachments/assets/382d4926-b115-4514-a453-c58cf550a457" /> View more traces in this project: https://wandb.ai/ayut/simple_calculator/weave/traces?view=traces_default cc: @mdemoret-nv This PR was opened in favor of NVIDIA#135 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Ayush Thakur (https://github.com/ayulockin) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Dhruv Nandakumar (https://github.com/dnandakumar-nv) URL: NVIDIA#170 Signed-off-by: Eric Evans <[email protected]>

@mdemoret-nv

Closes This PR adds support tracing of AIQ intemediate steps and the underlying framework (Langchain, CrewAI, etc.) using Weights and Biases [Weave](https://weave-docs.wandb.ai) platform for better interactivity and off-the-shelf LLMOps life-cycle. As a user all you need to do is update the `config.yaml` with the following: ``` general: use_uvloop: true telemetry: tracing: weave: _type: weave entity: <your-wandb-entity> project: my-awesome-genai-app ``` An example trace timeline logged to Weave: <img width="1728" alt="image" src="https://github.com/user-attachments/assets/382d4926-b115-4514-a453-c58cf550a457" /> View more traces in this project: https://wandb.ai/ayut/simple_calculator/weave/traces?view=traces_default cc: @mdemoret-nv This PR was opened in favor of NVIDIA#135 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Ayush Thakur (https://github.com/ayulockin) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Dhruv Nandakumar (https://github.com/dnandakumar-nv) URL: NVIDIA#170 Signed-off-by: Eric Evans <[email protected]>

@mdemoret-nv

Closes This PR adds support tracing of AIQ intemediate steps and the underlying framework (Langchain, CrewAI, etc.) using Weights and Biases [Weave](https://weave-docs.wandb.ai) platform for better interactivity and off-the-shelf LLMOps life-cycle. As a user all you need to do is update the `config.yaml` with the following: ``` general: use_uvloop: true telemetry: tracing: weave: _type: weave entity: <your-wandb-entity> project: my-awesome-genai-app ``` An example trace timeline logged to Weave: <img width="1728" alt="image" src="https://github.com/user-attachments/assets/382d4926-b115-4514-a453-c58cf550a457" /> View more traces in this project: https://wandb.ai/ayut/simple_calculator/weave/traces?view=traces_default cc: @mdemoret-nv This PR was opened in favor of NVIDIA#135 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Ayush Thakur (https://github.com/ayulockin) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Dhruv Nandakumar (https://github.com/dnandakumar-nv) URL: NVIDIA#170

@mdemoret-nv

Closes This PR adds support tracing of AIQ intemediate steps and the underlying framework (Langchain, CrewAI, etc.) using Weights and Biases [Weave](https://weave-docs.wandb.ai) platform for better interactivity and off-the-shelf LLMOps life-cycle. As a user all you need to do is update the `config.yaml` with the following: ``` general: use_uvloop: true telemetry: tracing: weave: _type: weave entity: <your-wandb-entity> project: my-awesome-genai-app ``` An example trace timeline logged to Weave: <img width="1728" alt="image" src="https://github.com/user-attachments/assets/382d4926-b115-4514-a453-c58cf550a457" /> View more traces in this project: https://wandb.ai/ayut/simple_calculator/weave/traces?view=traces_default cc: @mdemoret-nv This PR was opened in favor of NVIDIA#135 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Ayush Thakur (https://github.com/ayulockin) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Dhruv Nandakumar (https://github.com/dnandakumar-nv) URL: NVIDIA#170

mdemoret-nv approved these changes Apr 23, 2025

View reviewed changes

src/aiq/observability/register.py Outdated Show resolved Hide resolved

mdemoret-nv added feature request New feature or request non-breaking Non-breaking change labels Apr 23, 2025

NVIDIA deleted a comment from copy-pr-bot bot Apr 23, 2025

ayulockin marked this pull request as ready for review April 23, 2025 06:32

ayulockin requested a review from a team as a code owner April 23, 2025 06:32

ayulockin commented Apr 23, 2025

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

dagardner-nv and others added 15 commits April 28, 2025 18:34

init weave integration

96122a1

Signed-off-by: ayulockin <[email protected]>

endpoint optional field

6dc3f40

Signed-off-by: ayulockin <[email protected]>

some progress

0dafad5

Signed-off-by: ayulockin <[email protected]>

handle input

26a4d40

Signed-off-by: ayulockin <[email protected]>

handle output

dd5c63f

Signed-off-by: ayulockin <[email protected]>

refactor

44a718c

Signed-off-by: ayulockin <[email protected]>

remove print

99af97a

Signed-off-by: ayulockin <[email protected]>

add usage info

be1004e

Signed-off-by: ayulockin <[email protected]>

remove print

a314f17

Signed-off-by: ayulockin <[email protected]>

dnandakumar-nv and others added 17 commits April 28, 2025 18:34

unify aiq and underlying framework's tracing

a0699ad

Signed-off-by: ayulockin <[email protected]>

lint

8c798c0

Signed-off-by: ayulockin <[email protected]>

add weave as dependency

184ec54

Signed-off-by: ayulockin <[email protected]>

Update colorama to core dependency (#149)

9943224

Signed-off-by: Yuchen Zhang <[email protected]> Signed-off-by: ayulockin <[email protected]>

init weave subpackage

9bc201e

Signed-off-by: ayulockin <[email protected]>

working weave subpackage

9439526

Signed-off-by: ayulockin <[email protected]>

lint

8e9c5d3

Signed-off-by: ayulockin <[email protected]>

Merge branch 'develop' into develop

6fb02b1

Signed-off-by: Ayush Thakur <[email protected]>

Merge branch 'develop' into develop

5c43402

Signed-off-by: Ayush Thakur <[email protected]>

ayulockin closed this Apr 29, 2025

ayulockin mentioned this pull request Apr 30, 2025

Add support for fine-grained tracing using W&B Weave #170

Merged

Add W&B Weave tracing #135

Add W&B Weave tracing #135

Uh oh!

Conversation

ayulockin commented Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

By Submitting this PR I confirm:

Uh oh!

copy-pr-bot bot commented Apr 17, 2025

Uh oh!

Uh oh!

mdemoret-nv commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mdemoret-nv commented Apr 23, 2025

Uh oh!

ayulockin commented Apr 23, 2025

Uh oh!

mdemoret-nv commented Apr 23, 2025

Uh oh!

mdemoret-nv commented Apr 23, 2025

Uh oh!

ayulockin commented Apr 23, 2025

Uh oh!

Uh oh!

mdemoret-nv commented Apr 25, 2025

Uh oh!

mdemoret-nv commented Apr 28, 2025

Uh oh!

ayulockin commented Apr 29, 2025

Uh oh!

ayulockin commented Apr 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

ayulockin commented Apr 17, 2025 •

edited

Loading

mdemoret-nv commented Apr 23, 2025 •

edited

Loading