add tracing support to langchain #741
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
first pass at stdout callback for the most part, went pretty smoothly. aside from the code here, here are some comments/observations. 1. should somehow default to stdouthandler so i dont have to do ``` from langchain.callbacks import get_callback_manager from langchain.callbacks.stdout import StdOutCallbackHandler get_callback_manager().add_handler(StdOutCallbackHandler()) ``` 2. I kept around the verbosity flag. 1) this is pretty important for getting the stdout to look good for agents (and other things). 2) I actually added this for LLM class since it didn't have it. 3. The only part that isn't basically perfectly moved over is the end of the agent run. Here's a screenshot of the new stdout tracing  Noticing it is missing logging of the final thought, eg before this is what it looked like  The reason its missing is that this was previously logged as part of agent end (lines 205 and 206) this is probably only relevant for the std out logger? any thoughts for how to get it back in?
i kinda like this just because we call `self.callback_manager` so many times, and thats nicer than `self._get_callback_manager()`?
…nkush/base-tracing
1. remove verbose from someplace it didnt relaly belong 2. everywhere else, make verbose Optional[bool] with default to None 3. make base classes accept None, and then look up globabl verbosity if thats the case
deprecate all prints in favor of callback_manager.on_text (open to better naming)
remove old logging class (no longer used anyways)
also add a set handler method usage is: ``` from langchain.callbacks.streamlit import StreamlitCallbackHandler import langchain langchain.set_handler(StreamlitCallbackHandler()) ``` produces the following output  only works for agent stuff currently
* fix tracer tests * update notebook
* default session * cr
hwchase17
reviewed
Jan 26, 2023
langchain/__init__.py
Outdated
| @@ -4,6 +4,7 @@ | |||
|
|
|||
| from langchain.agents import MRKLChain, ReActChain, SelfAskWithSearchChain | |||
| from langchain.cache import BaseCache | |||
| from langchain.callbacks import set_tracing_callback_manager # noqa: F401 | |||
There was a problem hiding this comment.
It's complaining about how it's not being used anywhere yet, but we need set_tracing_callback_manager to be here so you can import it from top level langchain
There was a problem hiding this comment.
i think if you add it to the all it should be good!
langchain/callbacks/tracers/base.py
Outdated
|
|
||
|
|
||
| class BaseTracer(BaseCallbackHandler, ABC): | ||
| """An implementation of the Tracer interface that prints trace as nested json.""" |
langchain/callbacks/tracers/base.py
Outdated
| self._tracer_session = value | ||
|
|
||
|
|
||
| class BaseLangChainTracer(BaseTracer, ABC): |
There was a problem hiding this comment.
big file, could maybe be split up? base.py and then like langchain.py?
langchain/run.py
Outdated
| @@ -0,0 +1,14 @@ | |||
| """Script to run langchain-server locally using docker-compose.""" | |||
There was a problem hiding this comment.
better file name than run probably. server.py?
pyproject.toml
Outdated
| @@ -29,8 +32,10 @@ tiktoken = {version = "^0", optional = true, python="^3.9"} | |||
| pinecone-client = {version = "^2", optional = true} | |||
| weaviate-client = {version = "^3", optional = true} | |||
| google-api-python-client = {version = "2.70.0", optional = true} | |||
| freezegun = "^1.2.2" | |||
There was a problem hiding this comment.
unit tests. Since the tracer sets the start time and end time of each run using utcnow, this library overrides that for tests so we can test of equality. probably should add it as a test dep tho
There was a problem hiding this comment.
yup test dep probably best!
hwchase17
approved these changes
Jan 27, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
BaseCallbackHandlerto support tracing:SharedTracerwhich is thread-safe andTracerwhich is not and is meant to be used locally.langchain-server