Skip to content

Added hindsight_liteLLM implementation - #17

Merged
nicoloboschi merged 10 commits into
mainfrom
hindsight_litellm
Dec 15, 2025
Merged

Added hindsight_liteLLM implementation#17
nicoloboschi merged 10 commits into
mainfrom
hindsight_litellm

Conversation

@DK09876

@DK09876 DK09876 commented Dec 5, 2025

Copy link
Copy Markdown
Contributor

No description provided.

nicoloboschi
nicoloboschi reviewed Dec 5, 2025
View reviewed changes
Comment thread hindsight-integrations/litellm/README.md Outdated
nicoloboschi
nicoloboschi requested changes Dec 8, 2025
View reviewed changes
Comment thread hindsight-api/hindsight_api/engine/llm_wrapper.py Outdated
# Model-specific max output token limits
# These represent the maximum tokens a model can generate in a single response
MODEL_MAX_OUTPUT_TOKENS = {
# OpenAI models

@nicoloboschi nicoloboschi Dec 8, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is a good idea, models get depreacted and replaced very fast. can't we just keep 65k and make other fails if not supported? unless there's a particular model we want to support

@DK09876 DK09876 Dec 8, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the reason why I did this was because Hindsight kept failing with the 65k default we had for GPT 4o-mini as it doesn't support that number for max tokens. What do you think is more appropriate to get around an issue like this?

@nicoloboschi nicoloboschi Dec 9, 2025 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can make an exclusive list where we know they don't support it and we can keep it track in the code. new models usually support >=65k tokens

I'd do someting like

16K_CONTEXT_WINDOW_MODELS = ["..."]

..

if model in 16K_CONTEXT_WINDOW_MODELS:
  max_tokens=16000

16k is just an example

The main advantage here is that the code is forward compatible by design

@DK09876
DK09876 force-pushed the hindsight_litellm branch 2 times, most recently from d583d6e to 028f809 Compare December 9, 2025 06:47
nicoloboschi
nicoloboschi requested changes Dec 9, 2025
View reviewed changes
Comment thread hindsight-api/hindsight_api/engine/llm_wrapper.py Outdated
)

@property
def max_output_tokens(self) -> int:

@nicoloboschi nicoloboschi Dec 9, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we reallny need to expose it ?

Comment thread hindsight-integrations/litellm/hindsight_litellm/wrappers.py Outdated

def recall(
query: str,
limit: int = 10,

@nicoloboschi nicoloboschi Dec 9, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hindsight doesn't have the limit/top-k parameter by design. can we get rid of this?

Comment thread hindsight-integrations/litellm/tests/test_integration.py
from unittest.mock import Mock, patch, MagicMock
from typing import List, Dict, Any

from hindsight_litellm import (

@nicoloboschi nicoloboschi Dec 9, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you make this test to run in CI ?

@DK09876
DK09876 requested a review from nicoloboschi December 10, 2025 18:48
@DK09876
DK09876 force-pushed the hindsight_litellm branch 3 times, most recently from 060e60d to 3c6a35e Compare December 12, 2025 18:53
DK09876 and others added 10 commits December 12, 2025 15:43
@DK09876
Added hindsight_liteLLM implementation
8b62b6b
@DK09876
Add instructions for entity vs bank id
b05e375
@DK09876
Add another line about entity
ebdcf78
@DK09876 @claude
Address PR review comments and enhance litellm integration
6d71dec 
- Remove deprecated limit parameter from recall() and arecall() functions
  since Hindsight uses budget/max_tokens for result control
- Remove dead MODEL_MAX_OUTPUT_TOKENS dict and max_output_tokens property
  from LLMProvider (superseded by hardcoded max_completion_tokens)
- Add test-litellm-integration job to CI workflow
- Add reflect API support with use_reflect config option
- Add verbose mode debug info via get_last_injection_debug()
- Add entity_id support for multi-user memory isolation
- Add retain() and reflect() wrapper functions
- Update docstrings and examples

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@DK09876 @claude
Make max_memories optional to allow unlimited memory injection
77be122 
- Change max_memories default from 10 to None (no limit)
- When max_memories is None, all results from the API are used
- Fix recall result handling to properly detect list vs object return
- Update wrappers (OpenAI, Anthropic) with same optional behavior

This allows users to control memory limits via max_memory_tokens
and recall_budget without an artificial count limit.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@DK09876 @claude
Remove entity_id from hindsight_litellm; add gpt-4o token cap
3aad275 
Multi-user support now uses separate bank_ids per user instead of
entity_id scoping (e.g., bank_id=f"user-{user_id}"). This simplifies
the API and aligns with the Hindsight architecture.

Also fixes max_completion_tokens error for gpt-4o models by capping
the value at 16384 (gpt-4o's limit) instead of sending the default
65000 which exceeds the model's supported maximum.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@DK09876 @claude
Fix dark mode styling across Control Plane UI components
4773d76 
Improvements to ensure proper text visibility and contrast in both light
and dark modes:

- Add global CSS rules for datetime-local calendar picker icon visibility
  using filter: invert() for both light (0.5) and dark (1) modes
- Fix text colors in dialog components to use theme-aware foreground colors
- Update memory detail panel, document/chunk modals, and data views to use
  proper dark mode text classes (text-foreground, text-card-foreground)
- Fix form labels, headings, and content text in bank selector dialogs
- Update entities view and documents view table styling for dark mode
- Bump package versions to 0.1.4

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@DK09876 @claude
Remove session_id feature and add How It Works section to README
6e9dd26 
- Remove session_id and session management (new_session, set_session,
  get_session) from config.py, callbacks.py, and __init__.py
- Session management was a client-only abstraction not backed by core API
- Add "How It Works" section to README with visual flow diagram
- Update README to remove session management documentation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@DK09876
Fix readme example
40d6a54
@DK09876
Add dark mode again
ba34bc9
@nicoloboschi
nicoloboschi merged commit dfccbf2 into main Dec 15, 2025
14 of 15 checks passed
DK09876 added a commit that referenced this pull request Mar 18, 2026
@DK09876 @claude
chore: remove manual cookbook recipe page
7e0aebd 
The sync-cookbook script will auto-generate this from the notebook
in hindsight-cookbook once PR #17 is merged.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
nicoloboschi pushed a commit that referenced this pull request Mar 20, 2026
@DK09876 @claude
feat: add LangGraph integration (#610)
b432025 
* feat: add LangGraph integration with tools, nodes, and store patterns

Add hindsight-langgraph SDK providing three integration patterns:
- Tools: retain/recall/reflect as LangChain tools for ReAct agents
- Nodes: automatic memory injection and storage as graph steps
- Store: LangGraph BaseStore implementation for checkpoint-based memory

Fix: remove `from __future__ import annotations` in nodes.py which
prevented LangGraph from passing RunnableConfig to node functions
(runtime type inspection saw string annotations instead of actual types).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: register langgraph with independent versioning system

- Set version to 0.1.0 (integrations are versioned independently)
- Add langgraph to VALID_INTEGRATIONS in release-integration.sh
- Add changelog page for langgraph integration

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: remove manual cookbook recipe page

The sync-cookbook script will auto-generate this from the notebook
in hindsight-cookbook once PR #17 is merged.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: comprehensive improvements to langgraph integration

Code fixes:
- Retain node only stores latest messages instead of all history (prevents duplicates)
- Handle multimodal msg.content (list type) in nodes
- Fix store docstring separator "/" → "."
- Apply search filters before pagination in store
- Add ttl parameter to store.aput for LangGraph BaseStore compat
- Fix _ensure_bank to not cache failed bank creations
- Fix falsy value bugs (or → is not None) in tools
- Remove from __future__ import annotations from all files
- Consistent default budget="mid" across tools/nodes/store
- Bump langgraph floor to >=0.3.0, remove duplicate dev deps

Docs fixes:
- Fix broken Cloud client example (base_url is required)
- Complete API reference tables with all parameters
- Add Limitations and Notes section (async-only store, etc.)
- Add Requirements section
- Fix broken cookbook link and Cloud claim in blog post

All 61 unit tests pass. E2E tested against Hindsight Cloud:
tools, nodes, store, configure(), multimodal content.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: remove blog post (lives in hindsight-marketing-content)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: remove Hindsight Cloud section from langgraph docs

Keep OSS docs self-hosted-first, consistent with other integration
docs (crewai, pydantic-ai, agno). Cloud setup details live in the
cookbook notebooks.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: explicitly mention LangChain compatibility in langgraph integration

The tools pattern (create_hindsight_tools) only depends on
langchain-core and works with plain LangChain via bind_tools() —
no LangGraph required. Update docs to make this clear with both
LangGraph and LangChain quick start examples.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address PR review findings

1. Guard manual test files with if __name__ == "__main__" so pytest
   doesn't collect and execute them during test runs
2. Remove semantic fallback in HindsightStore.aget() — only return
   exact document_id matches, not unrelated semantic search hits
3. Make langgraph an optional dependency — tools pattern only needs
   langchain-core. Install with pip install hindsight-langgraph[langgraph]
   for nodes and store patterns. Lazy imports with clear error messages.
4. Clean up README to be self-hosted-first, consistent with other
   integration docs
5. Update docs requirements section to reflect optional langgraph dep

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address PR review feedback for langgraph integration

- Fix #2: Add per-bank asyncio.Lock to _ensure_bank for concurrency safety
- Fix #3: Clamp search score to max(0.0, ...) to prevent negative values
- Fix #4: Implement suffix matching in _handle_list_namespaces
- Fix #5: Truncate namespaces to max_depth instead of filtering (per BaseStore contract)
- Fix #6: Remove list_namespaces/alist_namespaces overrides — let base class handle prefix=/suffix= kwargs
- Fix #7: Document ephemeral namespace tracking and get() limitations in class docstring
- Fix #8: Add stable ID to recall node SystemMessage, document ordering behavior
- Fix #9: Change budget/max_tokens/recall_tags_match defaults to None so global config fallback works
- Fix #10: Conditionally populate __all__ so import * works without langgraph installed
- Fix #11: Bump langgraph lower bound from >=0.3.0 to >=0.5.0
- Fix #12: Extract _resolve_client to shared _client.py module

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address remaining review gaps for langgraph integration

- Add output_key parameter to create_recall_node for prompt ordering control
- Add prefix/suffix/combined filter tests for list_namespaces
- Add output_key unit tests (memory text, none on empty, none on error)
- Remove unused imports and backward-compat alias in tools.py
- Update docs with output_key usage example and API reference

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: relax langgraph version constraint to >=0.3.0

Research confirmed all required APIs (BaseStore, SearchItem, Result,
GetOp, PutOp, SearchOp, ListNamespacesOp) are available since
langgraph-checkpoint 2.0.7, which maps to langgraph >=0.2.63.
Using >=0.3.0 as a clean semver boundary — >=0.5.0 was unnecessarily
conservative and excluded many compatible versions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
nepenth pushed a commit to nepenth/hindsight that referenced this pull request Aug 7, 2026
@DK09876 @claude
chore: remove manual cookbook recipe page
05d6ba1 
The sync-cookbook script will auto-generate this from the notebook
in hindsight-cookbook once PR vectorize-io#17 is merged.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
nepenth pushed a commit to nepenth/hindsight that referenced this pull request Aug 7, 2026
@DK09876 @claude
chore: remove manual cookbook recipe page
57696f8 
The sync-cookbook script will auto-generate this from the notebook
in hindsight-cookbook once PR vectorize-io#17 is merged.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nicoloboschi nicoloboschi nicoloboschi approved these changes
@cdbartholomew cdbartholomew Awaiting requested review from cdbartholomew

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@DK09876 @nicoloboschi