fix: Redis Cache Serialization Issues with Rich Library Objects #9982
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the WalkthroughAdds cache normalization and Rich pickling support across backend and lfx modules. Introduces normalize_for_cache, integrates normalization in ChatService caching, enhances RedisCache serialization with dill fallback and sanitization, adds utilities to enable/validate Rich pickling, updates health check to report Rich pickle status, adjusts lfx graph/io for normalized cache DTOs, and pins Rich to a pickle-support branch. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor Client
participant ChatService
participant Normalizer as normalize_for_cache
participant Cache as CacheService/RedisCache
Client->>ChatService: set_cache(key, data)
ChatService->>Normalizer: normalize_for_cache(data)
Normalizer-->>ChatService: normalized DTO (envelope v1)
alt Async cache available
ChatService->>Cache: upsert(key, normalized)
else Fallback thread path
ChatService->>Cache: upsert(key, normalized) (threaded)
end
Cache-->>ChatService: status
ChatService-->>Client: {"status": true, "type": "normalized", "__envelope_version__": 1}
sequenceDiagram
autonumber
actor Client
participant API as /health
participant Utils as cache.utils
participant Rich as rich Console/ThreadLocals
Client->>API: GET /health
API->>Utils: setup_rich_pickle_support()
Utils->>Rich: patch for pickling (best-effort)
Rich-->>Utils: success/failure
API->>Utils: validate_rich_pickle_support()
Utils-->>API: ok / not ok
API-->>Client: HealthResponse{rich_pickle: status}
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/backend/base/langflow/services/chat/service.py (1)
41-47: Normalize cache contains to use str(key)
Apply tosrc/backend/base/langflow/services/chat/service.py:@@ if isinstance(self.cache_service, AsyncBaseCacheService): - return await self.cache_service.contains(key) + return await self.cache_service.contains(str(key)) @@ - return key in self.cache_service + return str(key) in self.cache_serviceAdd a unit test asserting
Truefor a non-string key (e.g.,key = 123).
🧹 Nitpick comments (11)
src/backend/base/pyproject.toml (1)
136-138: Direct references enabled — confirm packaging/distribution strategy.allow-direct-references = true is fine for internal builds, but sdists uploaded to PyPI often reject direct VCS pins and some environments block them. If you plan to ship to PyPI, consider:
- Vendoring the patch, or
- Publishing your fork as a versioned package and depending on that, or
- Using wheels-only distribution for environments that can access VCS.
If distribution to PyPI is intended, verify your release pipeline can build and install without VCS access.
src/lfx/src/lfx/io/schema.py (2)
241-247: Registering dynamic model improves importability — also add a registry to avoid name collisions.Overwriting module attribute "InputSchema" on repeated calls can collide; add a registry map for observability and uniqueness.
Apply this diff:
model = create_model("InputSchema", **fields) model.model_rebuild() - # Register class on module to improve importability for serializers + # Register class on module to improve importability for serializers import sys current_module = sys.modules[__name__] model.__module__ = __name__ setattr(current_module, "InputSchema", model) + # Keep a registry to avoid silent collisions + registry = getattr(current_module, "_INPUT_SCHEMA_REGISTRY", None) + if registry is None: + registry = {} + setattr(current_module, "_INPUT_SCHEMA_REGISTRY", registry) + registry[id(model)] = model
295-302: Also register inner models when present (param_key branch).When param_key is used, the created inner_model remains unregistered; serializers/picklers may still encounter it. Register both.
Apply this diff:
model.model_rebuild() - # Register class on module to improve importability for serializers + # Register class on module to improve importability for serializers import sys current_module = sys.modules[__name__] model.__module__ = __name__ setattr(current_module, "InputSchema", model) + # If an inner model was created earlier, register it as well + try: + inner_model # type: ignore[name-defined] + except NameError: + inner_model = None # type: ignore[assignment] + if inner_model is not None: + inner_model.__module__ = __name__ + setattr(current_module, "InputSchemaInner", inner_model) + registry = getattr(current_module, "_INPUT_SCHEMA_REGISTRY", None) + if registry is None: + registry = {} + setattr(current_module, "_INPUT_SCHEMA_REGISTRY", registry) + registry[id(model)] = model + if inner_model is not None: + registry[id(inner_model)] = inner_modelsrc/backend/base/langflow/services/cache/__init__.py (1)
6-8: Side-effectful setup at import time — consider centralizing in the factory.It’s idempotent, but you’re also calling setup in the factory. Prefer a single setup point to avoid redundant work.
src/backend/tests/unit/cache/test_normalizer.py (1)
1-3: Minor: remove unused imports and add brief test docstrings.Keeps tests lean and aligned with repo guidelines.
Apply this diff:
-import types -from pydantic import BaseModel, create_model +from pydantic import create_modelOptionally add one-line docstrings atop each test.
src/backend/tests/unit/cache/test_chatservice_cache.py (1)
43-47: Add assertion for envelope version for future-proofing.Ensures the versioned contract remains intact.
Apply this diff:
stored = fake.get("k1") assert stored["type"] == "normalized" + assert stored["__envelope_version__"] == 1 result = stored["result"]src/backend/base/langflow/services/cache/factory.py (1)
27-37: Enablement + validation flow — consider reflecting validation in the flag.If validation fails, set the flag to False to reflect actual capability.
Apply this diff:
self._rich_pickle_enabled = setup_rich_pickle_support() if self._rich_pickle_enabled: logger.debug("Rich pickle support enabled for cache serialization") # Optionally validate the support - if validate_rich_pickle_support(): - logger.debug("Rich pickle support validation successful") - else: - logger.warning("Rich pickle support validation failed") + ok = validate_rich_pickle_support() + if ok: + logger.debug("Rich pickle support validation successful") + else: + logger.warning("Rich pickle support validation failed") + self._rich_pickle_enabled = self._rich_pickle_enabled and ok else: logger.info("Rich pickle support could not be enabled")src/backend/base/langflow/services/cache/service.py (1)
236-316: Reduce duplication by reusing normalize_for_cacheConsider delegating the fallback sanitization to lfx.serialization.normalizer.normalize_for_cache for consistency across cache paths and to avoid divergent behaviors.
scripts/run_local_cache_checks.py (2)
22-33: Import placement and type-ignore specificity
- Move
_typesand_pickleimports to top (see prior diff) to fix E402.- Replace broad
# type: ignorewith a specific code (attr-defined).Apply this diff:
-_normalizer = _load_normalizer() -normalize_for_cache = _normalizer.normalize_for_cache # type: ignore +_normalizer = _load_normalizer() +normalize_for_cache = _normalizer.normalize_for_cache # type: ignore[attr-defined]
33-38: Limit dill shim impactThe shim forcibly injects a fake "dill" module. Consider scoping this to only when import fails to avoid masking a real dill install.
Apply this diff:
-# Provide a minimal dill shim for imports in cache.service -_dill = _types.ModuleType("dill") -_dill.dumps = lambda obj, *a, **k: _pickle.dumps(obj) -_dill.loads = lambda b: _pickle.loads(b) -sys.modules["dill"] = _dill +# Provide a minimal dill shim only if dill is unavailable +try: + import dill as _real_dill # noqa: F401 +except Exception: + _dill = _types.ModuleType("dill") + _dill.dumps = lambda obj, *a, **k: _pickle.dumps(obj) + _dill.loads = lambda b: _pickle.loads(b) + sys.modules["dill"] = _dillsrc/backend/base/langflow/services/cache/utils.py (1)
250-284: Validate with dill too (not just pickle)Runtime uses dill for Redis serialization. Extend validation to try dill when available.
Apply this diff:
- import pickle - - from rich.console import Console + import pickle + from rich.console import Console + try: + import dill # type: ignore[import-not-found] + except Exception: + dill = None ... - # Serialize and deserialize - pickled = pickle.dumps(test_data) - restored = pickle.loads(pickled) + # Serialize and deserialize (pickle) + pickled = pickle.dumps(test_data) + restored = pickle.loads(pickled) ... - validation_passed = "validation_test" in capture.get() + validation_passed = "validation_test" in capture.get() + # Also try dill if present + if dill is not None and validation_passed: + pickled2 = dill.dumps(test_data) + restored2 = dill.loads(pickled2) + with restored2["console"].capture() as cap2: + restored2["console"].print("validation_test_dill") + validation_passed = validation_passed and ("validation_test_dill" in cap2.get())
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (13)
scripts/run_local_cache_checks.py(1 hunks)src/backend/base/langflow/api/health_check_router.py(3 hunks)src/backend/base/langflow/services/cache/__init__.py(1 hunks)src/backend/base/langflow/services/cache/factory.py(2 hunks)src/backend/base/langflow/services/cache/service.py(3 hunks)src/backend/base/langflow/services/cache/utils.py(3 hunks)src/backend/base/langflow/services/chat/service.py(2 hunks)src/backend/base/pyproject.toml(2 hunks)src/backend/tests/unit/cache/test_chatservice_cache.py(1 hunks)src/backend/tests/unit/cache/test_normalizer.py(1 hunks)src/lfx/src/lfx/graph/graph/base.py(2 hunks)src/lfx/src/lfx/io/schema.py(2 hunks)src/lfx/src/lfx/serialization/normalizer.py(1 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
{src/backend/**/*.py,tests/**/*.py,Makefile}
📄 CodeRabbit inference engine (.cursor/rules/backend_development.mdc)
{src/backend/**/*.py,tests/**/*.py,Makefile}: Run make format_backend to format Python code before linting or committing changes
Run make lint to perform linting checks on backend Python code
Files:
src/backend/tests/unit/cache/test_normalizer.pysrc/backend/base/langflow/services/cache/factory.pysrc/backend/base/langflow/services/cache/utils.pysrc/backend/base/langflow/services/chat/service.pysrc/backend/base/langflow/api/health_check_router.pysrc/backend/base/langflow/services/cache/__init__.pysrc/backend/tests/unit/cache/test_chatservice_cache.pysrc/backend/base/langflow/services/cache/service.py
src/backend/tests/unit/**/*.py
📄 CodeRabbit inference engine (.cursor/rules/backend_development.mdc)
Test component integration within flows using create_flow, build_flow, and get_build_events utilities
Files:
src/backend/tests/unit/cache/test_normalizer.pysrc/backend/tests/unit/cache/test_chatservice_cache.py
src/backend/tests/**/*.py
📄 CodeRabbit inference engine (.cursor/rules/testing.mdc)
src/backend/tests/**/*.py: Unit tests for backend code must be located in the 'src/backend/tests/' directory, with component tests organized by component subdirectory under 'src/backend/tests/unit/components/'.
Test files should use the same filename as the component under test, with an appropriate test prefix or suffix (e.g., 'my_component.py' → 'test_my_component.py').
Use the 'client' fixture (an async httpx.AsyncClient) for API tests in backend Python tests, as defined in 'src/backend/tests/conftest.py'.
When writing component tests, inherit from the appropriate base class in 'src/backend/tests/base.py' (ComponentTestBase, ComponentTestBaseWithClient, or ComponentTestBaseWithoutClient) and provide the required fixtures: 'component_class', 'default_kwargs', and 'file_names_mapping'.
Each test in backend Python test files should have a clear docstring explaining its purpose, and complex setups or mocks should be well-commented.
Test both sync and async code paths in backend Python tests, using '@pytest.mark.asyncio' for async tests.
Mock external dependencies appropriately in backend Python tests to isolate unit tests from external services.
Test error handling and edge cases in backend Python tests, including using 'pytest.raises' and asserting error messages.
Validate input/output behavior and test component initialization and configuration in backend Python tests.
Use the 'no_blockbuster' pytest marker to skip the blockbuster plugin in tests when necessary.
Be aware of ContextVar propagation in async tests; test both direct event loop execution and 'asyncio.to_thread' scenarios to ensure proper context isolation.
Test error handling by mocking internal functions using monkeypatch in backend Python tests.
Test resource cleanup in backend Python tests by using fixtures that ensure proper initialization and cleanup of resources.
Test timeout and performance constraints in backend Python tests using 'asyncio.wait_for' and timing assertions.
Test Langflow's Messag...
Files:
src/backend/tests/unit/cache/test_normalizer.pysrc/backend/tests/unit/cache/test_chatservice_cache.py
🧬 Code graph analysis (7)
scripts/run_local_cache_checks.py (1)
src/lfx/src/lfx/serialization/normalizer.py (1)
normalize_for_cache(12-105)
src/backend/tests/unit/cache/test_normalizer.py (1)
src/lfx/src/lfx/serialization/normalizer.py (1)
normalize_for_cache(12-105)
src/backend/base/langflow/services/cache/factory.py (2)
src/backend/base/langflow/services/cache/service.py (2)
AsyncInMemoryCache(397-458)RedisCache(179-394)src/backend/base/langflow/services/cache/utils.py (2)
setup_rich_pickle_support(179-247)validate_rich_pickle_support(250-283)
src/backend/base/langflow/services/chat/service.py (1)
src/lfx/src/lfx/serialization/normalizer.py (1)
normalize_for_cache(12-105)
src/backend/base/langflow/api/health_check_router.py (1)
src/backend/base/langflow/services/cache/utils.py (2)
is_rich_pickle_enabled(286-297)validate_rich_pickle_support(250-283)
src/backend/base/langflow/services/cache/__init__.py (1)
src/backend/base/langflow/services/cache/utils.py (2)
is_rich_pickle_enabled(286-297)setup_rich_pickle_support(179-247)
src/backend/tests/unit/cache/test_chatservice_cache.py (1)
src/backend/base/langflow/services/chat/service.py (2)
ChatService(13-71)set_cache(23-46)
🪛 GitHub Check: Ruff Style Check (3.13)
scripts/run_local_cache_checks.py
[failure] 26-26: Ruff (E402)
scripts/run_local_cache_checks.py:26:1: E402 Module level import not at top of file
[failure] 23-23: Ruff (PGH003)
scripts/run_local_cache_checks.py:23:56: PGH003 Use specific rule codes when ignoring type issues
[failure] 16-16: Ruff (PT018)
scripts/run_local_cache_checks.py:16:5: PT018 Assertion should be broken down into multiple parts
[failure] 16-16: Ruff (S101)
scripts/run_local_cache_checks.py:16:5: S101 Use of assert detected
[failure] 14-14: Ruff (PTH118)
scripts/run_local_cache_checks.py:14:12: PTH118 os.path.join() should be replaced by Path with / operator
[failure] 10-10: Ruff (E402)
scripts/run_local_cache_checks.py:10:1: E402 Module level import not at top of file
[failure] 8-8: Ruff (PTH118)
scripts/run_local_cache_checks.py:8:20: PTH118 os.path.join() should be replaced by Path with / operator
[failure] 7-7: Ruff (PTH118)
scripts/run_local_cache_checks.py:7:20: PTH118 os.path.join() should be replaced by Path with / operator
[failure] 6-6: Ruff (PTH120)
scripts/run_local_cache_checks.py:6:24: PTH120 os.path.dirname() should be replaced by Path.parent
[failure] 6-6: Ruff (PTH120)
scripts/run_local_cache_checks.py:6:8: PTH120 os.path.dirname() should be replaced by Path.parent
🪛 GitHub Actions: Ruff Style Check
scripts/run_local_cache_checks.py
[error] 6-6: PTH120: os.path.dirname() should be replaced by Path.parent.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (49)
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 31/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 40/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 38/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 19/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 28/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 36/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 32/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 39/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 17/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 25/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 23/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 34/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 27/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 29/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 30/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 35/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 33/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 22/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 26/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 9/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 16/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 24/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 10/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 21/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 18/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 13/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 20/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 15/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 4/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 14/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 12/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 11/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 1/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 3/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 6/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 8/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 5/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 2/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 7/40
- GitHub Check: Run Backend Tests / Unit Tests - Python 3.10 - Group 3
- GitHub Check: Run Backend Tests / Unit Tests - Python 3.10 - Group 4
- GitHub Check: Run Backend Tests / Unit Tests - Python 3.10 - Group 2
- GitHub Check: Run Backend Tests / Unit Tests - Python 3.10 - Group 1
- GitHub Check: Run Backend Tests / Integration Tests - Python 3.10
- GitHub Check: Run Backend Tests / Unit Tests - Python 3.10 - Group 5
- GitHub Check: Test Starter Templates
- GitHub Check: Optimize new Python code in this PR
- GitHub Check: Run Ruff Check and Format
- GitHub Check: Update Starter Projects
🔇 Additional comments (13)
src/backend/base/langflow/services/cache/__init__.py (1)
2-2: Public export looks good.Exposing is_rich_pickle_enabled via all aligns with the health check and consumers.
Also applies to: 15-15
src/backend/tests/unit/cache/test_normalizer.py (1)
7-50: Good coverage for dynamic classes/functions, pydantic models, and vertex placeholders.The assertions exercise the critical paths in normalize_for_cache.
src/backend/base/langflow/services/chat/service.py (1)
34-39: Normalization envelope before caching — LGTM.Storing a normalized DTO with an envelope and version makes cache entries safer and evolvable.
src/backend/base/langflow/api/health_check_router.py (1)
71-83: Health check Rich pickle path — LGTM with clear states.The status states are explicit and safe; errors don’t leak details.
src/backend/tests/unit/cache/test_chatservice_cache.py (1)
22-47: Solid test of normalization path through ChatService.Covers type flag, vertex marker, and placeholder replacement.
src/backend/base/langflow/services/cache/factory.py (1)
59-65: Helpful runtime logging for Redis cache creation.Clear messaging if Rich serialization isn’t available.
src/lfx/src/lfx/graph/graph/base.py (2)
33-33: LGTM: Import of UnbuiltObject for DTO compatibilityImporting UnbuiltObject aligns with the new cache DTO shape.
1535-1556: LGTM: DTO-based vertex cache hydration with UnbuiltObject placeholderThe new branch correctly handles "cache_vertex" payloads, defaulting safely and preserving backwards compatibility. The UnbuiltObject placeholder avoids executing cached objects.
Please verify UnbuiltObject is available from lfx.graph.utils in all runtimes (server/tests).
src/lfx/src/lfx/serialization/normalizer.py (1)
12-105: LGTM: Comprehensive, cache-safe normalization with cycle protectionSolid handling of primitives, pydantic models, classes/callables, containers, generators, and vertex snapshots. Set→list conversion avoids unhashables. Matches DTO expectations elsewhere.
src/backend/base/langflow/services/cache/utils.py (3)
12-19: LGTM: Resilient logger import with fallbackGood defensive import; avoids hard dependency on lfx logger.
30-37: LGTM: CacheMiss sentinelSimple and effective compatibility shim.
286-298: Expose status check is fine; ensure setup is called earlyis_rich_pickle_enabled is fine. Ensure setup_rich_pickle_support runs early in cache factory init; otherwise this check will incorrectly return False.
Can you confirm setup_rich_pickle_support() is invoked during startup (e.g., in cache/init.py or factory)? If not, I can suggest a minimal init hook.
src/backend/base/langflow/services/cache/service.py (1)
325-351: Declare and pindillin your dependencies
Ensuredillis added (e.g., in pyproject.toml or requirements.txt) with an explicit version constraint.
pkusnail
force-pushed
the
fix-rich-pickle
branch
from
401f964 to
1337ce3
Compare
pkusnail
changed the title
github-actions
bot
added
bug
github-actions
bot
added
bug
…ocals serialization
…bjects Implements custom __getstate__ and __setstate__ methods for Rich library's ConsoleThreadLocals and Console classes to enable Redis caching compatibility. Fixes 'cannot pickle ConsoleThreadLocals object' error when using Redis cache.
Automatically setup and validate Rich pickle serialization when creating cache services. Includes comprehensive logging for Redis cache creation with Rich object serialization status.
Automatically initialize Rich pickle support when the cache module is imported, ensuring ConsoleThreadLocals serialization compatibility across all cache service implementations.
Extends health check response to include rich_pickle field indicating the status of Rich library serialization support. Enables monitoring of ConsoleThreadLocals pickle compatibility in production environments.
Refactor try-except blocks to use else clauses instead of early returns, following TRY300 linting rule. Maintains identical functionality while improving code style compliance.
- Implement normalize_for_cache() to convert complex objects to descriptors - Handle classes, functions, Pydantic models, and dynamic instances - Support vertex snapshots with built_object placeholders - Include cycle protection and recursive container handling - Pass all ruff checks with proper exception handling
- Add cache_solution.md documenting normalizer architecture - Add unit tests for ChatService cache and normalizer functionality - Add run_local_cache_checks.py script for validation - Update ChatService to use normalize_for_cache - Update graph base for cache restore compatibility - Add notes.md for implementation tracking - Add __init__.py for test package structure
- Add _sanitize_for_pickle method to handle problematic dynamic schemas - Replace InputSchema classes and instances with placeholder references - Handle callable objects with path-like representations - Filter out dynamically created components to prevent dill issues - Maintain two-step serialization fallback for Redis compatibility
- Add setup_rich_pickle_support function for ConsoleThreadLocals serialization - Add validate_rich_pickle_support for testing pickle functionality - Add is_rich_pickle_enabled status checker for monitoring - Implement custom __getstate__ and __setstate__ methods for Rich Console objects - Enable Redis compatibility for Rich library's threading.local objects
- Import normalize_for_cache in ChatService for safe Redis serialization - Add envelope structure with version tracking for cached data - Integrate user's normalizer architecture into chat caching pipeline - Enable cache restoration compatibility with vertex snapshots
- Register InputSchema class to module for improved serialization compatibility - Set model module to __name__ for proper importability by serializers - Enable cache-safe dynamic model creation and registration - Support both schema_to_langflow_inputs and create_input_schema workflows
- Remove cache_solution.md and notes.md from git tracking - Keep files locally for review purposes - Documentation will remain unversioned for now
Address all ruff linting violations in scripts/run_local_cache_checks.py: - Convert os.path operations to pathlib.Path usage (PTH120, PTH118) - Move imports to appropriate locations (E402) - Replace assert statements with proper error handling (S101, PT018) - Add comprehensive docstrings for better code coverage (D1) - Fix lambda argument names and type ignore annotations - Resolve whitespace and formatting issues (W293) - Add security annotations for pickle usage (S301) The script maintains full functionality while meeting CI/CD standards.
- Fix f-string usage in logging statements (G004) - Replace broad exception handling with specific exception types (BLE001) - Add proper logging instead of silent exception handling (S110) - Specify rule codes for type ignore comments (PGH003) - Fix variable naming to follow lowercase convention (N806) - Ensure all cache serialization error handling provides debug information
…es/sets), Rich pickling exclusions, pin rich SHA
pkusnail
force-pushed
the
fix-rich-pickle
branch
from
c719a4c to
c36f8c6
Compare
github-actions
bot
added
bug
github-actions
bot
added
bug
|
|
Hi @Cristhianzl, this PR looks ready to review (mergeable=MERGEABLE). Could you please take a look when you have a moment? Thank you! 🙏 |
1 participant
Fix Redis Cache Serialization Issues with Rich Library Objects
📋 Summary
Fixes "Flow build failed cannot pickle 'ConsoleThreadLocals' object" error when using Redis caching. The issue occurs because Rich library objects contain unpickleable
threading primitives (threading.local, threading.RLock).
🔍 Problem
🛠 Solution
Multi-layered approach combining Rich library fixes with comprehensive cache normalization:
📦 Dependencies
Requires custom Rich library fork with native pickle support:
The Rich fork provides the foundational fix by adding native pickle methods to Rich classes. This PR builds additional protection and normalization layers on top of
that core fix.
✅ Changes Made
Core Implementation:
Schema & Compatibility:
Testing:
🧪 Testing
📊 Impact
Files Changed: 10 files, +347 lines
Summary by CodeRabbit
New Features
Improvements
Tests
Chores