added persistent caching for the uploaded files

[LinoGiger]

No description provided.

[Copilot]

Pull request overview

This pull request implements persistent disk-based caching for uploaded files, replacing the in-memory LRU cache with a persistent FanoutCache from the diskcache library.

Key Changes

• Persistent cache implementation: Replaces cachetools.LRUCache with diskcache.FanoutCache to persist uploaded file metadata across application restarts
• Cache configuration: Cache stored at ~/.rapidata/upload_cache with 100MB size limit and sharding based on upload worker count
• Dependency updates: Removed cachetools dependency and added diskcache ^5.6.3

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 3 comments.

File	Description
src/rapidata/rapidata_client/datapoints/_asset_uploader.py	Replaced in-memory LRU cache with persistent FanoutCache, updated cache access pattern from dictionary-style to .get() method
pyproject.toml	Replaced cachetools dependency with diskcache; added unrelated dev dependencies (ipykernel, jupyter)
poetry.lock	Updated lock file with new dependencies and transitive dependencies for diskcache, ipykernel, and jupyter

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The timeout of 0.01 seconds (10 milliseconds) is extremely low for a disk cache operation. This could cause frequent timeout errors when multiple processes or threads try to access the cache simultaneously. Consider increasing this to at least 1 second (timeout=1.0) or using the default timeout value to avoid cache access failures under concurrent load.

Suggested change

	timeout=0.01,
	timeout=1.0,

[Copilot]

The FanoutCache is initialized at class definition time, which means it creates the cache directory immediately when the module is imported, even if AssetUploader is never used. This could cause issues if the .rapidata directory cannot be created (e.g., in read-only environments or CI/CD pipelines). Consider lazy initialization by moving the cache creation to __init__ or using a class method with @functools.cached_property to defer the cache creation until first use.

[Copilot]

The ipykernel and jupyter dependencies are added to the dev dependencies group, but they are unrelated to the persistent caching feature described in the PR title. These should be removed unless they are part of a separate change, or this PR should be split into two separate PRs to keep changes focused and easier to review.

Suggested change

	ipykernel = "^7.1.0"
	jupyter = "^1.1.1"

[claude]

Pull Request Review: Added Persistent Caching for Uploaded Files

Summary

This PR adds persistent disk-based caching for uploaded files using the diskcache library. The implementation caches uploaded assets to avoid re-uploading the same files, which is a valuable optimization.

Code Quality and Best Practices

✅ Positive Aspects

1. Good cache key design - The cache key includes environment, file path, size, and modification time (_get_cache_key at line 29-39), which properly invalidates cache when files change
2. Configuration integration - The cacheUploads flag in UploadConfig allows users to disable caching if needed
3. Type safety - Proper use of type hints and cast for cache retrieval (line 50)
4. Shared cache instance - Using a class-level _shared_upload_cache is efficient for sharing across instances

⚠️ Concerns and Suggestions

1. Cache Timeout Configuration (Medium Priority)

Location: _asset_uploader.py:22

timeout=0.01,  # 10ms timeout

The timeout is set to 10ms, which is very aggressive. This means if a cache operation takes longer than 10ms, it will fail silently. Consider:

• Increasing this to a more reasonable value (e.g., 1.0 second) for production use
• Making it configurable via UploadConfig
• Adding logging when cache operations timeout

2. Missing Error Handling (Medium Priority)

Location: _asset_uploader.py:47-50

cached_value = self._shared_upload_cache.get(asset_key)

Cache operations could fail due to:

• Disk full conditions
• Permission issues
• Corrupted cache files

Recommendation: Wrap cache operations in try-except blocks and log failures without breaking the upload flow:

try:
    cached_value = self._shared_upload_cache.get(asset_key)
    if cached_value is not None:
        logger.debug("Asset found in cache")
        return cast(str, cached_value)
except Exception as e:
    logger.warning(f"Cache read failed: {e}, proceeding with upload")

3. Cache Size Limit (Low Priority)

Location: _asset_uploader.py:23

size_limit=100_000_000,  # 100MB

The 100MB size limit seems reasonable, but consider:

• Documenting this limit in the code or user-facing documentation
• Making it configurable for users with different disk space constraints
• The limit is per shard, so total cache size = 100MB × maxWorkers (default 10 = 1GB total)

4. Cache Directory Creation (Low Priority)

The cache directory ~/.rapidata/upload_cache is created automatically by diskcache, but users might want to:

• Know where this cache lives (documentation)
• Configure the cache location
• Have a way to clear the cache programmatically

Potential Bugs

1. URL Caching Issue (High Priority)

Location: _asset_uploader.py:32-33

if re.match(r"^https?://", asset):
    return f"{env}@{asset}"

Problem: URL-based assets are cached indefinitely with no invalidation. If the content at a URL changes, the cached filename will still be returned, potentially pointing to stale content.

Impact: Users uploading URLs that frequently change (e.g., dynamic CDN URLs) will get incorrect results.

Recommendation: Consider one of these approaches:

• Add a timestamp component to URL cache keys to limit cache lifetime
• Make URL caching configurable separately from file caching
• Document this behavior clearly for users
• Consider fetching HTTP headers (Last-Modified, ETag) for smarter invalidation

2. Race Condition on File Changes (Medium Priority)

Location: _asset_uploader.py:38-39

stat = os.stat(asset)
return f"{env}@{asset}:{stat.st_size}:{stat.st_mtime_ns}"

Problem: There's a TOCTOU (time-of-check-time-of-use) race condition. A file could be modified between:

1. When _get_cache_key reads st_mtime_ns
2. When the file is actually uploaded (line 57-59)

Impact: In rare cases, the wrong version of a file could be cached. This is unlikely in typical usage but possible in automated systems with rapid file updates.

Severity: Low in practice, but worth noting.

Performance Considerations

✅ Good Decisions

1. FanoutCache with sharding - Using shards=rapidata_config.upload.maxWorkers enables concurrent access without lock contention
2. Early return on cache hit - Avoids unnecessary work when assets are cached

💡 Optimization Opportunities

1. Batch cache operations - If uploading multiple assets, could potentially batch cache lookups/writes
2. Cache warming - Could provide a utility to pre-populate cache for known assets

Security Concerns

⚠️ Minor Security Considerations

1. Cache Location in Home Directory (Low Risk)

   • Cache is stored in ~/.rapidata/upload_cache which is user-accessible
   • On shared systems, ensure proper file permissions
   • No sensitive data should be stored in the cache (looks good - only filenames)

2. Path Traversal (Already Mitigated)

   • The code doesn't construct file paths from cached data
   • Cache keys are deterministic and safe

Test Coverage

❌ Missing Tests

Critical Gap: No tests found for the caching functionality.

Recommended Test Cases:

1. Cache hit when uploading same file twice
2. Cache miss when file is modified
3. Cache miss when file size changes
4. URL caching behavior
5. Cache behavior when disabled via config
6. Cache operation failures (disk full, permission denied)
7. Concurrent uploads with caching
8. Environment isolation (same file, different environments)

Example test structure:

def test_cache_hit_for_identical_file():
    # Upload file once, verify API called
    # Upload same file again, verify API not called
    # Verify same filename returned

def test_cache_miss_on_file_modification():
    # Upload file, get cached
    # Modify file (change mtime or size)
    # Upload again, verify API called again

Dependencies

✅ Dependency Addition

• diskcache 5.6.3 - Well-maintained, mature library with good performance characteristics
• Adds significant dependencies (visible in poetry.lock) but they're reasonable for the functionality
• Note: Also adds ipykernel and jupyter to dev dependencies, which is unrelated to caching but appears in this PR

Recommendations Summary

High Priority

1. ✅ Add comprehensive test coverage for caching functionality
2. ⚠️ Address URL caching invalidation strategy
3. ⚠️ Add error handling around cache operations

Medium Priority

4. Make cache timeout configurable and increase default value
5. Add documentation for cache location and size limits
6. Consider adding cache management utilities (clear cache, get cache stats)

Low Priority

7. Make cache size limit configurable
8. Add metrics/logging for cache hit rates
9. Document the TOCTOU race condition or add file content hashing

Overall Assessment

This is a solid implementation of a useful feature. The cache key design is well thought out, and the integration with the existing config system is clean. The main concerns are around error handling, test coverage, and the URL caching strategy. With the suggested improvements, this would be production-ready.

Recommendation: Approve with requested changes (tests + error handling).

---

Review generated by Claude Code 🤖

[claude]

Pull Request Review: Persistent Caching for Uploaded Files

Summary

This PR adds persistent caching functionality for uploaded files using diskcache.FanoutCache, which is a solid improvement for reducing redundant uploads. The implementation is generally well-structured, but there are several important considerations to address.

---

🟢 Strengths

1. Good architectural pattern: Using a handler registration pattern for config updates is clean and allows the AssetUploader to react to configuration changes dynamically.

2. Cache key design: The cache key generation in _get_cache_key() is well thought out:

   • Includes environment to prevent cache pollution across different environments
   • Uses file metadata (size + mtime) for local files to detect changes
   • Handles both URLs and local file paths

3. Error handling: The _handle_config_update() method properly handles exceptions when closing the old cache.

4. Configurability: All cache parameters are configurable through UploadConfig, giving users control over behavior.

---

🔴 Critical Issues

1. Race Condition in Config Update Handler (src/rapidata/rapidata_client/config/upload_config.py:44-46)

def __setattr__(self, name: str, value) -> None:
    super().__setattr__(name, value)
    self._notify_handlers()  # Called on EVERY attribute set

Problem: __setattr__ is called during __init__ for every field, meaning handlers will be triggered multiple times during object construction. This could cause:

• Multiple cache reinitializations during a single config update
• Race conditions if multiple threads are accessing the cache during recreation
• Performance overhead from unnecessary cache recreations

Recommendation:

• Add a flag to track initialization state, or
• Only notify handlers after full initialization, or
• Debounce handler notifications

Example fix:

def __init__(self, **kwargs):
    self._initializing = True
    super().__init__(**kwargs)
    self._initializing = False
    self._notify_handlers()

def __setattr__(self, name: str, value) -> None:
    super().__setattr__(name, value)
    if not getattr(self, '_initializing', False):
        self._notify_handlers()

2. Thread Safety Concerns (src/rapidata/rapidata_client/datapoints/_asset_uploader.py:31-62)

Problem: The _handle_config_update() method closes and recreates the shared cache without thread synchronization. If multiple threads are using AssetUploader concurrently:

• Thread A could be reading/writing to the cache
• Thread B triggers a config update, closing the cache
• Thread A's operations fail or corrupt data

Recommendation: Add a lock around cache operations and config updates:

import threading

class AssetUploader:
    _cache_lock = threading.RLock()
    _shared_upload_cache: FanoutCache = ...
    
    def upload_asset(self, asset: str) -> str:
        with self._cache_lock:
            # ... cache operations
    
    @classmethod
    def _handle_config_update(cls, config):
        with cls._cache_lock:
            # ... cache recreation

3. Potential Memory Leak with Handler Registration (src/rapidata/rapidata_client/datapoints/_asset_uploader.py:29)

def __init__(self, openapi_service: OpenAPIService):
    self.openapi_service = openapi_service
    register_upload_config_handler(self._handle_config_update)  # Leaks!

Problem: Handlers are registered but never unregistered. Each AssetUploader instance adds a handler to the global list, but since _handle_config_update is a classmethod, this creates unnecessary duplicate registrations. If multiple RapidataClient instances are created, the handler list grows unbounded.

Recommendation:

• Only register once (use a flag), or
• Unregister in a __del__ method, or
• Use weak references

---

🟡 Important Issues

4. Silent Failures in Handler Notification (src/rapidata/rapidata_client/config/upload_config.py:50-54)

for handler in _upload_config_handlers:
    try:
        handler(self)
    except Exception as e:
        print(f"Warning: UploadConfig handler failed: {e}")  # Only prints!

Problem: Exceptions are caught and only printed, not logged. This could hide serious issues.

Recommendation: Use the logger:

from rapidata.rapidata_client.config import logger

def _notify_handlers(self) -> None:
    for handler in _upload_config_handlers:
        try:
            handler(self)
        except Exception as e:
            logger.warning(f"UploadConfig handler failed: {e}", exc_info=True)

5. No Cache Directory Creation (src/rapidata/rapidata_client/datapoints/_asset_uploader.py:19-24)

Problem: FanoutCache is initialized with Path.home() / ".rapidata" / "upload_cache", but there's no guarantee this directory exists. While diskcache might create it, explicit directory creation with proper error handling is better.

Recommendation: Add directory creation:

@classmethod
def _handle_config_update(cls, config):
    logger.debug("Updating AssetUploader shared upload cache with new config")
    try:
        # Ensure directory exists
        config.cacheLocation.mkdir(parents=True, exist_ok=True)
        
        # ... rest of the method

6. No Cache Eviction Strategy Documentation

Problem: The PR sets cacheSizeLimit to 100MB but doesn't document:

• What happens when the limit is reached?
• How does diskcache handle eviction?
• Are there any performance implications?

Recommendation: Add documentation in the docstring explaining the eviction policy (LRU by default for diskcache).

---

🟡 Code Quality Issues

7. Inconsistent Naming Conventions

The config uses camelCase (cacheUploads, maxWorkers) instead of Python's snake_case convention. While this appears to be a project-wide pattern, consider documenting this decision or gradually migrating to snake_case for better Python idiomaticity.

8. Missing Type Hints (src/rapidata/rapidata_client/config/upload_config.py:40-46)

def __setattr__(self, name: str, value) -> None:  # 'value' lacks type hint

Recommendation: Add value: Any or use a more specific type.

9. Magic Numbers (src/rapidata/rapidata_client/config/upload_config.py:38)

cacheSizeLimit: int = Field(default=100_000_000)  # 100MB

Recommendation: Define as a constant with documentation:

DEFAULT_CACHE_SIZE_MB = 100
cacheSizeLimit: int = Field(default=DEFAULT_CACHE_SIZE_MB * 1024 * 1024)

---

🔵 Testing Concerns

10. No Tests for New Functionality

Critical Gap: The PR adds significant new functionality but includes no tests. Required test coverage:

1. Cache hit/miss behavior

   • Verify cache returns cached values for duplicate uploads
   • Verify cache miss triggers actual upload

2. File change detection

   • Verify cache invalidation when file content changes
   • Verify mtime changes trigger re-upload

3. Config update handling

   • Verify cache recreation on config changes
   • Verify no data loss during config updates

4. Thread safety (if supporting concurrent uploads)

   • Verify concurrent uploads don't corrupt cache
   • Verify config updates during active uploads

5. Cache clearing

   • Verify clear_cache() works correctly

6. Environment isolation

   • Verify different environments don't share cache entries

Recommendation: Add tests to /tests/ directory covering these scenarios.

---

🟢 Performance Considerations

11. Positive: Using FanoutCache with shards

The use of shards=maxWorkers is good for concurrent access as it reduces lock contention.

12. Potential Issue: Cache Timeout

The default cacheTimeout=0.1 (100ms) is very short. This might cause operations to fail unnecessarily under load.

Recommendation: Consider increasing default to 1.0 seconds or making it more generous, with documentation about when users might need to adjust it.

---

🟣 Security Considerations

13. Cache Location Permissions

Problem: The cache is stored in ~/.rapidata/upload_cache without explicit permission settings. Cached data might contain sensitive information.

Recommendation: Set restrictive permissions on cache directory:

cache_dir = config.cacheLocation
cache_dir.mkdir(parents=True, exist_ok=True, mode=0o700)  # Owner-only access

14. Cache Key Collision Risk

Problem: The cache key for URLs is just f"{env}@{asset}". Two identical URLs to different content (e.g., updated resources) won't be detected.

Recommendation: Consider adding timestamp or content hash for URLs if this is a concern, or document this limitation.

---

📝 Documentation Gaps

15. Missing Documentation

• No docstrings for register_upload_config_handler() and unregister_upload_config_handler() explaining use cases
• No documentation about when cache is used vs. bypassed
• No migration guide for existing users
• No performance characteristics documentation

---

Summary & Recommendations

Must Fix Before Merge:

1. ✅ Fix race condition in __setattr__ (First version #1)
2. ✅ Add thread safety for cache operations (Basic documentation #2)
3. ✅ Fix handler registration leak (Add pyporject.toml #3)
4. ✅ Add comprehensive tests (Move generated client into rapidata package #10)

Should Fix Before Merge:
5. Use logger instead of print (#4)
6. Add explicit directory creation (#5)
7. Document cache eviction policy (#6)
8. Set secure cache directory permissions (#13)

Consider for Future:
9. Improve cache timeout default (#12)
10. Better URL cache key (#14)
11. Add comprehensive documentation (#15)

---

Verdict

The feature is valuable and the implementation shows good architectural thinking, but the critical issues around thread safety and the race condition in config updates need to be addressed before merging. Additionally, the lack of tests is concerning for production code that manages persistent state.

Recommendation: Request changes for critical issues #1-#4 and #10.

[claude]

Pull Request Review: Persistent File Caching for Uploads

Summary

This PR adds persistent caching functionality for uploaded files using the diskcache library. The implementation stores uploaded file metadata on disk to avoid re-uploading files that haven't changed.

---

✅ Strengths

1. Good Architecture: The observer pattern implementation with register_upload_config_handler is clean and allows dynamic cache reconfiguration.

2. Comprehensive Cache Key Generation: The _get_cache_key method in _asset_uploader.py:65-75 properly includes environment, file path, size, and modification time, which should effectively detect file changes.

3. Configurable: All cache settings are exposed through UploadConfig with sensible defaults (100MB size limit, 10 shards, etc.).

4. Resource Management: The cache update handler properly closes the old cache before creating a new one (_asset_uploader.py:40-46).

---

🔴 Critical Issues

1. Race Condition in __setattr__ (Multiple Files)

Files: upload_config.py:45-47, logging_config.py:45-47

The __setattr__ override calls _notify_handlers() on every attribute set, including during __init__. This creates problems:

• During object construction, handlers are called before the object is fully initialized
• The handlers are called during super().__init__(**kwargs) in line 42, when not all fields may be set yet
• Multiple notifications fire unnecessarily when setting multiple attributes

Recommendation:

def __init__(self, **kwargs):
    self._initializing = True
    super().__init__(**kwargs)
    self._initializing = False
    self._notify_handlers()

def __setattr__(self, name: str, value) -> None:
    super().__setattr__(name, value)
    if not getattr(self, '_initializing', True):
        self._notify_handlers()

2. Missing Cache Invalidation on Error

File: _asset_uploader.py:77-100

If an upload succeeds but returns a corrupted/incorrect file_name, it gets cached. There's no mechanism to invalidate bad cache entries except manual clear_cache().

Recommendation: Add error handling or validation before caching:

if rapidata_config.upload.cacheUploads and response.file_name:
    self._shared_upload_cache[asset_key] = response.file_name

3. Unsafe Global State During Initialization

File: _asset_uploader.py:20-25

The class-level _shared_upload_cache is initialized when the module loads, reading from rapidata_config.upload before users can configure it. This means:

• The cache is created with default settings even if users want to customize them
• First instantiation determines cache location for all instances

Recommendation: Use lazy initialization:

_shared_upload_cache: FanoutCache | None = None

@classmethod
def _get_shared_cache(cls) -> FanoutCache:
    if cls._shared_upload_cache is None:
        config = rapidata_config.upload
        cls._shared_upload_cache = FanoutCache(...)
    return cls._shared_upload_cache

---

⚠️ Moderate Issues

4. No Cache Versioning

File: _asset_uploader.py:65-75

Cache keys don't include any version information. If the upload API response format changes or file processing logic changes, old cached values may be incompatible.

Recommendation: Add a version prefix to cache keys:

CACHE_VERSION = "v1"
return f"{CACHE_VERSION}:{env}@{asset}:{stat.st_size}:{stat.st_mtime_ns}"

5. Error Handling Could Be More Specific

File: _asset_uploader.py:62-63 and upload_config.py:54-55

Generic exception handling with pass or just logging may hide important errors.

Recommendation: Log the specific error type and consider whether certain exceptions should propagate.

6. Missing Documentation for Cache Behavior

Users should be informed about:

• Where cache files are stored (default: ~/.rapidata/upload_cache)
• How to clear the cache (the method exists but may not be documented)
• Cache size limits and eviction policy
• When cache is validated vs. used blindly

---

🟡 Minor Issues

7. Inconsistent Type Casting

File: _asset_uploader.py:86

The cast(str, cached_value) is only for the linter but doesn't provide runtime safety. The cache could theoretically contain non-string values.

Recommendation: Add runtime validation:

if cached_value is not None and isinstance(cached_value, str):
    return cached_value

8. Potential Performance Issue with mtime_ns

File: _asset_uploader.py:75

Using st_mtime_ns is precise but may cause unnecessary cache misses on filesystems that don't preserve nanosecond precision, or due to metadata-only updates.

Recommendation: Consider using st_mtime (seconds) or adding a flag for strict vs. lenient cache validation.

9. Missing Metrics/Observability

There's no way to track cache hit rate, which would be valuable for understanding if the cache is effective.

Recommendation: Add counters for cache hits/misses and expose them via a method or logs.

---

🔍 Security Considerations

10. No Path Traversal Protection

File: _asset_uploader.py:71-75

The code uses user-provided paths directly. While os.path.exists is called, there's no validation against path traversal attacks or accessing sensitive files.

Recommendation: Add path validation to ensure assets are within expected directories, especially if this SDK is used in web applications.

11. Cache Poisoning Risk

If an attacker can control the environment variable or filesystem timestamps, they could potentially serve malicious cached responses.

Recommendation: Consider adding integrity checks (e.g., hash validation) for cached entries.

---

📊 Test Coverage

Critical Gap: No tests were added for the new caching functionality. The following should be tested:

1. Cache hit when file hasn't changed
2. Cache miss when file content/mtime changes
3. Cache behavior across different environments
4. Cache reconfiguration via config updates
5. Cache clear functionality
6. Cache size limit enforcement
7. Concurrent access handling (FanoutCache should handle this, but verify)
8. Error handling when cache directory isn't writable

Recommendation: Add comprehensive unit tests in a new file like tests/unit/test_asset_uploader_cache.py.

---

📦 Dependencies

The addition of diskcache = "^5.6.3" and related dependencies (jupyter ecosystem) is appropriate. The library is well-maintained and suitable for persistent caching.

Note: The Jupyter dependencies (ipykernel, jupyter, etc.) appear to be added as side effects and should already be in dev dependencies, so this is fine.

---

🎯 Performance Considerations

1. Good: Using FanoutCache with shards should provide good concurrent access performance.
2. Good: The 100MB default size limit prevents unbounded cache growth.
3. Concern: The 0.1 second timeout (cacheTimeout: float = Field(default=0.1)) seems quite short and could cause cache misses under load. Consider documenting this or increasing it.

---

✨ Recommendations Summary

Must Fix Before Merge:

1. Fix the __setattr__ race condition in both config files
2. Add lazy initialization for the shared cache
3. Add basic test coverage for cache functionality

Should Fix:
4. Add cache versioning to keys
5. Improve error handling specificity
6. Add runtime type validation for cached values

Nice to Have:
7. Add cache metrics/observability
8. Document cache behavior for users
9. Add path validation for security
10. Consider cache integrity checks

---

📝 Code Quality

• Code style is consistent with the rest of the codebase
• Type hints are used appropriately
• Logging is well-placed
• The observer pattern is a good design choice

---

Final Verdict

This is a solid feature implementation with good architectural decisions, but it has some critical issues that should be addressed before merging:

Rating: 6.5/10 - Good concept and architecture, but needs fixes for thread safety and initialization order, plus tests.

Recommendation: Request changes to address the critical issues and add test coverage.

---

Review generated by Claude Code

[LinoGiger]

@claude do these peotry.lock changes seem right?