Error handling #71
Error handling #71
Conversation
| async for chunk in self.read_chunks(file): | ||
| yield CollectedBytes(file=file_path, data=chunk, error=None) | ||
| except PermissionError as exc: | ||
| print(f"Unable to open this file {file_path}, getting error as {exc}") |
There was a problem hiding this comment.
We should raise exceptions we fail file read and or permission most of the time it will be issue with user
| try: | ||
| self.client = storage.Client.from_service_account_info(self.credentials) | ||
| except FileNotFoundError as exc: | ||
| raise common_errors.FileNotFoundError( |
There was a problem hiding this comment.
This should be connection error or ( failed to initial GCS bucket client)
| # Handle exceptions gracefully, e.g., log the error | ||
| print(f"An error occurred while streaming blob: {e}") | ||
| except PermissionError as exc: | ||
| print(f"Unable to open this file {blob_file}, getting error as {exc}") |
There was a problem hiding this comment.
Same if we can't get to file there is issue with container and we should stop that specific collector instead of doing a partial pull with missing files
| except PermissionError as exc: | ||
| print(f"Unable to open this file {blob_file}, getting error as {exc}") | ||
| except OSError as exc: | ||
| print(f"Getting OS Error on file {blob_file}, as {exc}") |
There was a problem hiding this comment.
We should also handle generic Exception as last option which will catch anything we cant predict so there will be an extra catch for just Exception
| processed_text = await self.process_data(text) | ||
| yield processed_text | ||
| except Exception as exc: | ||
| yield "" |
There was a problem hiding this comment.
Let's yield None here
There was a problem hiding this comment.
Instead of empty, and possibly at other places, that way at the queue side we will know that file ended and new started
Ansh5461
force-pushed
the
error-handling
branch
from
7c9e684 to
3a55a87
Compare
No description provided.