Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: fixing pdf parsing #3349

Merged
merged 3 commits into from
Oct 14, 2024
Merged

fix: fixing pdf parsing #3349

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
499119a
fix: fixing pdf parsing, by configuring the default pdf parser (Unstr…
jacopo-chevallard Oct 8, 2024
4da6f50
fix: parsing fallback
chloedia Oct 14, 2024
a6875d6
comments fix
chloedia Oct 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion backend/core/MegaParse/megaparse/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,6 @@ def from_yaml(cls, file_path: str):


class MegaparseConfig(MegaparseBaseConfig):
strategy: str = "fast"
strategy: str = "auto"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to check the impact of this on the worker

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chloedia This was set for a reason in megaparse ? The OCR took too much time ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, indeed the 'auto' option makes it significantly slower, even though, in principle, it should automatically revert to the 'fast' approach when possible... another (smarter) approach would be to first try with 'fast', then fallback to 'auto' if it fails... but still, we risk of parsing only partially some pdf files that would have been correctly (entirely) parsed with 'auto'

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(we should develop some heuristics on when a parsing has failed, besides the obvious case of an empty parsed document)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The heuristic I set up in previous project was to check the area of the "image" vs text in the page.
Parsers like fitz return the the bbox of image, I guess unstructured should return that. Nonnative text in pdfs is represented as an image. a threshold of 0.8 worked but it depends heavily on the document being processed.
Another heuristic is the existence of glyph encoding in the pdf. We should test that.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes normally it already fallbacks to auto when fast is empty (according to unstructured) that s why I used it !

llama_parse_api_key: str | None = None
pdf_parser: PdfParser = PdfParser.UNSTRUCTURED
1 change: 0 additions & 1 deletion backend/core/quivr_core/processor/implementations/megaparse_processor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,6 @@ def processor_metadata(self):
async def process_file_inner(self, file: QuivrFile) -> list[Document]:
mega_parse = MegaParse(file_path=file.path, config=self.megaparse_config) # type: ignore
document: Document = await mega_parse.aload()
print("\n\n document: ", document.page_content)
if len(document.page_content) > self.splitter_config.chunk_size:
docs = self.text_splitter.split_documents([document])
for doc in docs:
Expand Down
3 changes: 3 additions & 0 deletions backend/worker/quivr_worker/celery_monitor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,9 @@ def is_being_executed(task_name: str) -> bool:
running currently.
"""
active_tasks = celery.control.inspect().active()
if not active_tasks:
return False

for worker, running_tasks in active_tasks.items():
for task in running_tasks:
if task["name"] == task_name: # type: ignore
Expand Down
Loading