fix: fixing pdf parsing #3349
fix: fixing pdf parsing #3349
Conversation
…uctured) to use the 'auto' strategy instead of the 'fast' one
dosubot
bot
added
the
size:XS
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
dosubot
bot
added
the
area: backend
| @@ -22,6 +22,6 @@ def from_yaml(cls, file_path: str): | |||
|
|
|||
|
|
|||
| class MegaparseConfig(MegaparseBaseConfig): | |||
| strategy: str = "fast" | |||
| strategy: str = "auto" | |||
There was a problem hiding this comment.
Need to check the impact of this on the worker
There was a problem hiding this comment.
@chloedia This was set for a reason in megaparse ? The OCR took too much time ?
There was a problem hiding this comment.
good point, indeed the 'auto' option makes it significantly slower, even though, in principle, it should automatically revert to the 'fast' approach when possible... another (smarter) approach would be to first try with 'fast', then fallback to 'auto' if it fails... but still, we risk of parsing only partially some pdf files that would have been correctly (entirely) parsed with 'auto'
There was a problem hiding this comment.
(we should develop some heuristics on when a parsing has failed, besides the obvious case of an empty parsed document)
There was a problem hiding this comment.
The heuristic I set up in previous project was to check the area of the "image" vs text in the page.
Parsers like fitz return the the bbox of image, I guess unstructured should return that. Nonnative text in pdfs is represented as an image. a threshold of 0.8 worked but it depends heavily on the document being processed.
Another heuristic is the existence of glyph encoding in the pdf. We should test that.
There was a problem hiding this comment.
Yes normally it already fallbacks to auto when fast is empty (according to unstructured) that s why I used it !
dosubot
bot
added
size:M
🤖 I have created a release *beep* *boop* --- ## 0.0.322 (2024-10-15) ## What's Changed * feat: Add new documentation files by @StanGirard in #3351 * fix: separate english and french ingredients by @chloedia in #3358 * docs(core): init by @StanGirard in #3365 * docs: quivr core storage by @AmineDiro in #3366 * fix: fixing pdf parsing by @jacopo-chevallard in #3349 * feat: Improve user credit calculation in get_user_credits by @StanGirard in #3367 * fix unwanted parsing effect by @chloedia in #3371 * add fallback on llamaparse by @chloedia in #3374 **Full Changelog**: v0.0.321...v0.0.322 --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
🤖 I have created a release *beep* *boop* --- ## [0.0.17](core-0.0.16...core-0.0.17) (2024-10-16) ### Features * **assistant:** cdp ([#3305](#3305)) ([b767f19](b767f19)) * **assistants:** mock api ([#3195](#3195)) ([282fa0e](282fa0e)) * introducing configurable retrieval workflows ([#3227](#3227)) ([ef90e8e](ef90e8e)) ### Bug Fixes * fixing pdf parsing ([#3349](#3349)) ([367242a](367242a)) ### Documentation * **core:** init ([#3365](#3365)) ([bb572a2](bb572a2)) * **fix:** fixed warnings from griffe ([#3381](#3381)) ([1a38798](1a38798)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Our default pdf parser is Unstructured, for which we were using the 'fast' strategy, which fails in parsing some pdf. We instead use the 'auto' strategy which is more flexible and powerful.
Closes TICK-86
Description
Please include a summary of the changes and the related issue. Please also include relevant motivation and context.
Checklist before requesting a review
Please delete options that are not relevant.
Screenshots (if appropriate):