Convert to markdown function to include reading order as an option (so it can also be disabled)

[Pmmoks]

Hi, this is an amazing package. It's really good for processing PDF files. Massive thanks to the team! One BIG wish from me though is to have the natural reading order as an option. I have many PDF's where the natural reading order actually makes it worse (based on using get_text() from pymupdf) - it seems the automatic column identification isn't working for my documents. Would it be possible to add this as a feature / an option to the to_markdown() function?

[JorjMcKie]

This is a no-can-do sorry. Detecting / Establishing natural reading order is an integral part of the package making it impossible to disable it.

If it doesn't work in your case, it would instead be worthwhile to find and fix the problem.

[Pmmoks]

Thanks for the quick reply! Is there any way to supply some kind of column information e.g. if we know it has 2 columns? Or to manually split the pages ahead of calling this function?

I saw an old post you had made where you recommend this approach for the PyMuPDF TextPage.extractBLOCKS function.

Stackoverflow link to your old answer

[JorjMcKie]

Really old 😎!

If you know thins about your PDF, then there are ways to pour in information. For example, you can add a middle line to the pages (in a temporary way) to help the column recognition process.
Maybe you want to share a PDF with a handful of pages, so I can try a few things and demo an approach.
Always welcome!

[Pmmoks]

Thanks for being so helpful! Here are a few pages from a pdf with 2 columns. I can already fairly accurately determine whether it’s a single page or double page spread, but I’m struggling more with getting the .to_markdown function to recognise that.

pillar-3-disclosure-2022-part-2.pdf

[JorjMcKie]

With the newest version 0.0.17 at least most of the pages are processed correctly BTW.

[Pmmoks]

Thanks @JorjMcKie! I’ve tested it and indeed it does work better in terms of reading order. Although some form of manual control to split pages would be awesome for the remaining pages.

If you have any more suggestions I’d love to try them!