Skip to content

feat: idoctags serialization and deserialization matching the iso proposal #457

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
PeterStaar-IBM merged 29 commits into from
Dec 17, 2025
Merged

feat: idoctags serialization and deserialization matching the iso proposal #457

Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
d983a6f
Working on ISO standard for IDocTags
PeterStaar-IBM Dec 11, 2025
4de44f1
fixed the pre-commit, still tons of work to do
PeterStaar-IBM Dec 11, 2025
6a4653d
updated idoctag token class
PeterStaar-IBM Dec 11, 2025
b7c641d
need to start fixing the bugs and add proper tests for idoctags
PeterStaar-IBM Dec 11, 2025
312ebe9
fixed the tests
PeterStaar-IBM Dec 12, 2025
0e6173c
refactored the testing for dcotags and idoctags
PeterStaar-IBM Dec 12, 2025
607441e
updating the location tokens in idoctags
PeterStaar-IBM Dec 12, 2025
77e9914
removed the DocumentToken
PeterStaar-IBM Dec 12, 2025
98ed500
fixed the idt quot issue
PeterStaar-IBM Dec 12, 2025
bac016a
reformatted
PeterStaar-IBM Dec 12, 2025
95b9219
Added IDocTagsSerializationMode
PeterStaar-IBM Dec 12, 2025
6d2c7f2
working on the deserializer
PeterStaar-IBM Dec 12, 2025
09c4f9a
fixed the captions for floating items
PeterStaar-IBM Dec 12, 2025
3c706e2
still need to fix some seserialization tests
PeterStaar-IBM Dec 12, 2025
75c92f8
finally all is working and reformatted
PeterStaar-IBM Dec 12, 2025
5b9a1dc
made the OTSL serialization and deserialization self-contained in IDo…
PeterStaar-IBM Dec 14, 2025
cc7f2a4
fixed some unclean code
PeterStaar-IBM Dec 14, 2025
ce5f1b7
did some fixes and clean up
PeterStaar-IBM Dec 16, 2025
0cf8fc3
added extra tests, expanded to deserialize to deal with nested lists
PeterStaar-IBM Dec 16, 2025
16b965f
added the deserialization for formatting
PeterStaar-IBM Dec 16, 2025
e132f04
fixed the formatting and the complex inline groups in nested lists
PeterStaar-IBM Dec 16, 2025
20bc4a0
added get_category
PeterStaar-IBM Dec 16, 2025
2e779a9
make IS_SELFCLOSING a set
PeterStaar-IBM Dec 16, 2025
ce45c76
removed the regex
PeterStaar-IBM Dec 16, 2025
b8e85bb
fixed the if ... return into if ... elif ... return
PeterStaar-IBM Dec 17, 2025
33954cf
fixed the facets parsing
PeterStaar-IBM Dec 17, 2025
3ee7dd1
fixed the footnotes for Tables and Pictures in IDocTags
PeterStaar-IBM Dec 17, 2025
f56e5bf
updated with latest doctags token table
PeterStaar-IBM Dec 17, 2025
4184ad3
removed private name
PeterStaar-IBM Dec 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2,352 changes: 2,156 additions & 196 deletions docling_core/experimental/idoctags.py
View file
Open in desktop

Large diffs are not rendered by default.

696 changes: 696 additions & 0 deletions examples/convert_to_idoctags.py
View file
Open in desktop

Large diffs are not rendered by default.

6 changes: 6 additions & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,12 @@ chunking-openai = [
# specific:
'tiktoken (>=0.9.0,<0.13.0)',
]
examples = [
"datasets>=4.0.0",
"matplotlib>=3.7.0",
"openpyxl>=3.1.5",
"pandas>=2.3.3",
]

[dependency-groups]
dev = [
Expand Down
250 changes: 250 additions & 0 deletions test/data/doc/ddoc_0.v0.gt.idt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,250 @@
<doctag version="1.0.0">
<text>
<location value="172" resolution="512"/>
<location value="24" resolution="512"/>
<location value="355" resolution="512"/>
<location value="31" resolution="512"/>
ndbinfo_select_all - Select From ndbinfo Tables
</text>
<floating_group class="table">
<otsl>
<location value="93" resolution="512"/>
<location value="47" resolution="512"/>
<location value="482" resolution="512"/>
<location value="79" resolution="512"/>
<rhed/>
Default Value
<fcel/>
1
<nl/>
<rhed/>
Minimum Value
<fcel/>
0
<nl/>
<rhed/>
Maximum Value
<fcel/>
MAX_INT
<nl/>
</otsl>
</floating_group>
<text>
<location value="94" resolution="512"/>
<location value="87" resolution="512"/>
<location value="479" resolution="512"/>
<location value="94" resolution="512"/>
This option sets the number of times to execute the select. Use --delay to set the time between loops.
</text>
<list ordered="false">
<list_text>
<location value="86" resolution="512"/>
<location value="101" resolution="512"/>
<location value="189" resolution="512"/>
<location value="108" resolution="512"/>
• --ndb-connectstring
</list_text>
</list>
<floating_group class="table">
<otsl>
<location value="93" resolution="512"/>
<location value="121" resolution="512"/>
<location value="482" resolution="512"/>
<location value="153" resolution="512"/>
<rhed/>
Command-Line Format
<fcel/>
--ndb-connectstring=connection-string
<nl/>
<rhed/>
Type
<fcel/>
String
<nl/>
<rhed/>
Default Value
<fcel/>
[none]
<nl/>
</otsl>
</floating_group>
<text>
<location value="94" resolution="512"/>
<location value="161" resolution="512"/>
<location value="478" resolution="512"/>
<location value="176" resolution="512"/>
Set connect string for connecting to ndb_mgmd. Syntax: &quot;[nodeid=id;][host=]hostname[:port]&quot;. Overrides entries in NDB_CONNECTSTRING and my.cnf.
</text>
<list ordered="false">
<list_text>
<location value="86" resolution="512"/>
<location value="184" resolution="512"/>
<location value="169" resolution="512"/>
<location value="190" resolution="512"/>
• --ndb-mgmd-host
</list_text>
</list>
<floating_group class="table">
<otsl>
<location value="93" resolution="512"/>
<location value="203" resolution="512"/>
<location value="482" resolution="512"/>
<location value="236" resolution="512"/>
<rhed/>
Command-Line Format
<fcel/>
--ndb-mgmd-host=connection-string
<nl/>
<rhed/>
Type
<fcel/>
String
<nl/>
<rhed/>
Default Value
<fcel/>
[none]
<nl/>
</otsl>
</floating_group>
<text>
<location value="94" resolution="512"/>
<location value="244" resolution="512"/>
<location value="227" resolution="512"/>
<location value="251" resolution="512"/>
Same as --ndb-connectstring .
</text>
<list ordered="false">
<list_text>
<location value="85" resolution="512"/>
<location value="259" resolution="512"/>
<location value="154" resolution="512"/>
<location value="263" resolution="512"/>
• --ndb-nodeid
</list_text>
</list>
<floating_group class="table">
<otsl>
<location value="93" resolution="512"/>
<location value="278" resolution="512"/>
<location value="482" resolution="512"/>
<location value="310" resolution="512"/>
<rhed/>
Command-Line Format
<fcel/>
--ndb-nodeid=#
<nl/>
<rhed/>
Type
<fcel/>
Integer
<nl/>
<rhed/>
Default Value
<fcel/>
[none]
<nl/>
</otsl>
</floating_group>
<text>
<location value="94" resolution="512"/>
<location value="319" resolution="512"/>
<location value="355" resolution="512"/>
<location value="325" resolution="512"/>
Set node ID for this node, overriding any ID set by --ndb-connectstring.
</text>
<list ordered="false">
<list_text>
<location value="86" resolution="512"/>
<location value="333" resolution="512"/>
<location value="245" resolution="512"/>
<location value="338" resolution="512"/>
• --ndb-optimized-node-selection
</list_text>
</list>
<floating_group class="table">
<otsl>
<location value="94" resolution="512"/>
<location value="353" resolution="512"/>
<location value="483" resolution="512"/>
<location value="363" resolution="512"/>
<rhed/>
Command-Line Format
<fcel/>
--ndb-optimized-node-selection
<nl/>
</otsl>
</floating_group>
<text>
<location value="94" resolution="512"/>
<location value="372" resolution="512"/>
<location value="461" resolution="512"/>
<location value="386" resolution="512"/>
Enable optimizations for selection of nodes for transactions. Enabled by default; use --skip-ndb- optimized-node-selection to disable.
</text>
<list ordered="false">
<list_text>
<location value="85" resolution="512"/>
<location value="394" resolution="512"/>
<location value="158" resolution="512"/>
<location value="399" resolution="512"/>
• --no-defaults
</list_text>
</list>
<floating_group class="table">
<otsl>
<location value="93" resolution="512"/>
<location value="414" resolution="512"/>
<location value="482" resolution="512"/>
<location value="425" resolution="512"/>
<rhed/>
Command-Line Format
<fcel/>
--no-defaults
<nl/>
</otsl>
</floating_group>
<text>
<location value="94" resolution="512"/>
<location value="433" resolution="512"/>
<location value="342" resolution="512"/>
<location value="439" resolution="512"/>
Do not read default options from any option file other than login file.
</text>
<list ordered="false">
<list_text>
<location value="86" resolution="512"/>
<location value="448" resolution="512"/>
<location value="173" resolution="512"/>
<location value="454" resolution="512"/>
• --print-defaults
</list_text>
</list>
<floating_group class="table">
<otsl>
<location value="94" resolution="512"/>
<location value="467" resolution="512"/>
<location value="482" resolution="512"/>
<location value="478" resolution="512"/>
<rhed/>
Command-Line Format
<fcel/>
--print-defaults
<nl/>
</otsl>
</floating_group>
<text>
<location value="94" resolution="512"/>
<location value="486" resolution="512"/>
<location value="228" resolution="512"/>
<location value="493" resolution="512"/>
Print program argument list and exit.
</text>
<text>
<location value="463" resolution="512"/>
<location value="482" resolution="512"/>
<location value="482" resolution="512"/>
<location value="487" resolution="512"/>
4253
</text>
</doctag>
Loading
Loading