Skip to content

feat: aling idoctags to iso #457

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
PeterStaar-IBM wants to merge 17 commits into
base: main
Choose a base branch
from
+5,434 −316
Open

feat: aling idoctags to iso #457

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
d983a6f
Working on ISO standard for IDocTags
PeterStaar-IBM Dec 11, 2025
4de44f1
fixed the pre-commit, still tons of work to do
PeterStaar-IBM Dec 11, 2025
6a4653d
updated idoctag token class
PeterStaar-IBM Dec 11, 2025
b7c641d
need to start fixing the bugs and add proper tests for idoctags
PeterStaar-IBM Dec 11, 2025
312ebe9
fixed the tests
PeterStaar-IBM Dec 12, 2025
0e6173c
refactored the testing for dcotags and idoctags
PeterStaar-IBM Dec 12, 2025
607441e
updating the location tokens in idoctags
PeterStaar-IBM Dec 12, 2025
77e9914
removed the DocumentToken
PeterStaar-IBM Dec 12, 2025
98ed500
fixed the idt quot issue
PeterStaar-IBM Dec 12, 2025
bac016a
reformatted
PeterStaar-IBM Dec 12, 2025
95b9219
Added IDocTagsSerializationMode
PeterStaar-IBM Dec 12, 2025
6d2c7f2
working on the deserializer
PeterStaar-IBM Dec 12, 2025
09c4f9a
fixed the captions for floating items
PeterStaar-IBM Dec 12, 2025
3c706e2
still need to fix some seserialization tests
PeterStaar-IBM Dec 12, 2025
75c92f8
finally all is working and reformatted
PeterStaar-IBM Dec 12, 2025
5b9a1dc
made the OTSL serialization and deserialization self-contained in IDo…
PeterStaar-IBM Dec 14, 2025
cc7f2a4
fixed some unclean code
PeterStaar-IBM Dec 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2,104 changes: 1,919 additions & 185 deletions docling_core/experimental/idoctags.py
View file
Open in desktop

Large diffs are not rendered by default.

430 changes: 430 additions & 0 deletions examples/convert_to_idoctags.py
View file
Open in desktop

Large diffs are not rendered by default.

5 changes: 4 additions & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,10 @@ chunking-openai = [
# specific:
'tiktoken (>=0.9.0,<0.13.0)',
]

examples = [
"datasets>=4.0.0",
"matplotlib>=3.7.0",
]
[dependency-groups]
dev = [
"pre-commit~=3.7",
Expand Down
250 changes: 250 additions & 0 deletions test/data/doc/ddoc_0.v0.gt.idt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,250 @@
<doctag version="1.0.0">
<text>
<location value="172" resolution="512"/>
<location value="24" resolution="512"/>
<location value="355" resolution="512"/>
<location value="31" resolution="512"/>
ndbinfo_select_all - Select From ndbinfo Tables
</text>
<floating_group class="table">
<otsl>
<location value="93" resolution="512"/>
<location value="47" resolution="512"/>
<location value="482" resolution="512"/>
<location value="79" resolution="512"/>
<rhed/>
Default Value
<fcel/>
1
<nl/>
<rhed/>
Minimum Value
<fcel/>
0
<nl/>
<rhed/>
Maximum Value
<fcel/>
MAX_INT
<nl/>
</otsl>
</floating_group>
<text>
<location value="94" resolution="512"/>
<location value="87" resolution="512"/>
<location value="479" resolution="512"/>
<location value="94" resolution="512"/>
This option sets the number of times to execute the select. Use --delay to set the time between loops.
</text>
<list ordered="false">
<list_text>
<location value="86" resolution="512"/>
<location value="101" resolution="512"/>
<location value="189" resolution="512"/>
<location value="108" resolution="512"/>
• --ndb-connectstring
</list_text>
</list>
<floating_group class="table">
<otsl>
<location value="93" resolution="512"/>
<location value="121" resolution="512"/>
<location value="482" resolution="512"/>
<location value="153" resolution="512"/>
<rhed/>
Command-Line Format
<fcel/>
--ndb-connectstring=connection-string
<nl/>
<rhed/>
Type
<fcel/>
String
<nl/>
<rhed/>
Default Value
<fcel/>
[none]
<nl/>
</otsl>
</floating_group>
<text>
<location value="94" resolution="512"/>
<location value="161" resolution="512"/>
<location value="478" resolution="512"/>
<location value="176" resolution="512"/>
Set connect string for connecting to ndb_mgmd. Syntax: &quot;[nodeid=id;][host=]hostname[:port]&quot;. Overrides entries in NDB_CONNECTSTRING and my.cnf.
</text>
<list ordered="false">
<list_text>
<location value="86" resolution="512"/>
<location value="184" resolution="512"/>
<location value="169" resolution="512"/>
<location value="190" resolution="512"/>
• --ndb-mgmd-host
</list_text>
</list>
<floating_group class="table">
<otsl>
<location value="93" resolution="512"/>
<location value="203" resolution="512"/>
<location value="482" resolution="512"/>
<location value="236" resolution="512"/>
<rhed/>
Command-Line Format
<fcel/>
--ndb-mgmd-host=connection-string
<nl/>
<rhed/>
Type
<fcel/>
String
<nl/>
<rhed/>
Default Value
<fcel/>
[none]
<nl/>
</otsl>
</floating_group>
<text>
<location value="94" resolution="512"/>
<location value="244" resolution="512"/>
<location value="227" resolution="512"/>
<location value="251" resolution="512"/>
Same as --ndb-connectstring .
</text>
<list ordered="false">
<list_text>
<location value="85" resolution="512"/>
<location value="259" resolution="512"/>
<location value="154" resolution="512"/>
<location value="263" resolution="512"/>
• --ndb-nodeid
</list_text>
</list>
<floating_group class="table">
<otsl>
<location value="93" resolution="512"/>
<location value="278" resolution="512"/>
<location value="482" resolution="512"/>
<location value="310" resolution="512"/>
<rhed/>
Command-Line Format
<fcel/>
--ndb-nodeid=#
<nl/>
<rhed/>
Type
<fcel/>
Integer
<nl/>
<rhed/>
Default Value
<fcel/>
[none]
<nl/>
</otsl>
</floating_group>
<text>
<location value="94" resolution="512"/>
<location value="319" resolution="512"/>
<location value="355" resolution="512"/>
<location value="325" resolution="512"/>
Set node ID for this node, overriding any ID set by --ndb-connectstring.
</text>
<list ordered="false">
<list_text>
<location value="86" resolution="512"/>
<location value="333" resolution="512"/>
<location value="245" resolution="512"/>
<location value="338" resolution="512"/>
• --ndb-optimized-node-selection
</list_text>
</list>
<floating_group class="table">
<otsl>
<location value="94" resolution="512"/>
<location value="353" resolution="512"/>
<location value="483" resolution="512"/>
<location value="363" resolution="512"/>
<rhed/>
Command-Line Format
<fcel/>
--ndb-optimized-node-selection
<nl/>
</otsl>
</floating_group>
<text>
<location value="94" resolution="512"/>
<location value="372" resolution="512"/>
<location value="461" resolution="512"/>
<location value="386" resolution="512"/>
Enable optimizations for selection of nodes for transactions. Enabled by default; use --skip-ndb- optimized-node-selection to disable.
</text>
<list ordered="false">
<list_text>
<location value="85" resolution="512"/>
<location value="394" resolution="512"/>
<location value="158" resolution="512"/>
<location value="399" resolution="512"/>
• --no-defaults
</list_text>
</list>
<floating_group class="table">
<otsl>
<location value="93" resolution="512"/>
<location value="414" resolution="512"/>
<location value="482" resolution="512"/>
<location value="425" resolution="512"/>
<rhed/>
Command-Line Format
<fcel/>
--no-defaults
<nl/>
</otsl>
</floating_group>
<text>
<location value="94" resolution="512"/>
<location value="433" resolution="512"/>
<location value="342" resolution="512"/>
<location value="439" resolution="512"/>
Do not read default options from any option file other than login file.
</text>
<list ordered="false">
<list_text>
<location value="86" resolution="512"/>
<location value="448" resolution="512"/>
<location value="173" resolution="512"/>
<location value="454" resolution="512"/>
• --print-defaults
</list_text>
</list>
<floating_group class="table">
<otsl>
<location value="94" resolution="512"/>
<location value="467" resolution="512"/>
<location value="482" resolution="512"/>
<location value="478" resolution="512"/>
<rhed/>
Command-Line Format
<fcel/>
--print-defaults
<nl/>
</otsl>
</floating_group>
<text>
<location value="94" resolution="512"/>
<location value="486" resolution="512"/>
<location value="228" resolution="512"/>
<location value="493" resolution="512"/>
Print program argument list and exit.
</text>
<text>
<location value="463" resolution="512"/>
<location value="482" resolution="512"/>
<location value="482" resolution="512"/>
<location value="487" resolution="512"/>
4253
</text>
</doctag>
Loading
Loading