feat(embeddings): embed email and paragraph meaning units

[cursor]

Buyer next action

POST /v1/batch/embeddings with the raw invoice email and "chunking_strategy": "meaning_units". Read chunk_units in the completed document. Search those units for the invoice id — the greeting is a separate vector. Omit the field (or send JSON null) to keep the naruon one-vector-per-input contract.

Why

Token-budget map/reduce still averages provider parts of one input. That hides INV-20260816 / 1840.00 USD inside “Good morning.” This slice cuts email parties, HTML blocks, data:image spans (with source offset), and paragraphs before that path. Similarity-breakpoint chunking is not used (Qu et al., 2025).

Unknown chunking_strategy values return 400 invalid_chunking_strategy.

Tests

python3 tests/test_meaning_unit_chunking.py
python3 tests/test_embeddings_meaning_units_http_honesty.py
python3 tests/test_batch_embeddings.py

Invoice query ranks the balance paragraph first. The naruon fixture is unchanged when the field is omitted.

References

Zhao, J., Ji, Z., Ye, Y., Feng, X., Zhang, X., & Rong, C. (2024). Meta-chunking: Learning text segmentation and semantic completion via logical perception. arXiv. https://doi.org/10.48550/arXiv.2410.12788

Qu, R., Tu, R., & Bao, F. (2025). Is semantic chunking worth the computational cost? In Findings of the Association for Computational Linguistics: NAACL 2025 (pp. 2012–2027). Association for Computational Linguistics. https://aclanthology.org/2025.findings-naacl.114/

Unicode Consortium. (2024). Unicode Standard Annex #29: Unicode text segmentation. https://www.unicode.org/reports/tr29/

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, 33. https://doi.org/10.48550/arXiv.2005.11401

Independent non-author APPROVE is required. This automation will not self-approve or merge.

  

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[cursor]

Verdict

Unique 3f94151 is the meaning-unit landing. Prefer this head over #643. This is a COMMENT, not an approval.

Buyer next action

POST /v1/batch/embeddings with the raw invoice email and "chunking_strategy": "meaning_units". Read chunk_units in the completed document. Search those units for the invoice id — the greeting is a separate vector. Omit the field (or send JSON null) to keep the naruon one-vector-per-input contract.

Independent reviewer (not this automation): APPROVE after Full unit and Semgrep are green. Do not merge #643 in parallel. Do not open a second meaning-unit chunking PR.

What was checked

• semantic_chunking.meaning_unit_chunks cuts email parties, HTML blocks, data:image spans (with source offset), and paragraphs/sentences. Similarity-breakpoint chunking is not used (Qu et al., 2025).
• Real invoice tests bind retrieval: balance line ranks first; greeting is isolated; Korean sentence is its own unit; unknown strategy is 400 invalid_chunking_strategy.
• Omit/null keeps the one-vector-per-input contract.

Residual (do not fold onto this unique)

• Live NIM OCR / object tags on embedded_image (opt-in NVIDIA_NIM_API_KEY).
• Persist 3NF meaning_unit / unit_embedding tables when a store is selected.

Do not self-approve.

  

_{Sent by Cursor Automation: Fix Issues}

[cursor]

REQUEST_CHANGES (COMMENT event: GitHub rejects REQUEST_CHANGES on cursor[bot] PRs).

Opt-in chunking_strategy=meaning_units and the naruon omit/null path are honest. The buyer claim this slice sells — raw HTML/email/data:image bodies become separately searchable greeting vs invoice vectors — fails on the shapes naruon actually sends. Probed on 3f94151.

Do this next: do not merge #652 as-is. A repair successor on this head will keep innermost HTML leaves and RFC 2397 image payloads (charset params, URL-safe -_, MIME line wraps) so AP search can rank INV-20260816 without the greeting. Independent non-author APPROVE is still required after that lands. This automation will not self-approve or merge.

Findings

1. Wrapper HTML glues greeting to invoice. _HTML_BLOCK reserves the first matching div/section/article and consumes every inner <p>. <div><p>Good morning</p><p>Invoice INV-20260816 …</p></div> yields n_units=1. Outlook <tr>/<td> happens to split; Gmail wrappers do not. INVOICE_HTML is two unwrapped sibling <p> tags, so CI cannot catch this.
2. _IMAGE is the one-line textbook URI. data:image/png;charset=utf-8;base64,… is not detected. URL-safe -_ truncates the span and leaves INV-… glued to leftover payload. MIME-wrapped base64 (newline at 76) keeps the first line as embedded_image and concatenates the rest onto the invoice paragraph. Stored source_offset/source_length then do not cover the real image.

Unknown/non-string chunking_strategy already fail closed. Omit/null still omits chunk_units and keeps one vector per input.

Do not change on this slice

• naruon omit/null contract
• expansion before token-budget map/reduce
• chunk_text == source[offset:offset+length]
• no redact_value on embeddings _send (From/To stay operational)
• no live NIM OCR, Responses SSE, or KV allowlist folded here

Residual after the repair: opt-in NIM OCR/image_object_tag via KV; persist 3NF meaning_unit / unit_embedding / embedded_image when a store is selected.

  

_{Sent by Cursor Automation: Fix Issues}

[cursor]

Verdict: landing vehicle for meaning-unit embeddings. Do not APPROVE from this run.

#643 (ec88eea) always-on splits and averages unit vectors into the naruon embeddings[] slot. This PR keeps the token-budget path, gates expansion on chunking_strategy=meaning_units, and fail-closes unknown values. That is the correct commercial contract.

Close #643. Independent non-author APPROVE is still required here. This run will not self-approve or merge.

What is correct

• Omit / JSON null / whitespace → no chunk_units, len(embeddings) == len(inputs).
• meaning_units expands email parties, paragraphs, HTML blocks, and data:image spans; HTTP test ranks INV-20260816 / 1840.00 above “Good morning.”
• _force_token_safe_chunks is kept. Qu et al. (2025) is the right cost paper. Header detector uses a closed set and _looks_like_email.

Residuals (fix on this tip, do not open a third meaning-unit PR)

1. HTML block regex is a backtracking scan. <(p|div|…)>.*?</\1> with DOTALL is the only quadratic seam on untrusted input. Prefer a linear tag walk. Do not copy #643’s _pack_spans (first oversized sentence is emitted intact — reproduced at 21 tokens against max_tokens=3).
2. _EMAIL_HEADER.finditer runs on the whole document after a leading mail block is detected. A body line Subject: see attached SKU becomes email_subject. Restrict header matches to the leading block _looks_like_email already walked.
3. expand_embedding_inputs accepts source_document; HTTP _validate_chunking_strategy 400s it. Either document it as internal-only or accept it as omit-equivalent.
4. Opt-in still embeds raw data:image base64. Keep the span. Do not treat base64 as a retrieval unit until a NIM OCR/object-tag adapter exists (NVIDIA_NIM_API_KEY, not COPILOT_GITHUB_TOKEN).

Next action

Close #643. Address residuals 1–2 on this tip. Then wait for seonghobae (or another human) APPROVE. Do not fold honesty-stack PRs onto this branch.

  

_{Sent by Cursor Automation: Fix Issues}

[seonghobae]

Repair successor is #665 (ef4370e). Do not merge this head. #665 keeps innermost HTML leaves and RFC 2397 data:image payloads (charset, URL-safe -_, MIME wraps). Independent non-author APPROVE is still required on #665. This automation will not self-approve or merge.

[opencode-agent]

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

• Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

• Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

• Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

• Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

• Result: REQUEST_CHANGES

• Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head e80c2a120467d7a70d9069e27115d4dd3789a4dc.

• Head SHA: e80c2a120467d7a70d9069e27115d4dd3789a4dc

• Workflow run: 32166933842

• Workflow attempt: 1

Coverage evidence

Coverage evidence job did not run or did not publish coverage evidence.

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (10 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (10 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs (10 files)"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs (10 files)"]
  R2 --> V2["docs review"]
  Evidence --> S3["Test (6 files)"]
  S3 --> I3["regression suite"]
  I3 --> R3["Review risk: Test (6 files)"]
  R3 --> V3["targeted test run"]

Loading

[opencode-agent]

OpenCode Review Overview

• Head SHA: e80c2a120467d7a70d9069e27115d4dd3789a4dc
• Workflow run: 32166933842
• Workflow attempt: 1
• Gate result: REQUEST_CHANGES (approval step)

Pull request overview

OpenCode cannot approve yet because required coverage evidence did not pass.

Review outcome

1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence

• Problem: The required coverage-evidence job result was failure, so OpenCode cannot establish approval sufficiency for this head.

• Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.

• Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports success with required evidence or explicit no-source not-applicable evidence.

• Regression test: Keep the approval branch checking needs.coverage-evidence.result == success before posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present.

• Result: REQUEST_CHANGES

• Reason: coverage-evidence result was failure, so required test/docstring evidence was not proven for current head e80c2a120467d7a70d9069e27115d4dd3789a4dc.

• Head SHA: e80c2a120467d7a70d9069e27115d4dd3789a4dc

• Workflow run: 32166933842

• Workflow attempt: 1

Coverage evidence

Coverage evidence job did not run or did not publish coverage evidence.

Changed-File Evidence Map

flowchart LR
  PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
  Evidence --> S1["Changed file (10 files)"]
  S1 --> I1["repository behavior"]
  I1 --> R1["Review risk: Changed file (10 files)"]
  R1 --> V1["required checks"]
  Evidence --> S2["Docs (10 files)"]
  S2 --> I2["operator or user guidance"]
  I2 --> R2["Review risk: Docs (10 files)"]
  R2 --> V2["docs review"]
  Evidence --> S3["Test (6 files)"]
  S3 --> I3["regression suite"]
  I3 --> R3["Review risk: Test (6 files)"]
  R3 --> V3["targeted test run"]

Loading