Skip to content

Fix divide by zero error - #2226

Merged
kunal-vaishnavi merged 2 commits into
mainfrom
asonawane/tokenizer-icm
Jun 18, 2026
Merged

Fix divide by zero error#2226
kunal-vaishnavi merged 2 commits into
mainfrom
asonawane/tokenizer-icm

Conversation

@apsonawane

Copy link
Copy Markdown
Contributor

This pull request improves error handling for the EncodeBatch function in the tokenizer by adding input validation and corresponding unit tests. The main goal is to prevent crashes when the function is called with empty input.

Error handling improvements:

  • src/models/model.cpp: Added a check in Tokenizer::EncodeBatch to throw a std::runtime_error if the input string list is empty, preventing a crash when called with no input strings.

Testing enhancements:

  • test/c_api_tests.cpp: Added a new test (EncodeBatchEmptyInputThrows) to verify that calling EncodeBatch with zero input strings throws a std::runtime_error instead of causing a crash.

Copilot AI review requested due to automatic review settings June 12, 2026 05:38
@apsonawane
apsonawane requested a review from a team as a code owner June 12, 2026 05:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens the tokenizer batch-encoding path used by the C/C++ API by adding input validation to prevent a divide-by-zero crash when EncodeBatch is called with an empty batch, and adds a regression test to ensure the error is surfaced as an exception.

Changes:

  • Add an early strings.empty() guard in Tokenizer::EncodeBatch(std::span<const char*>) to prevent divide-by-zero when computing the output tensor shape.
  • Add a C API regression test that verifies EncodeBatch(nullptr, 0) throws instead of crashing.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
src/models/model.cpp Adds empty-input validation to prevent a divide-by-zero when shaping the encoded batch tensor.
test/c_api_tests.cpp Adds a regression test ensuring empty-batch encode throws rather than crashing.

Comment thread src/models/model.cpp
Comment thread test/c_api_tests.cpp
Comment thread src/ort_genai_c.cpp
@kunal-vaishnavi
kunal-vaishnavi enabled auto-merge (squash) June 13, 2026 23:07
@kunal-vaishnavi
kunal-vaishnavi disabled auto-merge June 18, 2026 21:26
@kunal-vaishnavi
kunal-vaishnavi merged commit 6450690 into main Jun 18, 2026
62 of 70 checks passed
@kunal-vaishnavi
kunal-vaishnavi deleted the asonawane/tokenizer-icm branch June 18, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants