Skip to content

Fix padded type_ids in Python encode paths - #34

Merged
AlonKejzman merged 1 commit into
crusoecloud:mainfrom
grp06:fix-padding-type-ids
May 24, 2026
Merged

Fix padded type_ids in Python encode paths#34
AlonKejzman merged 1 commit into
crusoecloud:mainfrom
grp06:fix-padding-type-ids

Conversation

@grp06

@grp06 grp06 commented May 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Route Python tokenizer padding through PyEncoding::pad so padded type_ids use the configured pad_type_id.
  • Remove the duplicated ids/mask-only padding helper.
  • Add right and left padding unit coverage for the shared encoding builder.

Tests

  • cargo test -p fastokens-python encoding_pad_applies_pad_type_id
  • cargo test -p fastokens-python encode_batch_pad_type_id_applied_to_type_ids
  • cargo test -p fastokens-python build_encoding_left_padding_applies_pad_type_id
  • cargo test -p fastokens-python --lib
  • cargo fmt --check
  • cargo clippy -- -D warnings
  • cargo test

@AlonKejzman

Copy link
Copy Markdown
Collaborator

Hello, thank you for the PR! Will look at it soon.

@AlonKejzman
AlonKejzman merged commit 2d64ec1 into crusoecloud:main May 24, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants