Skip to content

do not add EoD #3526

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jaredcasper merged 15 commits into from
Feb 21, 2026
Merged

do not add EoD #3526

Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
7891b3d
rebased main
arendu Feb 18, 2026
c6e9e3a
identity template and tool role mask
Oct 4, 2025
5ca350b
revert assert
Oct 8, 2025
ce6b5c4
Skip empty sequences and chunks in MTP tensor roll
rkarimimahab Dec 24, 2025
c979b29
add option for SFTTokenizer to build_tokenizers
arendu Feb 4, 2026
cabeda1
resolve conflicts between args.sft and args.hybrid_context_parallel
arendu Feb 4, 2026
d09521e
do not check for and add any EOD
arendu Feb 5, 2026
5905e8e
removed dup identity template
arendu Feb 18, 2026
9df1002
added from #2363
arendu Feb 18, 2026
a44049b
use core sft_tokenizer
arendu Feb 18, 2026
30f1600
fmt
arendu Feb 18, 2026
ba14c4d
commit from main
arendu Feb 20, 2026
735e6a3
checkout main
arendu Feb 20, 2026
78f74d8
feat: update scripts for perf. optimizations such as HybridEP, Numact…
seonjinn Feb 20, 2026
c41f533
eod not added
arendu Feb 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions megatron/training/datasets/sft_dataset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,10 +117,6 @@ def extend_with_padding(tokens, targets, positions, pad_len):
tokens_list = tokens.tolist()
targets_list = targets.tolist()

# Add EOD, unless it's already present
if tokens_list[-1] != eod:
tokens_list.append(eod)
targets_list.append(eod)

pack_tokens.extend(tokens_list)
pack_targets.extend(targets_list)
Expand Down
Loading