blt wip by itazap · Pull Request #38579 · huggingface/transformers

blt wip

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

cpu version

itazap

authored and

ita.zaporozhets@huggingface.co

committed

cpu friendly with full entropy model (real time patching)

itazap

authored and

ita.zaporozhets@huggingface.co

committed

adding config file instead of args file

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

enable MPS

LysandreJik

authored and

ita.zaporozhets@huggingface.co

committed

refactoring unused code

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

single config class in config file

ita.zaporozhets@huggingface.co

committed

inherit from PreTrainedModel

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

refactor LMTransformer --> BLTPatcher

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

add conversion script

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

load from new checkpoing with form_pretrained

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

fixed demo from_pretrained

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

clean up

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

clean a few comments

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

cleanup folder

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

clean up dir

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

cleaned up modeling further

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

rename classes

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

adding transformers Attention class and RotaryEmbedding class

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

exchanged blt modules for transformers modules: attention, rotary_emb, create_causal_mask, etc

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

seperate out patcher config, update modeling and conversion script

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

rename vars to be more transformers-like

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

rm unused functions

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

adding cross attention from transformers

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

pass arg

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

rename weights

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

updated conversion script

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

overwritten commit! fixing PR

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

apply feedback

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

adding BLTRMSNorm like Llama

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

add repeat_kv and eager_attention_forward copied from

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

BLTMLP identical to MllamTextMLP

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

clean up some args'

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

more like mllama, but busier inits

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

BLTTransformerLayer config

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

decoder, encoder, global configs

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

wip working on modular file

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

cleaning up patch and configs

ita.zaporozhets@huggingface.co

committed

clean up patcher helpers

ita.zaporozhets@huggingface.co

committed

clean up patcher helpers further

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

clean up

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

some config renaming

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

clean up unused configs

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

clean up configs

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

clean up configs

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

update modular

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

clean

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

update demo

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

config more like mllama, seperated subconfigs from subdicts

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

read from config instead of self args

ita.zaporozhets@huggingface.co

committed

update demo file

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

model weights to causal lm weights

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

missed file

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

added tied weights keys

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

BLTForCausalLM

ita.zaporozhets@huggingface.co

committed

adding files after add-new-model-like

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

update demo

ita.zaporozhets@huggingface.co

committed

working on tests

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

first running integration tests

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

added integration tests

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

adding tokenization tests, integration tests, and cleaned up tokenization file, + ruff

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

tokenizer clean up

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

modular file

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

fixing rebase

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

ruff

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

adding correct basemodel output and updating config with checkpoint vals (for testing)

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

BLTModelTests git status

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

enabling inputs_embeds, although won't be equal to input_ids since need ids for patching logic

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

fix sdpa == causal tests

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

fix small model test and some gradient checkpointing

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

skip training GC tests

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

fix test

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

updated modular

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

update modular

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

ruff

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

adding modular + modeling

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

modular

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

more modern is_casual check

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

cleaning up modular

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

more modular reduction

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

ruff

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

modular fix

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

fix styling

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

return 2

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

return 2

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

fix some tests

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

fix bltcrossattention after modular break

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

some fixes / feedback

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

try cache generate fix

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

try cache generate fix

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

fix generate tests

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

attn_impl workaround

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

refactoring to use recent TransformersKwargs changes

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

fix hidden_states shape test

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

refactor to new outputs

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

simplify outputs a bit

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

rm unneeded decoderlayer overwriting

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

rename blt

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

forgot tokenizer test renamed

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

Reorder

itazap

authored and

ita.zaporozhets@huggingface.co

committed

Reorder

itazap

authored and

ita.zaporozhets@huggingface.co

committed

working on modular

itazap

authored and

ita.zaporozhets@huggingface.co

committed

updates from modular

itazap

authored and

ita.zaporozhets@huggingface.co

committed

new modular

itazap

authored and

ita.zaporozhets@huggingface.co

committed

ruff and such

itazap

authored and

ita.zaporozhets@huggingface.co

committed

update pretrainedmodel modular

itazap

authored and

ita.zaporozhets@huggingface.co

committed

using cohere2 apply_rotary_pos_emb

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

small changes

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

apply feedback r2

itazap

authored and

ita.zaporozhets@huggingface.co

committed

fix cross_attention

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

apply more feedback

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

update modeling fix

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

load submodules from pretrainedmodel

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

set initializer_range to subconfigs

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

rm cross_attnetion_states pass when not needed

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

add 7b projection layer support

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

check repo

itazap

authored and

ita.zaporozhets@huggingface.co

committed

make copies

itazap

authored and

ita.zaporozhets@huggingface.co

committed

lost cohere2 rotate_half

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

ruff

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

copies?

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

don't tie weights for submodules

ita.zaporozhets@huggingface.co

committed

tie weights setting

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

check docstrings

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

apply feedback

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

rebase

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

rebased modeling

itazap

authored and

ita.zaporozhets@huggingface.co

committed

update docs

itazap

authored and

ita.zaporozhets@huggingface.co

committed

applying feedback

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

few more fixes

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

fix can_record_outputs

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

fast tokenizer

ita.zaporozhets@huggingface.co

committed

no more modulelist

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

tok auto

ita.zaporozhets@huggingface.co

authored and

ita.zaporozhets@huggingface.co

committed

rm tokenizersss

ita.zaporozhets@huggingface.co

committed

fix docs

ita.zaporozhets@huggingface.co

committed

ruff

ita.zaporozhets@huggingface.co

committed

fix after rebase

ita.zaporozhets@huggingface.co

committed

fix test, configs are not subscriptable

ita.zaporozhets@huggingface.co

committed

Merge branch 'main' into blt_wip

itazap

authored

Commits

Commits on Sep 18, 2025