Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/source/en/_toctree.yml
Original file line number Diff line number Diff line change
Expand Up @@ -701,6 +701,8 @@
title: GraniteMoeSWA
- local: model_doc/granite_swa
title: GraniteSWA
- local: model_doc/gte
title: GTE
- local: model_doc/helium
title: Helium
- local: model_doc/herbert
Expand Down
3 changes: 3 additions & 0 deletions docs/source/en/model_doc/deimv2.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ The abstract from the paper is the following:

*Driven by the simple and effective Dense O2O, DEIM demonstrates faster convergence and enhanced performance. In this work, we extend it with DINOv3 features, resulting in DEIMv2. DEIMv2 spans eight model sizes from X to Atto, covering GPU, edge, and mobile deployment. For the X, L, M, and S variants, we adopt DINOv3-pretrained / distilled backbones and introduce a Spatial Tuning Adapter (STA), which efficiently converts DINOv3's single-scale output into multi-scale features and complements strong semantics with fine-grained details to enhance detection. For ultra-lightweight models (Nano, Pico, Femto, and Atto), we employ HGNetv2 with depth and width pruning to meet strict resource budgets. Together with a simplified decoder and an upgraded Dense O2O, this unified design enables DEIMv2 to achieve a superior performance-cost trade-off across diverse scenarios, establishing new state-of-the-art results. Notably, our largest model, DEIMv2-X, achieves 57.8 AP with only 50.3M parameters, surpassing prior X-scale models that require over 60M parameters for just 56.5 AP. On the compact side, DEIMv2-S is the first sub-10M model (9.71M) to exceed the 50 AP milestone on COCO, reaching 50.9 AP. Even the ultra-lightweight DEIMv2-Pico, with just 1.5M parameters, delivers 38.5 AP-matching YOLOv10-Nano (2.3M) with ~50% fewer parameters.*

This model was contributed by [Harshal Janjani](https://huggingface.co/harshaljanjani).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's move it to a different PR but definitely merging then to credit!

@harshaljanjani harshaljanjani Sep 1, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Raised #48448 🤗

The original code can be found [here](https://github.com/Intellindust-AI-Lab/DEIMv2).

## Usage

```python
Expand Down
163 changes: 163 additions & 0 deletions docs/source/en/model_doc/gte.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
<!--Copyright 2026 the HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.


⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be rendered properly in your Markdown viewer.

-->
*This model was published in HF papers on 2024-07-29 and contributed to Hugging Face Transformers on 2026-08-30.*

# GTE

## Overview

GTE was proposed in [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://huggingface.co/papers/2407.19669) by Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li and Min Zhang.

The abstract from the paper is the following:

*We present systematic efforts in building long-context multilingual text representation model (TRM) and reranker from scratch for text retrieval. We first introduce a text encoder (base size) enhanced with RoPE and unpadding, pre-trained in a native 8192-token context (longer than 512 of previous multilingual encoders). Then we construct a hybrid TRM and a cross-encoder reranker by contrastive learning. Evaluations show that our text encoder outperforms the same-sized previous state-of-the-art XLM-R. Meanwhile, our TRM and reranker match the performance of large-sized state-of-the-art BGE-M3 models and achieve better results on long-context retrieval benchmarks. Further analysis demonstrate that our proposed models exhibit higher efficiency during both training and inference. We believe their efficiency and effectiveness could benefit various researches and industrial applications.*

GTE is a BERT-style bidirectional encoder that replaces absolute position embeddings with RoPE, uses a gated MLP, and applies layer normalization after each residual connection. The same architecture backs Alibaba's `gte-*-v1.5` and `gte-multilingual-*` checkpoints as well as Snowflake's `snowflake-arctic-embed-m-v2.0`.

This model was contributed by [Harshal Janjani](https://huggingface.co/harshaljanjani).
The original code can be found [here](https://huggingface.co/Alibaba-NLP/new-impl).

## Usage examples

Embeddings are taken from the `[CLS]` token and normalized.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this not for sequence classification then? Or am I missing something?

Might be nicer to just follow the other bert likes a bit with pipeline usage etc

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So [CLS] is being used for pooling into an embedding, not classification here, as the model was trained. Classification is a separate model path with an added head. Rewrote it like bert and jina_embeddings_v3


```python
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model_id = "harshaljanjani/gte-multilingual-base-hf"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, device_map="auto")

inputs = tokenizer("what is the capital of China?", return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model(**inputs)

embeddings = F.normalize(outputs.last_hidden_state[:, 0], p=2, dim=-1)
```

Batched inference scores a query against several documents:

```python
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model_id = "harshaljanjani/gte-multilingual-base-hf"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, device_map="auto")

texts = ["what is the capital of China?", "Beijing", "sorting algorithms"]
inputs = tokenizer(texts, padding=True, truncation=True, max_length=8192, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model(**inputs)

embeddings = F.normalize(outputs.last_hidden_state[:, 0], p=2, dim=-1)
scores = embeddings[:1] @ embeddings[1:].T
```

The reranker checkpoints are cross-encoders and expose a single relevance logit per pair:

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "harshaljanjani/gte-multilingual-reranker-base-hf"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id, device_map="auto")

pairs = [["what is the capital of China?", "Beijing"], ["what is the capital of China?", "sorting algorithms"]]
inputs = tokenizer(pairs, padding=True, truncation=True, max_length=8192, return_tensors="pt").to(model.device)
with torch.no_grad():
scores = model(**inputs).logits.view(-1)
```

Fine-tuning uses the standard forward and backward pass:

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "harshaljanjani/gte-multilingual-base-hf"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id, num_labels=2, device_map="auto")

inputs = tokenizer(["a positive review", "a negative review"], padding=True, return_tensors="pt").to(model.device)
labels = torch.tensor([1, 0], device=model.device)

loss = model(**inputs, labels=labels).loss
loss.backward()
```

The model is compatible with [`torch.compile`]:

```python
import torch
from transformers import AutoModel, AutoTokenizer

model_id = "harshaljanjani/gte-multilingual-base-hf"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, device_map="auto")
model = torch.compile(model)

inputs = tokenizer("what is the capital of China?", return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model(**inputs)
```

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think those make more sense for e.g. audio models as they are more special but here we can focus on more casual usage with auto and pipeline

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done :)


## Notes

- GTE uses RoPE, so for correct positional encoding either use right padding (the default), or use left padding and prepare `position_ids` accordingly.
- `type_vocab_size` differs across checkpoints. `Alibaba-NLP/gte-base-en-v1.5` sets it to `0`, in which case no token type embedding is created and `token_type_ids` are ignored.
- The `gte-*-v1.5` and `gte-multilingual-*` checkpoints apply static NTK scaling on top of RoPE. It is expressed as a `linear` [`~modeling_rope_utils.RopeParameters`] entry whose `rope_theta` is the base scaled by the NTK factor.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will see but ig we register in post init properly?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here, with the Hub PRs the configs now carry rope_parameters directly as you suggested :)


## GteConfig

[[autodoc]] GteConfig

## GteModel

[[autodoc]] GteModel
- forward

## GteForMaskedLM

[[autodoc]] GteForMaskedLM
- forward

## GteForSequenceClassification

[[autodoc]] GteForSequenceClassification
- forward

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super nit lol

## GteForMultipleChoice

[[autodoc]] GteForMultipleChoice
- forward

## GteForTokenClassification

[[autodoc]] GteForTokenClassification
- forward

## GteForQuestionAnswering

[[autodoc]] GteForQuestionAnswering
- forward
25 changes: 25 additions & 0 deletions src/transformers/conversion_mapping.py
Original file line number Diff line number Diff line change
Expand Up @@ -1321,6 +1321,31 @@ def _build_checkpoint_conversion_mapping():
operations=[Chunk(dim=0)],
),
],
"gte": [
PrefixChange(prefix_to_remove="new"),
WeightRenaming(r"encoder.layer", r"layers"),
WeightRenaming(r"attention.o_proj", r"self_attn.o_proj"),
WeightRenaming(r"attn_ln", r"post_attention_layernorm"),
WeightRenaming(r"mlp_ln", r"post_mlp_layernorm"),
WeightRenaming(r"lm_head.norm", r"lm_head.layer_norm"),
WeightConverter(
source_patterns="attention.qkv_proj",
target_patterns=[
"self_attn.q_proj",
"self_attn.k_proj",
"self_attn.v_proj",
],
operations=[Chunk(dim=0)],
),
WeightConverter(
source_patterns="mlp.up_gate_proj",
target_patterns=[
"mlp.up_proj",
"mlp.gate_proj",
],
operations=[Chunk(dim=0)],
),
],
"jina_embeddings_v3": [
WeightRenaming(source_patterns="emb_ln", target_patterns="embeddings.LayerNorm"),
WeightRenaming(source_patterns="encoder.layers", target_patterns="layers"),
Expand Down
1 change: 1 addition & 0 deletions src/transformers/models/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,7 @@
from .granitemoeshared import *
from .grounding_dino import *
from .groupvit import *
from .gte import *
from .helium import *
from .herbert import *
from .hgnet_v2 import *
Expand Down
1 change: 1 addition & 0 deletions src/transformers/models/auto/auto_mappings.py
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,7 @@
("groupvit", "GroupViTConfig"),
("groupvit_text_model", "GroupViTTextConfig"),
("groupvit_vision_model", "GroupViTVisionConfig"),
("gte", "GteConfig"),
("helium", "HeliumConfig"),
("hgnet_v2", "HGNetV2Config"),
("hiera", "HieraConfig"),
Expand Down
6 changes: 6 additions & 0 deletions src/transformers/models/auto/modeling_auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
("granitemoeshared", "GraniteMoeSharedModel"),
("grounding-dino", "GroundingDinoModel"),
("groupvit", "GroupViTModel"),
("gte", "GteModel"),
("helium", "HeliumModel"),
("hgnet_v2", "HGNetV2Backbone"),
("hiera", "HieraModel"),
Expand Down Expand Up @@ -1209,6 +1210,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
("flaubert", "FlaubertWithLMHeadModel"),
("fnet", "FNetForMaskedLM"),
("funnel", "FunnelForMaskedLM"),
("gte", "GteForMaskedLM"),
("ibert", "IBertForMaskedLM"),
("jina_embeddings_v3", "JinaEmbeddingsV3ForMaskedLM"),
("layoutlm", "LayoutLMForMaskedLM"),
Expand Down Expand Up @@ -1424,6 +1426,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
("gpt_neox", "GPTNeoXForSequenceClassification"),
("gpt_oss", "GptOssForSequenceClassification"),
("gptj", "GPTJForSequenceClassification"),
("gte", "GteForSequenceClassification"),
("helium", "HeliumForSequenceClassification"),
("hunyuan_v1_dense", "HunYuanDenseV1ForSequenceClassification"),
("hunyuan_v1_moe", "HunYuanMoEV1ForSequenceClassification"),
Expand Down Expand Up @@ -1534,6 +1537,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
("gpt_neo", "GPTNeoForQuestionAnswering"),
("gpt_neox", "GPTNeoXForQuestionAnswering"),
("gptj", "GPTJForQuestionAnswering"),
("gte", "GteForQuestionAnswering"),
("ibert", "IBertForQuestionAnswering"),
("jina_embeddings_v3", "JinaEmbeddingsV3ForQuestionAnswering"),
("layoutlmv2", "LayoutLMv2ForQuestionAnswering"),
Expand Down Expand Up @@ -1653,6 +1657,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
("gpt_neo", "GPTNeoForTokenClassification"),
("gpt_neox", "GPTNeoXForTokenClassification"),
("gpt_oss", "GptOssForTokenClassification"),
("gte", "GteForTokenClassification"),
("helium", "HeliumForTokenClassification"),
("ibert", "IBertForTokenClassification"),
("jina_embeddings_v3", "JinaEmbeddingsV3ForTokenClassification"),
Expand Down Expand Up @@ -1732,6 +1737,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
("flaubert", "FlaubertForMultipleChoice"),
("fnet", "FNetForMultipleChoice"),
("funnel", "FunnelForMultipleChoice"),
("gte", "GteForMultipleChoice"),
("ibert", "IBertForMultipleChoice"),
("longformer", "LongformerForMultipleChoice"),
("luke", "LukeForMultipleChoice"),
Expand Down
1 change: 1 addition & 0 deletions src/transformers/models/auto/tokenization_auto.py
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,7 @@
("granitemoeshared", "TokenizersBackend" if is_tokenizers_available() else None),
("grounding-dino", "BertTokenizer" if is_tokenizers_available() else None),
("groupvit", "CLIPTokenizer" if is_tokenizers_available() else None),
("gte", "XLMRobertaTokenizer" if is_tokenizers_available() else None),
("herbert", "HerbertTokenizer" if is_tokenizers_available() else None),
("hubert", "Wav2Vec2CTCTokenizer"),
("hunyuan_vl", "Qwen2Tokenizer" if is_tokenizers_available() else None),
Expand Down
28 changes: 28 additions & 0 deletions src/transformers/models/gte/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Copyright 2026 the HuggingFace Team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from typing import TYPE_CHECKING

from ...utils import _LazyModule
from ...utils.import_utils import define_import_structure


if TYPE_CHECKING:
from .configuration_gte import *
from .modeling_gte import *
else:
import sys

_file = globals()["__file__"]
sys.modules[__name__] = _LazyModule(__name__, _file, define_import_structure(_file), module_spec=__spec__)
Loading