model: Add GTE to Transformers

docs/source/en/_toctree.yml

@@ -701,6 +701,8 @@
         title: GraniteMoeSWA
       - local: model_doc/granite_swa
         title: GraniteSWA
+      - local: model_doc/gte
+        title: GTE
       - local: model_doc/helium
         title: Helium
       - local: model_doc/herbert

docs/source/en/model_doc/gte.md

@@ -0,0 +1,117 @@
+<!--Copyright 2026 the HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+
+⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be rendered properly in your Markdown viewer.
+
+-->
+*This model was published in HF papers on 2024-07-29 and contributed to Hugging Face Transformers on 2026-09-01.*
+
+# GTE
+
+## Overview
+
+GTE was proposed in [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://huggingface.co/papers/2407.19669) by Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li and Min Zhang.
+
+The abstract from the paper is the following:
+
+*We present systematic efforts in building long-context multilingual text representation model (TRM) and reranker from scratch for text retrieval. We first introduce a text encoder (base size) enhanced with RoPE and unpadding, pre-trained in a native 8192-token context (longer than 512 of previous multilingual encoders). Then we construct a hybrid TRM and a cross-encoder reranker by contrastive learning. Evaluations show that our text encoder outperforms the same-sized previous state-of-the-art XLM-R. Meanwhile, our TRM and reranker match the performance of large-sized state-of-the-art BGE-M3 models and achieve better results on long-context retrieval benchmarks. Further analysis demonstrate that our proposed models exhibit higher efficiency during both training and inference. We believe their efficiency and effectiveness could benefit various researches and industrial applications.*
+
+GTE is a BERT-style bidirectional encoder that replaces absolute position embeddings with RoPE, uses a gated MLP, and applies layer normalization after each residual connection. The same architecture backs Alibaba's `gte-*-v1.5` and `gte-multilingual-*` checkpoints as well as Snowflake's `snowflake-arctic-embed-m-v2.0`.
+
+This model was contributed by [Harshal Janjani](https://huggingface.co/harshaljanjani).
+The original code can be found [here](https://huggingface.co/Alibaba-NLP/new-impl).
+
+> [!TIP]
+> Click on the GTE models in the right sidebar for more examples of how to apply GTE to different language tasks.
+
+The example below demonstrates how to extract features (embeddings) with [`Pipeline`] and [`AutoModel`].
+
+<hfoptions id="usage">
+<hfoption id="Pipeline">
+
+```python
+from transformers import pipeline
+
+
+# TODO: Remove revision
+pipeline = pipeline(
+    task="feature-extraction",
+    model="Alibaba-NLP/gte-multilingual-base",
+    revision="refs/pr/31",
+    device=0
+)
+pipeline("Plants create oxygen through a process known as photosynthesis.")
+```
+
+</hfoption>
+<hfoption id="AutoModel">
+
+```python
+import torch
+
+from transformers import AutoModel, AutoTokenizer
+
+
+# TODO: Remove revision
+tokenizer = AutoTokenizer.from_pretrained("Alibaba-NLP/gte-multilingual-base", revision="refs/pr/31")
+model = AutoModel.from_pretrained(
+    "Alibaba-NLP/gte-multilingual-base",
+    revision="refs/pr/31",
+    device_map="auto",
+    attn_implementation="sdpa"
+)
+inputs = tokenizer("Plants create oxygen through a process known as photosynthesis.", return_tensors="pt").to(model.device)
+
+with torch.no_grad():
+    outputs = model(**inputs)
+    embeddings = outputs.last_hidden_state[:, 0]
+
+print(f"Embeddings shape: {embeddings.shape}")
+```
+
+</hfoption>
+</hfoptions>
+
+## Notes
+
+- GTE uses RoPE, so for correct positional encoding either use right padding (the default), or use left padding and prepare `position_ids` accordingly.
+- `type_vocab_size` differs across checkpoints. `Alibaba-NLP/gte-base-en-v1.5` sets it to `0`, in which case no token type embedding is created and `token_type_ids` are ignored.
+- The `gte-*-v1.5` and `gte-multilingual-*` checkpoints apply static NTK scaling on top of RoPE. It is expressed as a `linear` [`~modeling_rope_utils.RopeParameters`] entry whose `rope_theta` is the base scaled by the NTK factor.
+
+## GteConfig
+
+[[autodoc]] GteConfig
+
+## GteModel
+
+[[autodoc]] GteModel
+    - forward
+
+## GteForMaskedLM
+
+[[autodoc]] GteForMaskedLM
+    - forward
+
+## GteForSequenceClassification
+
+[[autodoc]] GteForSequenceClassification
+    - forward
+
+
+## GteForTokenClassification
+
+[[autodoc]] GteForTokenClassification
+    - forward
+

src/transformers/conversion_mapping.py

@@ -1321,6 +1321,31 @@ def _build_checkpoint_conversion_mapping():
                 operations=[Chunk(dim=0)],
             ),
         ],
+        "gte": [
+            PrefixChange(prefix_to_remove="new"),
+            WeightRenaming(r"encoder.layer", r"layers"),
+            WeightRenaming(r"attention.o_proj", r"self_attn.o_proj"),
+            WeightRenaming(r"attn_ln", r"post_attention_layernorm"),
+            WeightRenaming(r"mlp_ln", r"post_mlp_layernorm"),
+            WeightRenaming(r"lm_head.norm", r"lm_head.layer_norm"),
+            WeightConverter(
+                source_patterns="attention.qkv_proj",
+                target_patterns=[
+                    "self_attn.q_proj",
+                    "self_attn.k_proj",
+                    "self_attn.v_proj",
+                ],
+                operations=[Chunk(dim=0)],
+            ),
+            WeightConverter(
+                source_patterns="mlp.up_gate_proj",
+                target_patterns=[
+                    "mlp.up_proj",
+                    "mlp.gate_proj",
+                ],
+                operations=[Chunk(dim=0)],
+            ),
+        ],
         "jina_embeddings_v3": [
             WeightRenaming(source_patterns="emb_ln", target_patterns="embeddings.LayerNorm"),
             WeightRenaming(source_patterns="encoder.layers", target_patterns="layers"),

src/transformers/models/__init__.py

@@ -210,6 +210,7 @@
     from .granitemoeshared import *
     from .grounding_dino import *
     from .groupvit import *
+    from .gte import *
     from .helium import *
     from .herbert import *
     from .hgnet_v2 import *

src/transformers/models/auto/auto_mappings.py

@@ -284,6 +284,7 @@
         ("groupvit", "GroupViTConfig"),
         ("groupvit_text_model", "GroupViTTextConfig"),
         ("groupvit_vision_model", "GroupViTVisionConfig"),
+        ("gte", "GteConfig"),
         ("helium", "HeliumConfig"),
         ("hgnet_v2", "HGNetV2Config"),
         ("hiera", "HieraConfig"),

src/transformers/models/auto/modeling_auto.py

@@ -245,6 +245,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
         ("granitemoeshared", "GraniteMoeSharedModel"),
         ("grounding-dino", "GroundingDinoModel"),
         ("groupvit", "GroupViTModel"),
+        ("gte", "GteModel"),
         ("helium", "HeliumModel"),
         ("hgnet_v2", "HGNetV2Backbone"),
         ("hiera", "HieraModel"),
@@ -1210,6 +1211,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
         ("flaubert", "FlaubertWithLMHeadModel"),
         ("fnet", "FNetForMaskedLM"),
         ("funnel", "FunnelForMaskedLM"),
+        ("gte", "GteForMaskedLM"),
         ("ibert", "IBertForMaskedLM"),
         ("jina_embeddings_v3", "JinaEmbeddingsV3ForMaskedLM"),
         ("layoutlm", "LayoutLMForMaskedLM"),
@@ -1426,6 +1428,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
         ("gpt_neox", "GPTNeoXForSequenceClassification"),
         ("gpt_oss", "GptOssForSequenceClassification"),
         ("gptj", "GPTJForSequenceClassification"),
+        ("gte", "GteForSequenceClassification"),
         ("helium", "HeliumForSequenceClassification"),
         ("hunyuan_v1_dense", "HunYuanDenseV1ForSequenceClassification"),
         ("hunyuan_v1_moe", "HunYuanMoEV1ForSequenceClassification"),
@@ -1655,6 +1658,7 @@ class _BaseModelWithGenerate(PreTrainedModel, GenerationMixin):
         ("gpt_neo", "GPTNeoForTokenClassification"),
         ("gpt_neox", "GPTNeoXForTokenClassification"),
         ("gpt_oss", "GptOssForTokenClassification"),
+        ("gte", "GteForTokenClassification"),
         ("helium", "HeliumForTokenClassification"),
         ("ibert", "IBertForTokenClassification"),
         ("jina_embeddings_v3", "JinaEmbeddingsV3ForTokenClassification"),

src/transformers/models/auto/tokenization_auto.py

@@ -160,6 +160,7 @@
         ("granitemoeshared", "TokenizersBackend" if is_tokenizers_available() else None),
         ("grounding-dino", "BertTokenizer" if is_tokenizers_available() else None),
         ("groupvit", "CLIPTokenizer" if is_tokenizers_available() else None),
+        ("gte", "XLMRobertaTokenizer" if is_tokenizers_available() else None),
         ("herbert", "HerbertTokenizer" if is_tokenizers_available() else None),
         ("hubert", "Wav2Vec2CTCTokenizer"),
         ("hunyuan_vl", "Qwen2Tokenizer" if is_tokenizers_available() else None),

src/transformers/models/gte/__init__.py

@@ -0,0 +1,28 @@
+# Copyright 2026 the HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import TYPE_CHECKING
+
+from ...utils import _LazyModule
+from ...utils.import_utils import define_import_structure
+
+
+if TYPE_CHECKING:
+    from .configuration_gte import *
+    from .modeling_gte import *
+else:
+    import sys
+
+    _file = globals()["__file__"]
+    sys.modules[__name__] = _LazyModule(__name__, _file, define_import_structure(_file), module_spec=__spec__)

src/transformers/models/gte/configuration_gte.py

@@ -0,0 +1,86 @@
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+#           This file was automatically generated from src/transformers/models/gte/modular_gte.py.
+#               Do NOT edit this file manually as any edits will be overwritten by the generation of
+#             the file from the modular. If any change should be done, please apply the change to the
+#                          modular_gte.py file directly. One of our CI enforces this.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+# Copyright 2026 the HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from huggingface_hub.dataclasses import strict
+
+from ...configuration_utils import PreTrainedConfig
+from ...modeling_rope_utils import RopeParameters
+from ...utils import auto_docstring
+
+
+@auto_docstring(checkpoint="Alibaba-NLP/gte-multilingual-base")
+@strict
+class GteConfig(PreTrainedConfig):
+    r"""
+    Examples:
+
+    ```python
+    >>> from transformers import GteConfig, GteModel
+
+    >>> # Initializing a GTE Alibaba-NLP/gte-multilingual-base style configuration
+    >>> configuration = GteConfig()
+
+    >>> # Initializing a model (with random weights) from the Alibaba-NLP/gte-multilingual-base style configuration
+    >>> model = GteModel(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "gte"
+
+    vocab_size: int = 250048
+    hidden_size: int = 768
+    num_hidden_layers: int = 12
+    num_attention_heads: int = 12
+    intermediate_size: int = 3072
+    hidden_act: str = "gelu"
+    hidden_dropout_prob: float | int = 0.1
+    attention_probs_dropout_prob: float | int = 0.0
+    max_position_embeddings: int = 8192
+    type_vocab_size: int = 1
+    initializer_range: float = 0.02
+    layer_norm_eps: float = 1e-12
+    pad_token_id: int | None = 1
+    classifier_dropout: float | int | None = None
+    bos_token_id: int | None = None
+    eos_token_id: int | list[int] | None = None
+    tie_word_embeddings: bool = True
+    rope_parameters: RopeParameters | dict | None = None
+
+    def convert_rope_params_to_dict(self, **kwargs):
+        rope_scaling = kwargs.pop("rope_scaling", None)
+        self.rope_parameters = self.rope_parameters if self.rope_parameters is not None else {}
+        rope_theta = kwargs.pop("rope_theta", self.default_theta)
+
+        # Static NTK scaling in Alibaba-NLP/gte-multilingual-base is exactly a linear scaling of `base * factor`.
+        if rope_scaling is not None and rope_scaling["type"] == "ntk":
+            head_dim = self.hidden_size // self.num_attention_heads
+            factor = rope_scaling["factor"]
+            self.rope_parameters.setdefault("rope_type", "linear")
+            self.rope_parameters.setdefault("factor", factor ** (2 / head_dim))
+            rope_theta = rope_theta * factor
+
+        self.rope_parameters.setdefault("rope_theta", rope_theta)
+        self.standardize_rope_params()
+        return kwargs
+
+
+__all__ = ["GteConfig"]