Unsupported model type: jais

[SherifElfadaly]

Question

System Info

macOS, node v20.10, @xenova/transformers 2.17.2

Environment/Platform

• Website/web-app
• Browser extension
• Server-side (e.g., Node.js, Deno, Bun)
• Desktop app (e.g., Electron)
• Other (e.g., VSCode extension)

Description

Error: Unsupported model type: jais
    at Function.from_pretrained (file:///node_modules/@xenova/transformers/src/models.js:5526:19)
    at async Promise.all (index 1)
    at loadItems (file:///node_modules/@xenova/transformers/src/pipelines.js:3279:5)
    at pipeline (file:///node_modules/@xenova/transformers/src/pipelines.js:3219:21)
    at SearchQueryParser.initializeModel (src/search-engine/query-parser/search-query-parser.ts:27:18)

Reproduction

import { Logger } from '@nestjs/common';

export class SearchQueryParser {
  private tokenizer: any;
  private model: any;
  private logger: Logger;
  private systemPrompt = '';

  constructor() {
    this.logger = new Logger('query parser');
    this.initializeModel();
  }

  private async initializeModel() {
    const { AutoTokenizer, pipeline } = await import('@xenova/transformers');
    this.tokenizer = await AutoTokenizer.from_pretrained(
      'omarabb315/Query-5KM-no_synonyms_noon_1',
      {
        progress_callback: (data) => {
          this.logger.verbose(
            ${data.status} ${data.file || ''} ${data.progress || ''}`,
          );
        },
      },
    );
    this.model = await pipeline(
      'text-generation',
      'omarabb315/Query-5KM-no_synonyms_noon_1',
    );
  }

  async parse(query: string): Promise<any> {
    if (!this.model) {
      await this.initializeModel();
    }

    const tokenizerResponse = this.tokenizer.apply_chat_template(
      [
        { role: 'system', content: this.systemPrompt },
        { role: 'user', content: query },
      ],
      {
        tokenize: false,
        add_generation_prompt: true,
      },
    );

    const response = this.model(tokenizerResponse.toString());

    const parsedQuery = response[0].generated_text;

    return parsedQuery;
  }
}

[xenova]

The model you've linked to appears to be gated, so I added support for and tested w/ https://huggingface.co/onnx-community/tiny-random-jais. Once you convert your one to ONNX, it should work with Transformers.js v3 👍

[omarabb315]

@xenova Thanks for your support, I've granted you the access to the model, can you convert it to onnx or tell me how, because I tried the below code snippet but returned this error, so I wonder how did you prepared the "tiny-random-jais" model:

Error:

ValueError: Trying to export a jais model, that is a custom or unsupported architecture, but no custom onnx configuration was passed as `custom_onnx_configs`. Please refer to https://huggingface.co/docs/optimum/main/en/exporters/onnx/usage_guides/export_a_model#custom-export-of-transformers-models for an example on how to export custom models. Please open an issue at https://github.com/huggingface/optimum/issues if you would like the model type jais to be supported natively in the ONNX export.

The code snippet:

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer

model_checkpoint = "omarabb315/Query-5KM-no_synonyms_noon_1"
save_directory = "omarabb315/Query-5KM-no_synonyms_noon_onnx"

# Load a model from transformers and export it to ONNX
ort_model = ORTModelForSequenceClassification.from_pretrained(model_checkpoint, export=True, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

# Save the onnx model and tokenizer
ort_model.push_to_hub(save_directory)
tokenizer.push_to_hub(save_directory)

[xenova]

You can do it with our conversion script:

git clone -b v3 https://github.com/xenova/transformers.js.git
cd transformers.js
pip install -q -r scripts/requirements.txt
python -m scripts.convert \
  --quantize \
  --model_id katuni4ka/tiny-random-jais \
  --trust_remote_code \
  --skip_validation \
  --custom_onnx_configs '{"model":"gpt2"}' \
  --task text-generation-with-past

---

I can make PRs for your models, if you'd like 👍

[omarabb315]

That is clear, thanks a lot, yes please PR would be great

[xenova]

PR made 👍 I've also added fp16 and int8 quantized variants, which may help reduce computational load. I haven't been able to test them though... hopefully it works! 🤞

I have tested https://huggingface.co/inceptionai/jais-family-590m-chat, and that seems to work fine 👍

import { pipeline } from '@huggingface/transformers';

const generator = await pipeline('text-generation', 'onnx-community/jais-family-590m-chat', {
    dtype: 'q8',
});

const question = "What is the capital of UAE?";
const prompt_eng = `### Instruction: Your name is 'Jais', and you are named after Jebel Jais, the highest mountain in UAE. You were made by 'Inception' in the UAE. You are a helpful, respectful, and honest assistant. Always answer as helpfully as possible, while being safe. Complete the conversation between [|Human|] and [|AI|]:\n### Input: [|Human|] ${question}\n[|AI|]\n### Response:`;

const result = await generator(prompt_eng, { max_new_tokens: 128, return_full_text: false });
console.log(result);

produces:

"The capital city of the United Arab Emirates, a federation of seven emirates on the eastern side of the Arabian peninsula, is Abu Dhabi. This bustling metropolis is not only the political, economic, and cultural hub of the UAE but also a global hub for tourism and trade. It is home to the world's largest oil refinery, the Abu Dhabi National Petroleum Company (ADNOC), and the world's largest seafaring company, the Abu Dhabi Ports Corporation (ADPOC). The city is also a major hub for the United Arab Emirates'"

[SherifElfadaly]

@xenova The original error is now gone put a new error was raised

2024-08-28 18:03:45.956 node[14526:6341613] 2024-08-28 18:03:45.956927 [E:onnxruntime:, inference_session.cc:2105 operator()] Exception during initialization: filesystem error: in file_size: No such file or directory ["model.onnx_data"]

/node_modules/onnxruntime-node/lib/backend.ts:16
      this.#inferenceSession.loadModel(pathOrBuffer.buffer, pathOrBuffer.byteOffset, pathOrBuffer.byteLength, options);
                             ^
Error: Exception during initialization: filesystem error: in file_size: No such file or directory ["model.onnx_data"]
    at new OnnxruntimeSessionHandler (/node_modules/onnxruntime-node/lib/backend.ts:16:30)
    at Immediate.<anonymous> (/node_modules/onnxruntime-node/lib/backend.ts:61:19)
    at processImmediate (node:internal/timers:478:21)

Any idea what is missing?

[xenova]

Yes we are still adding the external data functionality (to load models > 2GB). For now, you can set dtype: 'q8' or dtype: 'fp16' to use a smaller model.

[SherifElfadaly]

@xenova Thanks for your support, it seems to be working fine now.

[omarabb315]

@xenova Thanks again for your effort, lastly, can you please share with me the snippet used to produce this PR, in order to use it in our future models

[xenova]

can you please share with me the snippet used to produce this PR, in order to use it in our future models

See here for the commit to add JAIS models to Transformers.js, here for the conversion code and here for the code snippet. Is there anything else you'd like?