Add new models (Janus, Qwen2-VL, JinaCLIP, LLaVA-OneVision, ViTPose, MGP-STR) & refactor processors.

[xenova]

New models

Janus (any-to-any)

This PR adds support for deepseek-ai/Janus-1.3B, a novel autoregressive framework that unifies multimodal understanding and generation.

In particular, it can do the following:

• text+image to text:

  // Example code based on https://github.com/deepseek-ai/Janus/blob/main/inference.py
  import { AutoProcessor, MultiModalityCausalLM } from '@huggingface/transformers';
  
  // Load processor and model
  const model_id = 'onnx-community/Janus-1.3B-ONNX';
  const processor = await AutoProcessor.from_pretrained(model_id);
  const model = await MultiModalityCausalLM.from_pretrained(model_id, {
      dtype: {
          prepare_inputs_embeds: 'q4',
          language_model: 'q4f16',
          lm_head: 'fp16',
          gen_head: 'fp16',
          gen_img_embeds: 'fp16',
          image_decode: 'fp32',
      },
  });
  
  // Prepare inputs
  const conversation = [
      {
          role: "User",
          content: "<image_placeholder>\nConvert the formula into latex code.",
          images: ["https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/quadratic_formula.png"],
      },
  ]
  const inputs = await processor(conversation);
  
  // Generate response
  const outputs = await model.generate({
      ...inputs,
      max_new_tokens: 150,
      do_sample: false,
  });
  
  // Decode output
  const new_tokens = outputs.slice(null, [inputs.input_ids.dims.at(-1), null])
  const decoded = processor.tokenizer.batch_decode(new_tokens, { skip_special_tokens: true });
  console.log(decoded);

  Example output:

  Sure, here is the LaTeX code for the given formula:
  
  ```
  x = \frac{-b \pm \sqrt{b^2 - 4a c}}{2a}
  ```
  
  This code represents the mathematical expression for the variable \( x \).
  

• image-to-text:

  // Example code based on https://github.com/deepseek-ai/Janus/blob/main/generation_inference.py
  import { AutoProcessor, MultiModalityCausalLM } from '@huggingface/transformers';
  
  // Load processor and model
  const model_id = 'onnx-community/Janus-1.3B-ONNX';
  const processor = await AutoProcessor.from_pretrained(model_id);
  const model = await MultiModalityCausalLM.from_pretrained(model_id, {
      dtype: {
          prepare_inputs_embeds: 'fp32',
          language_model: 'q4',
          lm_head: 'fp32',
          gen_head: 'fp32',
          gen_img_embeds: 'fp32',
          image_decode: 'fp32',
      },
  });
  
  // Prepare inputs
  const conversation = [
      {
          role: "User",
          content: "A cute and adorable baby fox with big brown eyes, autumn leaves in the background enchanting,immortal,fluffy, shiny mane,Petals,fairyism,unreal engine 5 and Octane Render,highly detailed, photorealistic, cinematic, natural colors.",
      },
  ]
  const inputs = await processor(conversation, {
      chat_template: "text_to_image",
  });
  
  // Generate response
  const num_image_tokens = processor.num_image_tokens;
  const outputs = await model.generate_images({
      ...inputs,
      min_new_tokens: num_image_tokens,
      max_new_tokens: num_image_tokens,
      do_sample: true,
  });
  
  // Save the generated image
  await outputs[0].save('test.png');

  Example outputs:

---

This PR also refactors the way that processor classes load image/text pre-preprocessors, aligning better with the python transformers library.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[kungfooman]

This is an amazing PR and I thought I give it a test in Chrome on Linux. I see randomly these two errors:

Or simply:

Is there anything special I need to do to test this in the browser? So far I couldn't get it to work.

By default it seems to pick WebGPU, but WebGPU is known to work very poorly on Linux, so I tried every other possibility:

const model = await MultiModalityCausalLM.from_pretrained(model_id, {
    dtype: {
        prepare_inputs_embeds: 'fp32',
        language_model: 'q4',
        lm_head: 'fp32',
        gen_head: 'fp32',
        gen_img_embeds: 'fp32',
        image_decode: 'fp32',
    },
    // Pick one: webnn-npu, webnn-gpu, webnn-cpu, webnn, webgpu, wasm
    device: 'wasm',
});

None worked 🙈

Thank you anyway, looking forward getting this to work somehow!

[pdufour]

This is great, will this PR support Qwen2-VL? 🙏

[xenova]

This is great, will this PR support Qwen2-VL? 🙏

Hey @pdufour, I was originally planning on doing this in a separate PR, but I've been following your work on getting it running (great work BTW!) and so it might be possible to squeeze into this PR! 👀

[xenova]

Merging to put out Transformers.js v3.1. Follow-up patches may be needed, but it's good to go for now imo!