Support deepseek/Janus-Pro-1B

examples/Janus-Pro-1B_Demo/README.md

@@ -0,0 +1,134 @@
+# Janus-Pro-1B Demo
+1. This demo demonstrates how to deploy the Janus-Pro-1B model. The Vision + Projector component is exported as an RKNN model using the `rknn-toolkit2`, while the LLM component is exported as an RKLLM model using the `rkllm-toolkit`.
+2. The open-source model used in this demo is available at: [Janus-Pro-1B](https://huggingface.co/deepseek-ai/Janus-Pro-1B)
+
+## 1. Requirements
+```
+rkllm-toolkit==1.1.4
+rknn-toolkit2==2.2.1
+python==3.8
+```
+
+rknn-toolkit2 installation guide：
+
+pip install rknn-toolkit2==2.2.1 -i https://mirrors.aliyun.com/pypi/simple
+
+install janus env: 
+
+```shell
+git clone https://github.com/deepseek-ai/Janus.git`
+
+cd Janus
+
+pip install -e .
+```
+
+install onnx:
+
+```shell
+pip install onnxruntime onnx
+```
+
+## 2. HuggingFace Demo
+
+```
+1、modify the modelpath in infer.py
+2、python infer.py
+3、expect results:
+[" 这张图片展示了一个宇航员在月球表面休息的场景。宇航员身穿白色太空服，头戴头盔，头盔上有反光镜片。他正躺在月球表面，双腿伸直，脚上穿着宇航服靴子，手里拿着一瓶绿色的啤酒，似乎在享受片刻的休息。
+
+背景中可以看到地球，它被月球表面的坑洞所环绕，显得格外壮观。月球表面布满了陨石坑和坑洞，看起来非常荒凉。宇航员旁边有一个绿色的箱子，上面有“Binty's”字样，可能是宇航员用来存放物品的。
+
+整个场景给人一种幽默和奇幻的感觉，仿佛宇航员在月球上悠闲地享受着一瓶啤酒，而不是在执行任务。这种构图和场景设计可能意在创造一种轻松愉快的氛围，同时又带有一些科幻的元素。"]
+```
+
+## 3. Model Conversion
+- ### convert to onnx
+
+1. Export the Vision + Projector component of the Janus-Pro-1B model to an ONNX model using the `export/export_vision.py` script.
+
+2. Since RKNN currently supports only `float32`, if the data type is restricted when loading weights, you need to set the `"use_flash_attn"` parameter in `config.json` to `false`.
+
+```bash
+python export/export_vision.py
+```
+
+- ### convert to rknn
+
+1. After successfully exporting the ONNX model, you can use the `export/export_vision_rknn.py` script along with the `rknn-toolkit2` tool to convert the ONNX model to an RKNN model.
+
+```bash
+python export/export_vision_rknn.py
+```
+
+- ### Export the LLM component
+
+Janus-Pro-1B model LLM component to an Huggingface model using the `export/export_llm.py` script.
+
+```shell
+python export/export_llm.py
+```
+
+- ### convert to rkllm component
+
+1. We collected 20 image-text examples from the MMBench_DEV_EN dataset, stored in `data/datasets.json` and `data/datasets`. To use these data, you first need to create `input_embeds` for quantizing the RKLLM model. Run the following code to generate `data/inputs.json`.
+
+```bash
+#Modify the Juans-Pro-1B ModelPath in data/make_input_embeds_for_quantize.py, and then
+python data/make_input_embeds_for_quantize.py
+```
+
+2. Use the following code to export the RKLLM model.
+
+```bash
+python export/export_rkllm.py
+```
+
+## 4. C++ Demo
+In the `deploy` directory, we provide example code for board-side inference. This code demonstrates the process of "image input to image features," where an input image is processed to output its corresponding image features. These features are then used by the RKLLM model for multimodal content inference.
+
+### 1. Compile and Build
+Users can directly compile the example code by running the `deploy/build-linux.sh` or `deploy/build-android.sh` script (replacing the cross-compiler path with the actual path). This will generate an `install/demo_Linux_aarch64` folder in the `deploy` directory, containing the executables `imgenc`, `llm`, `demo`, and the `lib` folder.
+
+```bash
+cd deploy
+# for linux
+./build-linux.sh
+# for android
+./build-android.sh
+# push install dir to device
+adb push ./install/demo_Linux_aarch64 /data
+# push model file to device
+adb push Janus_pro_vision_rk3588.rknn /data/models
+adb push Janus-Pro-1B-rk3588.rkllm /data/models
+# push demo image to device
+adb push ../data/demo.jpg /data/demo_Linux_aarch64
+```
+
+### 2. Run Demo
+Enter the `/data/demo_Linux_aarch64` directory on the board and run the example using the following code
+
+```bash
+adb shell
+cd /data/demo_Linux_aarch64
+# export lib path
+export LD_LIBRARY_PATH=./lib
+# soft link models dir
+ln -s /data/models .
+# run imgenc
+./imgenc models/Janus_pro_vision_rk3588.rknn demo.jpg
+# run llm(Pure Text Example)
+./llm models/Janus-Pro-1B-rk3588.rkllm 128 768
+# run demo(Multimodal Example)
+./demo demo.jpg models/Janus_pro_vision_rk3588.rknn models/Janus-Pro-1B-rk3588.rkllm 128 768
+```
+
+The user can view the relevant runtime logs in the terminal and obtain the `img_vec.bin` file in the current directory, which contains the image features corresponding to the input image.
+
+Multimodal Example
+
+```
+user: <image>What is in the image?
+robot: The image shows an astronaut sitting on the moon's surface while holding a green bottle labeled "Greenpeace." There are also two objects nearby—a cooler box marked as well-known and another object
+ that looks like part of some equipment or structure.
+```

examples/Janus-Pro-1B_Demo/README_CN.md

@@ -0,0 +1,146 @@
+Janus-Pro-1B
+
+本文档展示了如何部署 Janus-Pro-1B 模型。视觉 + 投影器组件使用 rknn-toolkit2 导出为 RKNN 模型，而大语言模型（LLM）组件使用 rkllm-toolkit 导出为 RKLLM 模型。
+
+本文档中使用的开源模型可在以下链接获取：[Janus-Pro-1B](https://huggingface.co/deepseek-ai/Janus-Pro-1B)
+
+## 1. 环境要求
+
+```shell
+rkllm-toolkit==1.1.4
+rknn-toolkit2==2.2.1
+python==3.8
+```
+
+rknn-toolkit2 安装指南：
+
+```shell
+pip install rknn-toolkit2==2.2.1 -i https://mirrors.aliyun.com/pypi/simple
+```
+
+Janus环境安装指南：
+
+```shell
+# 克隆仓库
+git clone https://github.com/deepseek-ai/Janus.git
+# 进入仓库目录
+cd Janus
+# 安装环境
+pip install -e .
+```
+
+onnx安装:
+
+```shell
+pip install onnxruntime onnx
+```
+
+## 2. 运行HuggingFace模型
+
+1、修改 infer.py 中的模型路径
+2、运行 python infer.py
+3、预期结果:
+[" 这张图片展示了一个宇航员在月球表面休息的场景。宇航员身穿白色太空服，头戴头盔，头盔上有反光镜片。他正躺在月球表面，双腿伸直，脚上穿着宇航服靴子，手里拿着一瓶绿色的啤酒，似乎在享受片刻的休息。
+
+背景中可以看到地球，它被月球表面的坑洞所环绕，显得格外壮观。月球表面布满了陨石坑和坑洞，看起来非常荒凉。宇航员旁边有一个绿色的箱子，上面有“Binty's”字样，可能是宇航员用来存放物品的。
+
+整个场景给人一种幽默和奇幻的感觉，仿佛宇航员在月球上悠闲地享受着一瓶啤酒，而不是在执行任务。这种构图和场景设计可能意在创造一种轻松愉快的氛围，同时又带有一些科幻的元素。"]
+
+## 3. 模型转换
+
+### 导出vision和Projectors组件，转成onnx格式
+
+使用 export/export_vision.py 脚本将 Janus-Pro-1B 模型的视觉 + 投影器组件导出为 ONNX 模型。
+
+由于 RKNN 当前仅支持 float32 数据类型，如果在加载权重时数据类型受限，需要将 config.json 中的 "use_flash_attn" 参数设置为 false。
+
+```shell
+# 修改 export/export_vision.py 中的模型路径
+python export/export_vision.py
+```
+
+### 转换为RKNN格式
+
+成功导出 ONNX 模型后，可以使用 export/export_vision_rknn.py 脚本结合 rknn-toolkit2 工具将 ONNX 模型转换为 RKNN 模型。
+
+```shell
+# 修改 export/export_vision_rknn.py 中的模型路径
+python export/export_vision_rknn.py
+```
+
+### 3.导出LLM组件，转成huggingface格式。
+
+把Janus-Pro-1B模型的LLM组件导出为huggingface格式。
+
+```shell
+# 修改 export/export_llm.py 中的模型路径
+python export/export_llm.py
+```
+
+### 4. LLM模型转成RKLLM格式
+
+我们从 MMBench_DEV_EN 数据集中收集了 20 个图像 - 文本示例，存储在 data/datasets.json 和 data/datasets 中。要使用这些数据，首先需要创建 input_embeds 来对 RKLLM 模型进行量化。运行以下代码生成 data/inputs.json。
+
+```shell
+# 修改 data/make_input_embeds_for_quantize.py 中的 Juans-Pro-1B 模型路径
+python data/make_input_embeds_for_quantize.py
+```
+
+使用以下代码导出 RKLLM 模型
+
+```shell
+# 修改 data/export_rkllm.py 中的 Juans-Pro-1B 模型路径
+python export/export_rkllm.py
+```
+
+### 5. C++ 演示
+
+在 deploy 目录中，我们提供了板端推理的示例代码。此代码展示了“图像输入到图像特征”的过程，即输入图像经过处理后输出其对应的图像特征。这些特征随后被 RKLLM 模型用于多模态内容推理。
+
+1. 编译和构建
+
+用户可以直接运行 deploy/build-linux.sh 或 deploy/build-android.sh 脚本（将交叉编译器路径替换为实际路径）来编译示例代码。这将在 deploy 目录中生成一个 install/demo_Linux_aarch64 文件夹，其中包含可执行文件 imgenc、llm、demo 以及 lib 文件夹。
+
+```shell
+cd deploy
+# 针对 Linux
+./build-linux.sh
+# 针对 Android
+./build-android.sh
+# 将安装目录推送到设备
+adb push ./install/demo_Linux_aarch64 /data
+# 将模型文件推送到设备
+adb push Janus_pro_vision_rk3588.rknn /data/models
+adb push Janus-Pro-1B-rk3588.rkllm /data/models
+# 将演示图像推送到设备
+adb push ../data/demo.jpg /data/demo_Linux_aarch64
+```
+
+2. 运行演示
+
+进入板端的 /data/demo_Linux_aarch64 目录，并使用以下代码运行示例
+
+```shell
+adb shell
+cd /data/demo_Linux_aarch64
+# 导出库路径
+export LD_LIBRARY_PATH=./lib
+# 创建模型目录的软链接
+ln -s /data/models .
+# 运行 imgenc
+./imgenc models/Janus_pro_vision_rk3588.rknn demo.jpg
+# 运行 llm（纯文本示例）
+./llm models/Janus-Pro-1B-rk3588.rkllm 128 768
+# 运行 demo（多模态示例）
+./demo demo.jpg models/Janus_pro_vision_rk3588.rknn models/Janus-Pro-1B-rk3588.rkllm 128 768
+```
+
+用户可以在终端中查看相关的运行时日志，并在当前目录中获取 img_vec.bin 文件，该文件包含输入图像对应的图像特征。
+
+多模态示例
+
+```
+user: <image>What is in the image?
+robot: The image shows an astronaut sitting on the moon's surface while holding a green bottle labeled "Greenpeace." There are also two objects nearby—a cooler box marked as well-known and another object
+ that looks like part of some equipment or structure.
+```

examples/Janus-Pro-1B_Demo/data/datasets.json

@@ -0,0 +1,22 @@
+[
+    {"image_path": "data/datasets", "image": "1.jpg", "input": "Question: What is correct Python code to generate the content of the image?\nOptions:\nA. for x in range(6):\n  print(x)\nelse:\n  print(\"Finally finished!\")\n\nB. thisdict = {\n  \"brand\": \"Ford\",\n  \"model\": \"Mustang\",\n  \"year\": 1964\n}\n\nprint(len(thisdict))\nC. x = 1\ny = 2.8\nz = 1j\n\nprint(type(x))\nprint(type(y))\nprint(type(z))\n\nD. fruits = [\"apple\", \"banana\", \"cherry\"]\nfor x in fruits:\n  print(x)\nPlease select the correct answer from the options above. \n", "target":"D"},
+    {"image_path": "data/datasets", "image": "2.jpg", "input": "Question: What is correct Python code to generate the content of the image?\nOptions:\nA. class Person:\n  def __init__(self, name, age):\n    self.name = name\n    self.age = age\n\np1 = Person(\"John\", 36)\n\nprint(p1.name)\nprint(p1.age)\nB. fruits = [\"apple\", \"banana\", \"cherry\"]\nfor x in fruits:\n  print(x)\nC. x = min(5, 10, 25)\ny = max(5, 10, 25)\n\nprint(x)\nprint(y)\nD. a = 33\nb = 200\nif b > a:\n  print(\"b is greater than a\")\nPlease select the correct answer from the options above. \n", "target":"D"},
+    {"image_path": "data/datasets", "image": "21.jpg", "input": "Question: Which one is the correct caption of this image?\nOptions:\nA. A man rides a surfboard on a large wave.\nB. a young boy barefoot holding an umbrella touching the horn of a cow\nC. A giraffe standing by a stall in a field.\nD. A stop sign that has been vandalized with graffiti.\nPlease select the correct answer from the options above. \n", "target":"B"},
+    {"image_path": "data/datasets", "image": "22.jpg", "input": "Question: Which one is the correct caption of this image?\nOptions:\nA. A narrow kitchen filled with appliances and cooking utensils.\nB. A person with glasses and a tie in a room.\nC. Tray of vegetables with cucumber, carrots, broccoli and celery.\nD. A pretty young woman riding a surfboard on a wave in the ocean.\nPlease select the correct answer from the options above. \n", "target":"A"},
+    {"image_path": "data/datasets", "image": "241.jpg", "input": "Hint: The passage below describes an experiment. Read the passage and then follow the instructions below.\n\nMadelyn applied a thin layer of wax to the underside of her snowboard and rode the board straight down a hill. Then, she removed the wax and rode the snowboard straight down the hill again. She repeated the rides four more times, alternating whether she rode with a thin layer of wax on the board or not. Her friend Tucker timed each ride. Madelyn and Tucker calculated the average time it took to slide straight down the hill on the snowboard with wax compared to the average time on the snowboard without wax.\nFigure: snowboarding down a hill.\nQuestion: Identify the question that Madelyn and Tucker's experiment can best answer.\nOptions:\nA. Does Madelyn's snowboard slide down a hill in less time when it has a thin layer of wax or a thick layer of wax?\nB. Does Madelyn's snowboard slide down a hill in less time when it has a layer of wax or when it does not have a layer of wax?\nPlease select the correct answer from the options above. \n", "target":"B"},
+    {"image_path": "data/datasets", "image": "252.jpg", "input": "Hint: People can use the engineering-design process to develop solutions to problems. One step in the process is testing if a potential solution meets the requirements of the design.\nThe passage below describes how the engineering-design process was used to test a solution to a problem. Read the passage. Then answer the question below.\n\nLaura and Isabella were making batches of concrete for a construction project. To make the concrete, they mixed together dry cement powder, gravel, and water. Then, they checked if each batch was firm enough using a test called a slump test.\nThey poured some of the fresh concrete into an upside-down metal cone. They left the concrete in the metal cone for 30 seconds. Then, they lifted the cone to see if the concrete stayed in a cone shape or if it collapsed. If the concrete in a batch collapsed, they would know the batch should not be used.\nFigure: preparing a concrete slump test.\nQuestion: Which of the following could Laura and Isabella's test show?\nOptions:\nA. if the concrete from each batch took the same amount of time to dry\nB. if a new batch of concrete was firm enough to use\nPlease select the correct answer from the options above. \n", "target":"B"},
+    {"image_path": "data/datasets", "image": "362.jpg", "input": "Hint: Native copper has the following properties:\nsolid\nnot made by living things\nfound in nature\nfixed crystal structure\nmade of the metal copper\nQuestion: Is native copper a mineral?\nOptions:\nA. no\nB. yes\nPlease select the correct answer from the options above. \n", "target":"B"},
+    {"image_path": "data/datasets", "image": "364.jpg", "input": "Hint: Plastic has the following properties:\nsolid\nno fixed crystal structure\nnot a pure substance\nmade in a factory\nQuestion: Is plastic a mineral?\nOptions:\nA. yes\nB. no\nPlease select the correct answer from the options above. \n", "target":"B"},
+    {"image_path": "data/datasets", "image": "448.jpg", "input": "Hint: Read the text.\nButterflies and moths are easily mistaken for each other, but one distinction between them often appears during their pupal stage. When most butterfly caterpillars reach full size, they attach themselves to a leaf or other object and shed their skin a final time, forming a chrysalis, a hard, shell-like skin, which protects the pupa inside. The chrysalis may be dull and rough or shiny and smooth, usually blending into its surroundings. Most moth caterpillars, by contrast, create a cocoon to protect the pupa, rather than forming a chrysalis. The cocoons usually resemble hard silk pouches, but some moths also incorporate materials like hairs and twigs.\nQuestion: Which term matches the picture?\nOptions:\nA. cocoon\nB. chrysalis\nPlease select the correct answer from the options above. \n", "target":"B"},
+    {"image_path": "data/datasets", "image": "477.jpg", "input": "Hint: Read the text.\nHeat transfer can occur in different ways. Two common ways are through conduction and convection. Conduction occurs when molecules from one object collide with molecules from another object. Burning your hand by touching a hot car door on a sunny summer day is an example of conduction.\nConvection is another form of heat transfer. When a liquid or gas is heated, the heated matter rises upward, away from the heat source. Hot bubbles rising in a pot of water boiling on a stove is an example of convection.\nQuestion: Which term matches the picture?\nOptions:\nA. conduction\nB. convection\nPlease select the correct answer from the options above. \n", "target":"B"},
+    {"image_path": "data/datasets", "image": "1231.jpg", "input": "Question: Which image is more brightful?\nOptions:\nA. The first image\nB. The second image\nPlease select the correct answer from the options above. \n", "target":"A"},
+    {"image_path": "data/datasets", "image": "1232.jpg", "input": "Question: Which image is more brightful?\nOptions:\nA. The first image\nB. The second image\nPlease select the correct answer from the options above. \n", "target":"A"},
+    {"image_path": "data/datasets", "image": "1085.jpg", "input": "Question: is this place crowded?\nOptions:\nA. yes\nB. no\nPlease select the correct answer from the options above. \n", "target":"A"},
+    {"image_path": "data/datasets", "image": "1086.jpg", "input": "Question: is this place crowded?\nOptions:\nA. yes\nB. no\nPlease select the correct answer from the options above. \n", "target":"A"},
+    {"image_path": "data/datasets", "image": "1128.jpg", "input": "Question: In this picture, are the two dolphins the same size?\nOptions:\nA. same\nB. Not the same\nC. Can't judge\nPlease select the correct answer from the options above. \n", "target":"B"},
+    {"image_path": "data/datasets", "image": "1129.jpg", "input": "Question: In this picture, are the two butterfly wings the same shape?\nOptions:\nA. same\nB. Not the same\nC. Can't judge\nPlease select the correct answer from the options above. \n", "target":"B"},
+    {"image_path": "data/datasets", "image": "1200.jpg", "input": "Question: What will happen next?\nOptions:\nA. the motorcyle is gonna go forward\nB. the motorcyle is gonna crash\nC. the motorcyle is gonna go backward\nD. both A,B, and C\nPlease select the correct answer from the options above. \n", "target":"B"},
+    {"image_path": "data/datasets", "image": "1201.jpg", "input": "Question: What will happen next?\nOptions:\nA. this person is gonna stay still\nB. this person is gonna keep walking\nC. this person is gonna fall into the water\nD. both A,B, and C\nPlease select the correct answer from the options above. \n", "target":"C"},
+    {"image_path": "data/datasets", "image": "1554.jpg", "input": "Question: The object shown in this figure:\nOptions:\nA. Is a colorless, flammable liquid that is commonly used as a solvent and fuel\nB. Has a boiling point of 64.7°C\nC. Can be toxic if ingested or absorbed through the skin\nD. None of these options are correct.\nPlease select the correct answer from the options above. \n", "target":"C"},
+    {"image_path": "data/datasets", "image": "1555.jpg", "input": "Question: The object shown in this figure:\nOptions:\nA. Is a lustrous, white metal that is highly reflective and ductile\nB. Has the highest electrical and thermal conductivity of all metals\nC. Has a boiling point of 2,162°C\nD. All of these options are correct.\nPlease select the correct answer from the options above. \n", "target":"D"}
+]