NuExtract-1.5-openvino

Extract data with LLM and openvino

NuExtract-tiny-v1.5 is a fine-tuning of Qwen/Qwen2.5-0.5B, trained on a private high-quality dataset for structured information extraction. It supports long documents and several languages (English, French, Spanish, German, Portuguese, and Italian). To use the model, provide an input text and a JSON template describing the information you need to extract.

Note: This model is trained to prioritize pure extraction, so in most cases all text generated by the model is present as is in the original text.

---

• AI structure data Extraction - A streamlit interface with NuExtract-1.5-tiny and openvino-genai to extract data from plain text into structured custom json formats. The Repo also gives a small tutorial on how to convert NuExtract-1.5-tiny into OpenVINO IR format.

---

Pre-requisites

Install dependencies in a new virtual environment

# Step 1: Create virtual environment
python -m venv venv
# Step 2: Activate virtual environment
venv\Scripts\activate
# Step 3: Upgrade pip to latest version
python -m pip install --upgrade pip

This is the OpenVINO IR format of the model, quantized in int8

The model was created with the Optimum-Intel libray cli-command

Dependencies required to create the model

There is an open clash in dependencies versions between optiumum-intel and openvino-genai

⚠️ Exporting tokenizers to OpenVINO is not supported for tokenizers version > 0.19 and openvino version <= 2024.4. Please downgrade to tokenizers version <= 0.19 to export tokenizers to OpenVINO.

So for the model conversion the only dependency you need is

pip install  -U "openvino>=2024.3.0" "openvino-genai"
pip install "torch>=2.1" "nncf>=2.7" "transformers>=4.40.0" "onnx<1.16.2" "optimum>=1.16.1" "accelerate" "datasets>=2.14.6" "git+https://github.com/huggingface/optimum-intel.git" --extra-index-url https://download.pytorch.org/whl/cpu

The instructions are from the amazing OpenVINO notebooks
vanilla pip install will create clashes among dependencies/versions
This command will install, among others:

tokenizers==0.20.3
torch==2.5.1+cpu
transformers==4.46.3
nncf==2.14.0
numpy==2.1.3
onnx==1.16.1
openvino==2024.5.0
openvino-genai==2024.5.0.0
openvino-telemetry==2024.5.0
openvino-tokenizers==2024.5.0.0
optimum==1.23.3
optimum-intel @ git+https://github.com/huggingface/optimum-intel.git@c454b0000279ac9801302d726fbbbc1152733315

How to quantized the original model

After the previous step you are enabled to run the following command (considering that you downloaded all the model weights and files into a subfolder called NuExtract-1.5-tiny from the official model repository)

optimum-cli export openvino --model NuExtract-1.5-tiny --task text-generation-with-past --trust-remote-code --weight-format int8 ov_NuExtract-1.5-tiny

this will start the process and produce the following messages, without any fatal error

Dependencies required to run the model with openvino-genai

If you simply need to run already converted models into OpenVINO IR format, you need to install only openvino-genai

pip install openvino-genai==2024.5.0

How to use the model with openvino-genai

considering you also have python-rich installed (that is coming together with optimum-intel... otherwise pip install rich)

"""
followed official tutorial
https://docs.openvino.ai/2024/notebooks/llm-question-answering-with-output.html
"""
# MAIN IMPORTS
import warnings
warnings.filterwarnings(action='ignore')
import datetime
from rich.console import Console
from rich.panel import Panel
import openvino_genai as ov_genai
# SETTING CONSOLE WIDTH
console = Console(width=80)
# LOADING THE MODEL
console.print('Loading the model...', end='')
model_dir = 'ov_NuExtract-1.5-tiny'
pipe = ov_genai.LLMPipeline(model_dir, 'CPU')
console.print('✅  done')
console.print('Ready for generation')
# PROMPT FORMATTING
jsontemplate = """{
    "Model": {
        "Name": "",
        "Number of parameters": "",
        "Number of max token": "",
        "Architecture": []
    },
    "Usage": {
        "Use case": [],
        "Licence": ""
    }
}"""
text = """We introduce Mistral 7B, a 7–billion-parameter language model engineered for
superior performance and efficiency. Mistral 7B outperforms the best open 13B
model (Llama 2) across all evaluated benchmarks, and the best released 34B
model (Llama 1) in reasoning, mathematics, and code generation. Our model
leverages grouped-query attention (GQA) for faster inference, coupled with sliding
window attention (SWA) to effectively handle sequences of arbitrary length with a
reduced inference cost. We also provide a model fine-tuned to follow instructions,
Mistral 7B – Instruct, that surpasses Llama 2 13B – chat model both on human and
automated benchmarks. Our models are released under the Apache 2.0 license.
Code: <https://github.com/mistralai/mistral-src>
Webpage: <https://mistral.ai/news/announcing-mistral-7b/>"""

prompt = f"""<|input|>\n### Template:
{jsontemplate}

### Text:
{text}
<|output|>
"""
# START PIPELINE setting eos_token_id = 151643
start = datetime.datetime.now()
with console.status("Generating json reply", spinner='dots8',):
    output = pipe.generate(prompt, temperature=0.2, 
                        do_sample=True, 
                        max_new_tokens=500, 
                        repetition_penalty=1.178,
                        eos_token_id = 151643)
delta = datetime.datetime.now() - start
# PRINT THE OUTPUT
console.print(output)
console.rule()
console.print(f'Generated in {delta}')

For the graphic interface also:

pip install tiktoken streamlit

Using Hugging Face Hub

You can find directly the model weights in OpenVINO IR format in my HF Model Repository

TIP: WHERE TO DOWNLOAD

put all the files into a subfolder called ov_NuExtract-1.5-tiny

Resources

• tutorial
• model-optimization-guide
• genai-guide
• examples
• API documentation:

Basic usage

import warnings
warnings.filterwarnings(action='ignore')
import datetime
import tiktoken
import json
from rich.console import Console
from rich.panel import Panel
import openvino_genai as ov_genai

console = Console(width=80)
console.print('Loading the model...', end='')
model_dir = 'ov_NuExtract-1.5-tiny'
pipe = ov_genai.LLMPipeline(model_dir, 'CPU')
console.print('✅  done')
console.print('Ready for generation')
# ACTIONS
jsontemplate = """{
    "Model": {
        "Name": "",
        "Number of parameters": "",
        "Number of max token": "",
        "Architecture": []
    },
    "Usage": {
        "Use case": [],
        "Licence": ""
    }
}"""
text = """We introduce Mistral 7B, a 7–billion-parameter language model engineered for
superior performance and efficiency. Mistral 7B outperforms the best open 13B
model (Llama 2) across all evaluated benchmarks, and the best released 34B
model (Llama 1) in reasoning, mathematics, and code generation. Our model
leverages grouped-query attention (GQA) for faster inference, coupled with sliding
window attention (SWA) to effectively handle sequences of arbitrary length with a
reduced inference cost. We also provide a model fine-tuned to follow instructions,
Mistral 7B – Instruct, that surpasses Llama 2 13B – chat model both on human and
automated benchmarks. Our models are released under the Apache 2.0 license.
Code: <https://github.com/mistralai/mistral-src>
Webpage: <https://mistral.ai/news/announcing-mistral-7b/>"""

prompt = f"""<|input|>\n### Template:
{jsontemplate}

### Text:
{text}
<|output|>
"""
start = datetime.datetime.now()
with console.status("Generating json reply", spinner='dots8',):
    output = pipe.generate(prompt, temperature=0.2, 
                        do_sample=True, 
                        max_new_tokens=500, 
                        repetition_penalty=1.178,
                        eos_token_id = 151643)
delta = datetime.datetime.now() - start
console.print(output)
console.rule()
console.print(f'Generated in {delta}')

---

Run with Streamlit interface

Download the files from the repo

• logo.png
• stapp1.5-nuextractTINY.py all of them in the main project directory

then with the venv activated run in the terminal:

streamlit run .\stapp1.5-nuextractTINY.py

---