Skip to content

Commit

Permalink
Update llama example information (#12640)
Browse files Browse the repository at this point in the history
Co-authored-by: ATMxsp01 <[email protected]>
  • Loading branch information
ATMxsp01 and ATMxsp01 authored Jan 2, 2025
1 parent 81211fd commit 6231896
Show file tree
Hide file tree
Showing 4 changed files with 9 additions and 15 deletions.
6 changes: 3 additions & 3 deletions python/llm/example/GPU/HuggingFace/LLM/llama3.1/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Llama3.1
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on Llama3.1 models on [Intel GPUs](../../../README.md). For illustration purposes, we utilize the [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) as a reference Llama3.1 model.
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on Llama3.1 models on [Intel GPUs](../../../README.md). For illustration purposes, we utilize the [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) as a reference Llama3.1 model.

## 0. Requirements
To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
Expand Down Expand Up @@ -104,12 +104,12 @@ python ./generate.py --repo-id-or-model-path REPO_ID_OR_MODEL_PATH --prompt PROM
```

Arguments info:
- `--repo-id-or-model-path REPO_ID_OR_MODEL_PATH`: argument defining the huggingface repo id for the Llama3.1 model (e.g. `meta-llama/Meta-Llama-3.1-8B-Instruct`) to be downloaded, or the path to the huggingface checkpoint folder. It is default to be `'meta-llama/Meta-Llama-3.1-8B-Instruct'`.
- `--repo-id-or-model-path REPO_ID_OR_MODEL_PATH`: argument defining the huggingface repo id for the Llama3.1 model (e.g. `meta-llama/Llama-3.1-8B-Instruct`) to be downloaded, or the path to the huggingface checkpoint folder. It is default to be `'meta-llama/Llama-3.1-8B-Instruct'`.
- `--prompt PROMPT`: argument defining the prompt to be infered (with integrated prompt format for chat). It is default to be `'What is AI?'`.
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `32`.

#### Sample Output
#### [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)
#### [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)
```log
Inference time: xxxx s
-------------------- Prompt --------------------
Expand Down
4 changes: 2 additions & 2 deletions python/llm/example/GPU/HuggingFace/LLM/llama3.1/generate.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ def get_prompt(user_input: str, chat_history: list[tuple[str, str]],

if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Predict Tokens using `generate()` API for Llama3.1 model')
parser.add_argument('--repo-id-or-model-path', type=str, default="meta-llama/Meta-Llama-3.1-8B-Instruct",
help='The huggingface repo id for the Llama3 (e.g. `meta-llama/Meta-Llama-3.1-8B-Instruct`) to be downloaded'
parser.add_argument('--repo-id-or-model-path', type=str, default="meta-llama/Llama-3.1-8B-Instruct",
help='The huggingface repo id for the Llama3 (e.g. `meta-llama/Llama-3.1-8B-Instruct`) to be downloaded'
', or the path to the huggingface checkpoint folder')
parser.add_argument('--prompt', type=str, default="What is AI?",
help='Prompt to infer')
Expand Down
8 changes: 4 additions & 4 deletions python/llm/example/GPU/HuggingFace/LLM/llama3.2/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Llama3.2
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on Llama3.2 models on [Intel GPUs](../../../README.md). For illustration purposes, we utilize the [meta-llama/Meta-Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.2-3B-Instruct) and [meta-llama/Meta-Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.2-1B-Instruct) as reference Llama3.2 models.
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on Llama3.2 models on [Intel GPUs](../../../README.md). For illustration purposes, we utilize the [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) and [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) as reference Llama3.2 models.

## 0. Requirements
To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
Expand Down Expand Up @@ -104,12 +104,12 @@ python ./generate.py --repo-id-or-model-path REPO_ID_OR_MODEL_PATH --prompt PROM
```

Arguments info:
- `--repo-id-or-model-path REPO_ID_OR_MODEL_PATH`: argument defining the huggingface repo id for the Llama3.2 model (e.g. `meta-llama/Meta-Llama-3.2-3B-Instruct`) to be downloaded, or the path to the huggingface checkpoint folder. It is default to be `'meta-llama/Meta-Llama-3.2-3B-Instruct'`.
- `--repo-id-or-model-path REPO_ID_OR_MODEL_PATH`: argument defining the huggingface repo id for the Llama3.2 model (e.g. `meta-llama/Llama-3.2-3B-Instruct`) to be downloaded, or the path to the huggingface checkpoint folder. It is default to be `'meta-llama/Llama-3.2-3B-Instruct'`.
- `--prompt PROMPT`: argument defining the prompt to be infered (with integrated prompt format for chat). It is default to be `'What is AI?'`.
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `32`.

#### Sample Output
#### [meta-llama/Meta-Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.2-3B-Instruct)
#### [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)
```log
Inference time: xxxx s
-------------------- Prompt --------------------
Expand All @@ -126,7 +126,7 @@ What is AI?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Artificial Intelligence (AI) refers to the development of computer systems that can perform tasks that would typically require human intelligence, such as learning, problem-solving, and
```

#### [meta-llama/Meta-Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.2-1B-Instruct)
#### [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct)
```log
Inference time: xxxx s
-------------------- Prompt --------------------
Expand Down
6 changes: 0 additions & 6 deletions python/llm/example/GPU/HuggingFace/LLM/llama3/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,6 @@ conda create -n llm python=3.11
conda activate llm
# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/

# transformers>=4.33.0 is required for Llama3 with IPEX-LLM optimizations
pip install transformers==4.37.0
```

#### 1.2 Installation on Windows
Expand All @@ -27,9 +24,6 @@ conda activate llm

# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/

# transformers>=4.33.0 is required for Llama3 with IPEX-LLM optimizations
pip install transformers==4.37.0
```

### 2. Configures OneAPI environment variables for Linux
Expand Down

0 comments on commit 6231896

Please sign in to comment.