Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 17 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,18 +211,23 @@ Check [the documentation of Optimum Habana](https://huggingface.co/docs/optimum/
## Validated Models

The following model architectures, tasks and device distributions have been validated for 🤗 Optimum Habana:
| | Text Classification | Question Answering | Language Modeling | Summarization | Translation | Image Classification | Audio Classification | Speech Recognition | Single Card | Multi Card | DeepSpeed |
|------------|:-------------------:|:------------------:|:------------------:|:------------------:|:-----------------:|:--------------------:|:--------------------:|:------------------:|:------------------:|:-----------------:|:------------------:|
| BERT | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | ✗ | ✗ | ✗ | ✗ | ✗ | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| RoBERTa | ✗ | :heavy_check_mark: | :heavy_check_mark: | ✗ | ✗ | ✗ | ✗ | ✗ | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| ALBERT | ✗ | :heavy_check_mark: | :heavy_check_mark: | ✗ | ✗ | ✗ | ✗ | ✗ | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| DistilBERT | ✗ | :heavy_check_mark: | :heavy_check_mark: | ✗ | ✗ | ✗ | ✗ | ✗ | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| GPT2 | ✗ | ✗ | :heavy_check_mark: | ✗ | ✗ | ✗ | ✗ | ✗ | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| T5 | ✗ | ✗ | ✗ | :heavy_check_mark: | :heavy_check_mark: | ✗ | ✗ | ✗ | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| ViT | ✗ | ✗ | ✗ | ✗ | ✗ | :heavy_check_mark: | ✗ | ✗ | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Swin | ✗ | ✗ | ✗ | ✗ | ✗ | :heavy_check_mark: | ✗ | ✗ | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Wav2Vec2 | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
| Stable Diffusion | | | | | | | | | :heavy_check_mark: | ✗ | ✗ |

<div align="center">

| Architecture | Single Card | Multi Card | DeepSpeed | <center>Tasks</center> |
|--------------|:-----------:|:----------:|:---------:|------------------------|
| BERT | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | <li>[text classification](https://github.com/huggingface/optimum-habana/tree/main/examples/text-classification)</li><li>[question answering](https://github.com/huggingface/optimum-habana/tree/main/examples/question-answering)</li><li>[language modeling](https://github.com/huggingface/optimum-habana/tree/main/examples/language-modeling)</li> |
| RoBERTa | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | <li>[question answering](https://github.com/huggingface/optimum-habana/tree/main/examples/question-answering)</li><li>[language modeling](https://github.com/huggingface/optimum-habana/tree/main/examples/language-modeling)</li> |
| ALBERT | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | <li>[question answering](https://github.com/huggingface/optimum-habana/tree/main/examples/question-answering)</li><li>[language modeling](https://github.com/huggingface/optimum-habana/tree/main/examples/language-modeling)</li> |
| DistilBERT |:heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | <li>[question answering](https://github.com/huggingface/optimum-habana/tree/main/examples/question-answering)</li><li>[language modeling](https://github.com/huggingface/optimum-habana/tree/main/examples/language-modeling)</li> |
| GPT2 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | <li>[language modeling](https://github.com/huggingface/optimum-habana/tree/main/examples/language-modeling)</li> |
| T5 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | <li>[summarization](https://github.com/huggingface/optimum-habana/tree/main/examples/summarization)</li><li>[translation](https://github.com/huggingface/optimum-habana/tree/main/examples/translation)</li> |
| ViT | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | <li>[image classification](https://github.com/huggingface/optimum-habana/tree/main/examples/image-classification)</li> |
| Swin | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | <li>[image classification](https://github.com/huggingface/optimum-habana/tree/main/examples/image-classification)</li> |
| Wav2Vec2 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | <li>[audio classification](https://github.com/huggingface/optimum-habana/tree/main/examples/audio-classification)</li><li>[speech recognition](https://github.com/huggingface/optimum-habana/tree/main/examples/speech-recognition)</li> |
| Stable Diffusion | :heavy_check_mark: | ✗ | ✗ | <li>[text-to-image generation](https://github.com/huggingface/optimum-habana/tree/main/examples/stable-diffusion)</li> |

</div>

Other models and tasks supported by the 🤗 Transformers library may also work. You can refer to this [section](https://github.com/huggingface/optimum-habana#how-to-use-it) for using them with 🤗 Optimum Habana. Besides, [this page](https://github.com/huggingface/optimum-habana/tree/main/examples) explains how to modify any [example](https://github.com/huggingface/transformers/tree/main/examples/pytorch) from the 🤗 Transformers library to make it work with 🤗 Optimum Habana.

Expand Down
24 changes: 12 additions & 12 deletions docs/source/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,18 +22,18 @@ limitations under the License.

The table below shows which model architectures, tasks and device distributions are currently supported for 🤗 Optimum Habana:

| | Text Classification | Question Answering | Language Modeling | Summarization | Translation | Image Classification | Audio Classification | Speech Recognition | Single Card | Multi Card | DeepSpeed |
|--------------------|:-------------------:|:------------------:|:------------------:|:-------------:|:-----------:|:--------------------:|:--------------------:|:------------------:|:-----------:|:----------:|:---------:|
| BERT | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
| RoBERTa | ❌ | | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | ✅ | ✅ |
| ALBERT | ❌ | | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | ✅ | ✅ |
| DistilBERT | ❌ | | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | | ✅ | ✅ |
| GPT2 | ❌ | ❌ | | ❌ | ❌ | ❌ | ❌ | ❌ | | ✅ | ✅ |
| T5 | ❌ | ❌ | ❌ | | ✅ | ❌ | ❌ | ❌ | | ✅ | ✅ |
| ViT | ❌ | ❌ | ❌ | ❌ | ❌ | | ❌ | ❌ | | ✅ | ✅ |
| Swin | ❌ | ❌ | ❌ | ❌ | ❌ | | ❌ | ❌ | | ✅ | ✅ |
| Wav2Vec2 | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ✅ | ✅ | ✅ | ✅ |
| Stable Diffusion | | | | | | | | | ✅ | | ❌ |
| Architecture | Single Card | Multi Card | DeepSpeed | <center>Tasks</center> |
|------------------|:-----------:|:----------:|:---------:|------------------------|
| BERT | ✅ | ✅ | ✅ | <li>[text classification](https://github.com/huggingface/optimum-habana/tree/main/examples/text-classification)</li><li>[question answering](https://github.com/huggingface/optimum-habana/tree/main/examples/question-answering)</li><li>[language modeling](https://github.com/huggingface/optimum-habana/tree/main/examples/language-modeling)</li> |
| RoBERTa | ✅ | ✅ | | <li>[question answering](https://github.com/huggingface/optimum-habana/tree/main/examples/question-answering)</li><li>[language modeling](https://github.com/huggingface/optimum-habana/tree/main/examples/language-modeling)</li> |
| ALBERT | ✅ | ✅ | | <li>[question answering](https://github.com/huggingface/optimum-habana/tree/main/examples/question-answering)</li><li>[language modeling](https://github.com/huggingface/optimum-habana/tree/main/examples/language-modeling)</li> |
| DistilBERT | ✅ | ✅ | | <li>[question answering](https://github.com/huggingface/optimum-habana/tree/main/examples/question-answering)</li><li>[language modeling](https://github.com/huggingface/optimum-habana/tree/main/examples/language-modeling)</li> |
| GPT2 | | ✅ | ✅ | <li>[language modeling](https://github.com/huggingface/optimum-habana/tree/main/examples/language-modeling)</li> |
| T5 | ✅ | ✅ | | <li>[summarization](https://github.com/huggingface/optimum-habana/tree/main/examples/summarization)</li><li>[translation](https://github.com/huggingface/optimum-habana/tree/main/examples/translation)</li> |
| ViT | | ✅ | ✅ | <li>[image classification](https://github.com/huggingface/optimum-habana/tree/main/examples/image-classification)</li> |
| Swin | | ✅ | ✅ | <li>[image classification](https://github.com/huggingface/optimum-habana/tree/main/examples/image-classification)</li> |
| Wav2Vec2 | ✅ | ✅ | ✅ | <li>[audio classification](https://github.com/huggingface/optimum-habana/tree/main/examples/audio-classification)</li><li>[speech recognition](https://github.com/huggingface/optimum-habana/tree/main/examples/speech-recognition)</li> |
| Stable Diffusion | ✅ | ❌ | | <li>[text-to-image generation](https://github.com/huggingface/optimum-habana/tree/main/examples/stable-diffusion)</li> |

Other models and tasks supported by the 🤗 Transformers library may also work. You can refer to the [Quickstart](./quickstart) for examples on using them with 🤗 Optimum Habana. Besides, [this page](https://github.com/huggingface/optimum-habana/tree/main/examples) explains how to modify any [example](https://github.com/huggingface/transformers/tree/main/examples/pytorch) from the 🤗 Transformers library to make it work with 🤗 Optimum Habana.

Expand Down
2 changes: 1 addition & 1 deletion examples/audio-classification/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Audio classification examples
# Audio Classification Examples

The following examples showcase how to fine-tune `Wav2Vec2` for audio classification on Habana Gaudi.

Expand Down
2 changes: 1 addition & 1 deletion examples/image-classification/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Image classification examples
# Image Classification Examples

This directory contains a script that showcases how to fine-tune any model supported by the [`AutoModelForImageClassification` API](https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.AutoModelForImageClassification) (such as [ViT](https://huggingface.co/docs/transformers/main/en/model_doc/vit) or [Swin Transformer](https://huggingface.co/docs/transformers/main/en/model_doc/swin)) on HPUs. They can be used to fine-tune models on both [datasets from the hub](#using-datasets-from-hub) as well as on [your own custom data](#using-your-own-data).

Expand Down
2 changes: 1 addition & 1 deletion examples/language-modeling/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Language model training
# Language Model Training

Fine-tuning (or training from scratch) the library models for language modeling on a text dataset.
GPT-2 is trained or fine-tuned using a causal language modeling (CLM) loss while ALBERT, BERT, DistilBERT and RoBERTa are trained or fine-tuned using a masked language modeling (MLM) loss. You can find more information about the differences between those objectives in our [model summary](https://huggingface.co/transformers/model_summary.html).
Expand Down
2 changes: 1 addition & 1 deletion examples/question-answering/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# SQuAD
# Question Answering Examples on SQuAD

Based on the script [`run_qa.py`](https://github.com/huggingface/transformers/blob/main/examples/pytorch/question-answering/run_qa.py).

Expand Down
18 changes: 17 additions & 1 deletion examples/stable-diffusion/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,20 @@
# Stable Diffusion
<!---
Copyright 2022 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

# Stable Diffusion Examples

This directory contains a script that showcases how to perform text-to-image generation using Stable Diffusion on Habana Gaudi.

Expand Down
2 changes: 1 addition & 1 deletion examples/summarization/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Summarization
# Summarization Examples

This directory contains examples for finetuning and evaluating transformers on summarization tasks.

Expand Down
2 changes: 1 addition & 1 deletion examples/text-classification/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Text classification examples
# Text Classification Examples

## GLUE tasks

Expand Down
2 changes: 1 addition & 1 deletion examples/translation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Translation
# Translation Examples

`run_translation.py` is a lightweight example of how to download and preprocess a dataset from the [🤗 Datasets](https://github.com/huggingface/datasets) library or use your own files (jsonlines or csv), then fine-tune one of the architectures above on it.

Expand Down