|
1 | | -# KerasNLP Example Models |
| 1 | +# KerasNLP Examples |
2 | 2 |
|
3 | | -This examples directory contains complete recipes for training popular model |
4 | | -architectures using KerasNLP. These are not part of the library itself, but |
5 | | -rather serve to demonstrate how to use the library for common tasks, while |
6 | | -simultaneously providing a mechanism to rigorously test library components. |
7 | | - |
8 | | -This directory is complementary to the NLP examples on |
9 | | -[keras.io](https://keras.io/examples/). If you want to contribute a KerasNLP |
10 | | -example and you're not sure if it should live on keras.io or in this directory, |
11 | | -here's how they differ: |
12 | | - |
13 | | -- If an example can fit in < 300 lines of code and run in a Colab, |
14 | | - put it on keras.io. |
15 | | -- If an example is too big for a single script or has high compute requirements, |
16 | | - add it here. |
17 | | - |
18 | | -In general, we will have a fairly high bar for what models we support in this |
19 | | -directory. They should be widely used, practical models for solving standard |
20 | | -problems in NLP. |
21 | | - |
22 | | -## Anatomy of an example |
23 | | - |
24 | | -Given a model named `modelname`, which involves both pretraining and finetuning |
25 | | -on a downstream task, the contents of the `modelname` directory should be as |
26 | | -follows: |
27 | | - |
28 | | -```shell |
29 | | -modelname |
30 | | -├── README.md |
31 | | -├── __init__.py |
32 | | -├── modelname_config.py |
33 | | -├── modelname_model.py |
34 | | -├── modelname_preprocess.py |
35 | | -├── modelname_train.py |
36 | | -└── modelname_finetune_X.py |
37 | | -``` |
38 | | - |
39 | | -- `README.md`: The README should contain complete instructions for downloading |
40 | | - data and training a model from scratch. |
41 | | -- `__init__.py`: Empty (it's for imports). |
42 | | -- `modelname_config.py`: This file should contain most of the configuration for |
43 | | - the model architecture, learning rate, etc, using simple Python constants. We |
44 | | - would like to avoid complex configuration setups (json, yaml, etc). |
45 | | -- `modelname_preprocess.py`: If necessary. Standalone script to preprocess |
46 | | - inputs. If possible, prefer doing preprocessing dynamically with tf.data |
47 | | - inside the training and finetuning scripts. |
48 | | -- `modelname_model.py`: This file should contain the actual `keras.Model` and |
49 | | - any custom layers needed for the example. Use KerasNLP components where ever |
50 | | - possible. |
51 | | -- `modelname_train.py`: This file should be a runnable training script for |
52 | | - pretraining. If possible, this script should preprocess data dynamically |
53 | | - during training using `tf.data` and KerasNLP components (e.g. tokenizers). |
54 | | -- `modelname_finetune_X.py`: Optional. There can be any number of these files, |
55 | | - for each task `X` we would like to support for finetuning. The file should be |
56 | | - a runnable training script which loads and finetunes a pretrained model. |
57 | | - |
58 | | -## Instructions for running on Google Cloud |
59 | | - |
60 | | -TODO(https://github.com/keras-team/keras-nlp/issues/178) |
| 3 | +The `examples/` directly contains scripts built on top of the library that do not fit well into |
| 4 | +the colab format used on [keras.io](https://keras.io/examples/). This includes recipes for |
| 5 | +pre-training models and evaluating models on benchmarks such as GLUE. |
0 commit comments