fixed docs

.readthedocs.yaml

@@ -5,28 +5,16 @@
 # Required
 version: 2
 
-# Set the OS, Python version and other tools you might need
-build:
-  os: ubuntu-22.04
-  tools:
-    python: "3.12"
-    # You can also specify other tool versions:
-    # nodejs: "19"
-    # rust: "1.64"
-    # golang: "1.19"
+# Set the version of Python and other tools you might need
+# build:
+#   os: ubuntu-22.04
+#   tools:
+#     python: "3.10"
 
-# Build documentation in the "docs/" directory with Sphinx
-sphinx:
-  configuration: docs/conf.py
+mkdocs:
+  configuration: mkdocs.yml
 
-# Optionally build your docs in additional formats such as PDF and ePub
-# formats:
-#    - pdf
-#    - epub
-
-# Optional but recommended, declare the Python requirements required
-# to build your documentation
-# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
-# python:
-#    install:
-#    - requirements: docs/requirements.txt
+# Optionally declare the Python requirements required to build your docs
+python:
+   install:
+   - requirements: docs/requirements.txt

README.md

@@ -311,7 +311,7 @@ To cite this repository:
   author = {Henry Ndubuaku},
   title = {NanoDL: A Jax-based library for designing and training transformer models from scratch.},
   url = {https://github.com/hmunachi/nanodl},
-  version = {0.1.0},
+  version = {1.0.1dev},
   year = {2024},
 }
 ```

docs/api.md

@@ -0,0 +1,104 @@
+# Welcome to NanoDL Documentation
+
+## API Reference
+
+::: nanodl.GAT
+::: nanodl.GraphAttentionLayer
+::: nanodl.T5
+::: nanodl.T5DataParallelTrainer
+::: nanodl.T5Encoder
+::: nanodl.T5Decoder
+::: nanodl.T5EncoderBlock
+::: nanodl.T5DecoderBlock
+::: nanodl.ViT
+::: nanodl.ViTDataParallelTrainer
+::: nanodl.ViTBlock
+::: nanodl.ViTEncoder
+::: nanodl.PatchEmbedding
+::: nanodl.Transformer
+::: nanodl.TransformerDataParallelTrainer
+::: nanodl.TransformerEncoder
+::: nanodl.TransformerDecoderBlock
+::: nanodl.PositionalEncoding
+::: nanodl.TokenAndPositionEmbedding
+::: nanodl.MultiHeadAttention
+::: nanodl.AddNorm
+::: nanodl.CLIP
+::: nanodl.CLIPDataParallelTrainer
+::: nanodl.ImageEncoder
+::: nanodl.TextEncoder
+::: nanodl.SelfMultiHeadAttention
+::: nanodl.LaMDA
+::: nanodl.LaMDADataParallelTrainer
+::: nanodl.LaMDABlock
+::: nanodl.LaMDADecoder
+::: nanodl.RelativeMultiHeadAttention
+::: nanodl.DiffusionModel
+::: nanodl.DiffusionDataParallelTrainer
+::: nanodl.UNet
+::: nanodl.UNetDownBlock
+::: nanodl.UNetUpBlock
+::: nanodl.UNetResidualBlock
+::: nanodl.GPT3
+::: nanodl.GPT4
+::: nanodl.GPTDataParallelTrainer
+::: nanodl.GPT3Block
+::: nanodl.GPT4Block
+::: nanodl.GPT3Decoder
+::: nanodl.GPT4Decoder
+::: nanodl.PositionWiseFFN
+::: nanodl.LlaMA2
+::: nanodl.LlaMADataParallelTrainer
+::: nanodl.RotaryPositionalEncoding
+::: nanodl.LlaMA2Decoder
+::: nanodl.LlaMA2DecoderBlock
+::: nanodl.GroupedRotaryMultiHeadAttention
+::: nanodl.Mistral
+::: nanodl.MistralDataParallelTrainer
+::: nanodl.MistralDecoder
+::: nanodl.MistralDecoderBlock
+::: nanodl.GroupedRotaryShiftedWindowMultiHeadAttention
+::: nanodl.Mixtral
+::: nanodl.MixtralDecoder
+::: nanodl.MixtralDecoderBlock
+::: nanodl.Whisper
+::: nanodl.WhisperDataParallelTrainer
+::: nanodl.WhisperSpeechEncoder
+::: nanodl.WhisperSpeechEncoderBlock
+::: nanodl.GAT
+::: nanodl.GraphAttentionLayer
+::: nanodl.NaiveBayesClassifier
+::: nanodl.LinearRegression
+::: nanodl.LogisticRegression
+::: nanodl.GaussianProcess
+::: nanodl.KMeans
+::: nanodl.GaussianMixtureModel
+::: nanodl.PCA
+::: nanodl.Dataset
+::: nanodl.ArrayDataset
+::: nanodl.DataLoader
+::: nanodl.batch_cosine_similarities
+::: nanodl.batch_pearsonr
+::: nanodl.classification_scores
+::: nanodl.count_parameters
+::: nanodl.entropy
+::: nanodl.gini_impurity
+::: nanodl.hamming
+::: nanodl.jaccard
+::: nanodl.kl_divergence
+::: nanodl.mean_reciprocal_rank
+::: nanodl.zero_pad_sequences
+::: nanodl.bleu
+::: nanodl.cider_score
+::: nanodl.meteor
+::: nanodl.perplexity
+::: nanodl.rouge
+::: nanodl.word_error_rate
+::: nanodl.adjust_brightness
+::: nanodl.adjust_contrast
+::: nanodl.flip_image
+::: nanodl.gaussian_blur
+::: nanodl.normalize_images
+::: nanodl.random_crop
+::: nanodl.random_flip_image
+::: nanodl.sobel_edge_detection

docs/index.md

@@ -0,0 +1,56 @@
+## Overview
+
+Developing and training transformer-based models is typically resource-intensive and time-consuming and AI/ML experts frequently need to build smaller-scale versions of these models for specific problems. Jax, a low-resource yet powerful framework, accelerates the development of neural networks, but existing resources for transformer development in Jax are limited. NanoDL addresses this challenge with the following features:
+
+- A wide array of blocks and layers, facilitating the creation of customised transformer models from scratch.
+- An extensive selection of models like LlaMa2, Mistral, Mixtral, GPT3, GPT4 (inferred), T5, Whisper, ViT, Mixers, GAT, CLIP, and more, catering to a variety of tasks and applications.
+- Data-parallel distributed trainers so developers can efficiently train large-scale models on multiple GPUs or TPUs, without the need for manual training loops.
+- Dataloaders, making the process of data handling for Jax/Flax more straightforward and effective.
+- Custom layers not found in Flax/Jax, such as RoPE, GQA, MQA, and SWin attention, allowing for more flexible model development.
+- GPU/TPU-accelerated classical ML models like PCA, KMeans, Regression, Gaussian Processes etc., akin to SciKit Learn on GPU.
+- Modular design so users can blend elements from various models, such as GPT, Mixtral, and LlaMa2, to craft unique hybrid transformer models.
+- A range of advanced algorithms for NLP and computer vision tasks, such as Gaussian Blur, BLEU etc.
+- Each model is contained in a single file with no external dependencies, so the source code can also be easily used. 
+
+Feedback on any of our discussion, issue and pull request threads are welcomed! Please report any feature requests, issues, questions or concerns in the [discussion forum](https://github.com/hmunachi/nanodl/discussions), or just let us know what you're working on! In case you want to reach out directly, we're at [email protected].
+
+# Contribution
+
+This is the first iteration of this project, roughness is expected, contributions are therefore highly encouraged! Follow the recommended steps:
+
+- Raise the issue/discussion to get second opinions
+- Fork the repository
+- Create a branch
+- Make your changes without ruining the design patterns
+- Write tests for your changes if necessary
+- Install locally with `pip install -e .`
+- Run tests with `python -m unittest discover -s tests`
+- Then submit a pull request from branch.
+
+Contributions can be made in various forms:
+
+- Writing documentation.
+- Fixing bugs.
+- Implementing papers.
+- Writing high-coverage tests.
+- OPtimizing existing codes.
+- Experimenting and submitting real-world examples to the examples section.
+- Reporting bugs.
+- Responding to reported issues.
+
+Coming features include:
+- Reinforcement Learning With Human Feedback (RLHF).
+- Tokenizers.
+- Code optimisations.
+
+To follow up or share thoughts, follow [here](https://forms.gle/vwveb9SKdPYywHx9A)
+
+## Sponsorships
+
+The name "NanoDL" stands for Nano Deep Learning. Models are exploding in size, therefore gate-keeping 
+experts and companies with limited resources from building flexible models without prohibitive costs.
+Following the success of Phi models, the long-term goal is to build and train nano versions of all available models,
+while ensuring they compete with the original models in performance, with total 
+number of parameters not exceeding 1B. Trained weights will be made available via this library.
+Any form of sponsorship, funding, grants or contribution will help with training resources.
+You can sponsor via the provided button, or reach out via [email protected].

docs/requirements.in

@@ -0,0 +1,3 @@
+mkdocs
+mkdocstrings[python]
+markdown-include

docs/requirements.txt

@@ -0,0 +1,66 @@
+#
+# This file is autogenerated by pip-compile with python 3.10
+# To update, run:
+#
+#    pip-compile docs/requirements.in
+#
+click==8.1.3
+    # via mkdocs
+ghp-import==2.1.0
+    # via mkdocs
+griffe==0.22.0
+    # via mkdocstrings-python
+importlib-metadata==4.12.0
+    # via mkdocs
+jinja2==3.1.2
+    # via
+    #   mkdocs
+    #   mkdocstrings
+markdown==3.3.7
+    # via
+    #   markdown-include
+    #   mkdocs
+    #   mkdocs-autorefs
+    #   mkdocstrings
+    #   pymdown-extensions
+markdown-include==0.6.0
+    # via -r docs/requirements.in
+markupsafe==2.1.1
+    # via
+    #   jinja2
+    #   mkdocstrings
+mergedeep==1.3.4
+    # via mkdocs
+mkdocs==1.3.0
+    # via
+    #   -r docs/requirements.in
+    #   mkdocs-autorefs
+    #   mkdocstrings
+mkdocs-autorefs==0.4.1
+    # via mkdocstrings
+mkdocstrings[python]==0.19.0
+    # via
+    #   -r docs/requirements.in
+    #   mkdocstrings-python
+mkdocstrings-python==0.7.1
+    # via mkdocstrings
+packaging==21.3
+    # via mkdocs
+pymdown-extensions==9.5
+    # via mkdocstrings
+pyparsing==3.0.9
+    # via packaging
+python-dateutil==2.8.2
+    # via ghp-import
+pyyaml==6.0
+    # via
+    #   mkdocs
+    #   pyyaml-env-tag
+pyyaml-env-tag==0.1
+    # via mkdocs
+six==1.16.0
+    # via python-dateutil
+watchdog==2.1.9
+    # via mkdocs
+zipp==3.8.0
+    # via importlib-metadata