01 Create compute.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c8dadf55-c643-492d-ad0a-2b007f178c14",
   "metadata": {},
   "source": [
    "# Create a compute target and check install\n",
    "\n",
    "__Goal__: Set up a training cluster \n",
    "\n",
    "This notebook is a pre-flight check to ensure that you've got the necessary requirements accessible and that a compute cluster exists. Like the subsequent notebooks, this is intended to be run on an [AzureML compute instance](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-create-attach-compute-studio#portal-create)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c8b69485-16a9-475f-8de2-9ffea79dee42",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here to check that requirements have been installed\n",
    "import onnx\n",
    "import transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "968be6c0-1d70-4081-8c55-90463353dddf",
   "metadata": {},
   "source": [
    "If the prior cell failed, please check that you are using the `AzureML_Py3.8` conda environment and that you have run `conda activate azureml_py38 && pip install -r requirements.txt` on this compute instance. \n",
    "\n",
    "Now we'll import the libraries we'll actually use and load the config file that specifies the cluster name and machine type. Throughout this notebook and those that follow we'll draw our configuration from `src/config.yml` where possible. This file contains most of the settings that need to be customized for a new job type. Reading through it gives a sense of how to configure an experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "839f4501-9ab7-4855-8d8f-af869f0780b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "import azureml.core\n",
    "\n",
    "with open('src/config.yml', 'r') as f:\n",
    "    config = yaml.safe_load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ae1849b-4e4b-4604-8297-d8b0305b7530",
   "metadata": {},
   "source": [
    "Now we'll connect to the AzureML [workspace]. The workspace can be thought of as the namespace that ties together all the models, runs, datasets, compute instances, cluster instances, and linked services we'll access. Each notebook will connect to this workspace before performing any operations with AzureML. \n",
    "\n",
    "We instantiate a connection to the workspace using a configuration file that is automatically provided on the AzureML compute instances but which [we must create] if we run this notebook on our own desktop or laptop machine. \n",
    "\n",
    "[workspace]: https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace\n",
    "[we must create]: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-environment#workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "622c7144-7892-46bd-99ec-59d0f0bd42d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "workspace = azureml.core.Workspace.from_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edee12a0-7ee0-438f-88a3-fc5f8676cdab",
   "metadata": {},
   "source": [
    "Next we'll connect to the cluster or [create it if it does not exist](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-create-attach-compute-cluster?tabs=python). This is easily doable through the AzureML Studio interface, but doing the creation here ensures we'll have an appropriate target (in name, size, and kind) for the next steps. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a1c3647-7cef-4e57-8c04-368cddc80505",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Does the cluster exist? If not, then create it\n",
    "try:\n",
    "    cluster = azureml.core.compute.ComputeTarget(\n",
    "        workspace=workspace, \n",
    "        name=config['compute_target']\n",
    "    )\n",
    "    print('Found existing compute cluster')\n",
    "except azureml.core.compute_target.ComputeTargetException:\n",
    "    compute_config = azureml.core.compute.AmlCompute.provisioning_configuration(\n",
    "        vm_size=config['compute_size'],\n",
    "        max_nodes=config['compute_node_count']\n",
    "    )\n",
    "    cluster = azureml.core.compute.ComputeTarget.create(\n",
    "        workspace=workspace,\n",
    "        name=config['compute_target'], \n",
    "        provisioning_configuration=compute_config\n",
    "    )\n",
    "    \n",
    "cluster.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3335331f-931c-4515-956e-befecdb4d33e",
   "metadata": {},
   "source": [
    "We now have:\n",
    "\n",
    "- verified that the requirements were installed into this conda environment\n",
    "- a functioning compute cluster we can use as a target for training\n",
    "\n",
    "Let's move on to the next notebook. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:aml]",
   "language": "python",
   "name": "conda-env-aml-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}