examples/goerli_testnet_example.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a7df523",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2023 Golem Factory GmbH\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#    http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4524a10",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This example is modification of \"Linear Regression with a Real Dataset\" Notebook from Machine Learning Crash Course created by Google and available here: https://github.com/google/eng-edu/blob/main/ml/cc/exercises/linear_regression_with_a_real_dataset.ipynb\n",
    "# This exercises is made available under the Apache License Version 2.0. You may obtain a copy of the License here: http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Data Set used in this example contains data drawn from the 1990 U.S. Census. It is also available as part of Machine Learning Crash Course created by Google and available here: https://download.mlcc.google.com/mledu-datasets/california_housing_train.csv \n",
    "# Data Set description is available here: https://developers.google.com/machine-learning/crash-course/california-housing-data-description"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d564dfbd",
   "metadata": {},
   "source": [
    "## Jupyter on Golem - Goerli Testnet Example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5bef596",
   "metadata": {},
   "source": [
    "Let's start with typing %help command to display all Jupyter on Golem magic commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b11c8b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "%help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a181afc4",
   "metadata": {},
   "source": [
    "Now let's check our wallet address and available funds with %status command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90a60779",
   "metadata": {},
   "outputs": [],
   "source": [
    "%status"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad493991",
   "metadata": {},
   "source": [
    "If You do not kave any tGLM and tETH yet on your Jupyter on Golem wallet, then You need to fund it first. To do that we need to use %fund command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea7c240b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%fund goerli"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c57065cf",
   "metadata": {},
   "source": [
    "Type %status command again to verify that your funds have arrived."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cda5f1b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "%status"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a46ed75",
   "metadata": {},
   "source": [
    "Looks like we have some tGLM and tETH from Golem Faucet. We are ready to start some real fun with Jupyter on Golem!\n",
    "\n",
    "Are You afraid that your tokens will be stuck on Jupyter on Golem wallet after some playtime? Do not worry! Your tokens won't be stuck. If You decide to take them back you can always export private key from YAGNA CLI. More information can be found here: https://handbook.golem.network/payments/using-golem-on-mainnet/backing-up-your-golem-wallet."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46821685",
   "metadata": {},
   "source": [
    "OK, now is the time to allocate some GLM tokens for future computations. Let's start with 2 GLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bad323ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "%budget goerli 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f6ccb00",
   "metadata": {},
   "source": [
    "With Allocation done, let's look for some provider. Nothing to fancy... 4GB of RAM, 10GB of disk space and 2 Cores will be more than enough for this example. By default %connect command is set with 10 minute timeout. If You won't be able to connect within this time just try again or try again with higher timeout (e.g. timeout=15m). When your provider is finally up and running You will see \"Ready.\" message."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b4c749d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%connect mem>4 disk>10 cores>2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "112266fb",
   "metadata": {},
   "source": [
    "Now we should be at the Provider's Host. Let's verify that with some commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d5a98c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14b1a836",
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dddf72a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "%status"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8d90378",
   "metadata": {},
   "source": [
    "Yep! We definitely are not in Kansas anymore :)\n",
    "\n",
    "Let's upload some Data Set into the Provider to make some computations out of it. Just remember that You need to have Your Data Set stored locally first. \n",
    "You can get Data Set for this example (california_housing_train.csv) from here: https://github.com/golemfactory/golem-kernel-python/blob/master/examples/california_housing_train.csv. \n",
    "Just be sure that You see that file on your left JupyterLab panel!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5db278e",
   "metadata": {},
   "outputs": [],
   "source": [
    "%upload california_housing_train.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d439dbb",
   "metadata": {},
   "source": [
    "Great! Our Data Set should be on the Provider's host. We need to remember that %upload command will put files into ./workdir folder. Let's go there and check that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60a8517f",
   "metadata": {},
   "outputs": [],
   "source": [
    "cd workdir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "972ce7ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -l"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2192418",
   "metadata": {},
   "source": [
    "You should be able to see california_housing_train.csv file in the workdir folder. Let's go back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87ebae5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "cd .."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87461559",
   "metadata": {},
   "source": [
    "It is time to start our computations. Let's import some pre-installed modules fist. It might take a moment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8b10a68",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "442012ce",
   "metadata": {},
   "source": [
    "Time to start some Data Analytic magic!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6eddbfc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The following lines adjust the granularity of reporting. \n",
    "pd.options.display.max_rows = 10\n",
    "pd.options.display.float_format = \"{:.1f}\".format\n",
    "training_df = pd.read_csv(filepath_or_buffer=\"workdir/california_housing_train.csv\")\n",
    "# Scale the label.\n",
    "training_df[\"median_house_value\"] /= 1000.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bd0a904",
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model(my_learning_rate):\n",
    "  \"\"\"Create and compile a simple linear regression model.\"\"\"\n",
    "  # Most simple tf.keras models are sequential.\n",
    "  model = tf.keras.models.Sequential()\n",
    "\n",
    "  # Describe the topography of the model.\n",
    "  # The topography of a simple linear regression model\n",
    "  # is a single node in a single layer.\n",
    "  model.add(tf.keras.layers.Dense(units=1, \n",
    "                                  input_shape=(1,)))\n",
    "\n",
    "  # Compile the model topography into code that TensorFlow can efficiently\n",
    "  # execute. Configure training to minimize the model's mean squared error. \n",
    "  model.compile(optimizer=tf.keras.optimizers.experimental.RMSprop(learning_rate=my_learning_rate),\n",
    "                loss=\"mean_squared_error\",\n",
    "                metrics=[tf.keras.metrics.RootMeanSquaredError()])\n",
    "\n",
    "  return model        \n",
    "\n",
    "\n",
    "def train_model(model, df, feature, label, epochs, batch_size):\n",
    "  \"\"\"Train the model by feeding it data.\"\"\"\n",
    "\n",
    "  # Feed the model the feature and the label.\n",
    "  # The model will train for the specified number of epochs. \n",
    "  history = model.fit(x=df[feature],\n",
    "                      y=df[label],\n",
    "                      batch_size=batch_size,\n",
    "                      epochs=epochs)\n",
    "\n",
    "  # Gather the trained model's weight and bias.\n",
    "  trained_weight = model.get_weights()[0][0][0]\n",
    "  trained_bias = model.get_weights()[1][0]\n",
    "\n",
    "  # The list of epochs is stored separately from the rest of history.\n",
    "  epochs = history.epoch\n",
    "  \n",
    "  # Isolate the error for each epoch.\n",
    "  hist = pd.DataFrame(history.history)\n",
    "\n",
    "  # To track the progression of training, we're going to take a snapshot\n",
    "  # of the model's root mean squared error at each epoch. \n",
    "  rmse = hist[\"root_mean_squared_error\"]\n",
    "\n",
    "  return trained_weight, trained_bias, epochs, rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56403133",
   "metadata": {},
   "source": [
    "Below functions will plot our results and save them as .png files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcc458e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_and_save_the_model(trained_weight, trained_bias, feature, label):\n",
    "  \"\"\"Plot the trained model against 200 random training examples.\"\"\"\n",
    "\n",
    "  # Label the axes.\n",
    "  plt.xlabel(feature)\n",
    "  plt.ylabel(label)\n",
    "\n",
    "  # Create a scatter plot from 200 random points of the dataset.\n",
    "  random_examples = training_df.sample(n=200)\n",
    "  plt.scatter(random_examples[feature], random_examples[label])\n",
    "\n",
    "  # Create a red line representing the model. The red line starts\n",
    "  # at coordinates (x0, y0) and ends at coordinates (x1, y1).\n",
    "  x0 = 0\n",
    "  y0 = trained_bias\n",
    "  x1 = random_examples[feature].max()\n",
    "  y1 = trained_bias + (trained_weight * x1)  \n",
    "  plt.plot([x0, x1], [y0, y1], c='r')\n",
    "\n",
    "  # Render the scatter plot and the red line.\n",
    "  plt.savefig('workdir/model_results.png')\n",
    "  plt.show()\n",
    "\n",
    "\n",
    "def plot_and_save_the_loss_curve(epochs, rmse):\n",
    "  \"\"\"Plot a curve of loss vs. epoch.\"\"\"\n",
    "\n",
    "  plt.figure()\n",
    "  plt.xlabel(\"Epoch\")\n",
    "  plt.ylabel(\"Root Mean Squared Error\")\n",
    "\n",
    "  plt.plot(epochs, rmse, label=\"Loss\")\n",
    "  plt.legend()\n",
    "  plt.ylim([rmse.min()*0.97, rmse.max()])\n",
    "  plt.savefig('workdir/loss_results.png')\n",
    "  plt.show()  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1716476d",
   "metadata": {},
   "source": [
    "And now let's ask our provider to do some \"heavy\" lifting for us :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e1d8c25",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The following variables are the hyperparameters.\n",
    "learning_rate = 0.01\n",
    "epochs = 30\n",
    "batch_size = 30\n",
    "\n",
    "# Specify the feature and the label.\n",
    "my_feature = \"total_rooms\"  # the total number of rooms on a specific city block.\n",
    "my_label=\"median_house_value\" # the median value of a house on a specific city block.\n",
    "# That is, you're going to create a model that predicts house value based \n",
    "# solely on total_rooms.  \n",
    "\n",
    "# Discard any pre-existing version of the model.\n",
    "my_model = None\n",
    "\n",
    "# Invoke the functions.\n",
    "my_model = build_model(learning_rate)\n",
    "weight, bias, epochs, rmse = train_model(my_model, training_df, \n",
    "                                         my_feature, my_label,\n",
    "                                         epochs, batch_size)\n",
    "\n",
    "print(\"\\nThe learned weight for your model is %.4f\" % weight)\n",
    "print(\"The learned bias for your model is %.4f\\n\" % bias )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13052b4e",
   "metadata": {},
   "source": [
    "It is time to plot our results to see how they look like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2d5889c",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_and_save_the_model(weight, bias, my_feature, my_label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a62091ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_and_save_the_loss_curve(epochs, rmse)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f927e9ef",
   "metadata": {},
   "source": [
    "Looks nice. Let's download them on our local machine so that we won't lose them!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c670edef",
   "metadata": {},
   "outputs": [],
   "source": [
    "%download model_results.png"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9025441a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%download loss_results.png"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff9c9f52",
   "metadata": {},
   "source": [
    "Ok. That was cool but what if I want to use some other PIP modules, which are not pre-installed?\n",
    "\n",
    "That is simple. Just use %pip install command. Let's use that occasion to add some color to this terminal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e56e895",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install colorama"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66c25723",
   "metadata": {},
   "outputs": [],
   "source": [
    "from colorama import Fore\n",
    "print(Fore.BLUE + 'Jupyter on Golem is great!')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce5cfb36",
   "metadata": {},
   "source": [
    "We are getting close to the end of this example. It is high time to disconnect from our provider and initiate the payment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7602c75",
   "metadata": {},
   "outputs": [],
   "source": [
    "%disconnect"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3b00568",
   "metadata": {},
   "source": [
    "And that is it! We hope You liked Jupyter on Golem. Please consider to give us a feedback."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Golem",
   "language": "python",
   "name": "golem"
  },
  "language_info": {
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}