notebooks/chapter04_book.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural network for regression and classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\newcommand{\\bw}{w}$\n",
    "$\\newcommand{\\bW}{W}$\n",
    "$\\newcommand{\\bhW}{\\hat{W}}$\n",
    "$\\newcommand{\\bH}{H}$\n",
    "$\\newcommand{\\bX}{X}$\n",
    "$\\newcommand{\\by}{y}$\n",
    "$\\newcommand{\\bh}{h}$\n",
    "$\\newcommand{\\bp}{p}$\n",
    "$\\newcommand{\\hy}{\\hat{y}}$\n",
    "$\\newcommand{\\bhy}{\\hat{y}}$\n",
    "$\\newcommand{\\bx}{x}$\n",
    "$\\newcommand{\\bbeta}{\\beta}$\n",
    "$\\newcommand{\\bepsilon}{\\epsilon}$\n",
    "$\\newcommand{\\bhbeta}{\\hat{\\beta}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Synthetic dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The work directory is given from the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def towdir(s):\n",
    "    return (str('./datasets_book/'+s))\n",
    "\n",
    "import deepglmlib.utils as utils\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import importlib\n",
    "importlib.reload(utils)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "n1 = n2 = 500\n",
    "n = 2 * n1\n",
    "beta = np.array([-0.5,3.5,2.0]).reshape((3,1))\n",
    "\n",
    "x1 = np.random.uniform(-2,2,n*100).reshape((n1*100,2))\n",
    "x2 = np.random.uniform(-2,2,n*100).reshape((n1*100,2))\n",
    "\n",
    "x1 = x1[np.sqrt(x1[:,0]**2+x1[:,1]**2)<0.44,:]\n",
    "x2 = x2[np.sqrt(x2[:,0]**2+x2[:,1]**2)>0.46,:]\n",
    "\n",
    "n1 = n2 = 50\n",
    "n = 2 * n1\n",
    "\n",
    "x = np.vstack([ x1[0:n1,:], x2[0:n2,:] ]) + \\\n",
    "    np.random.normal(0,0.01,n*2).reshape((n,2))\n",
    "y = np.vstack([ np.zeros((n1,1)), np.ones((n2,1)) ])\n",
    "\n",
    "X = np.hstack([ np.ones((len(x),1)), x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.savetxt(towdir(\"./xy_2d_diskandnoise_reglogistic.txt\"),np.hstack([x,y]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "xy = np.loadtxt(towdir(\"./xy_2d_diskandnoise_reglogistic.txt\"))\n",
    "x  = xy[:,[0,1]]\n",
    "y  = xy[:,2].reshape((xy.shape[0],1))\n",
    "X  = np.hstack([ np.ones((len(x),1)), x])\n",
    "n = len(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let shuffle the rows for avoiding any structure before mini-batches are cycled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ids_random = np.random.permutation(len(y))\n",
    "np.take(x,ids_random,axis=0,out=x)\n",
    "np.take(y,ids_random,axis=0,out=y)\n",
    "np.take(X,ids_random,axis=0,out=X)\n",
    "\n",
    "x.shape, y.shape, X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y  = y.ravel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import deepglmlib.utils as utils\n",
    "\n",
    "# true frontier from data generation\n",
    "theta = np.linspace(0, 2*np.pi, 20)\n",
    "x1_circle = 0.45*np.cos(theta)\n",
    "x2_circle = 0.45*np.sin(theta)\n",
    "\n",
    "fig, (ax1) = plt.subplots(1, 1, figsize=(5,5))\n",
    "\n",
    "utils.f_vizu2d_beta(ax1,x[y.ravel()==0,0],x[y.ravel()==1,0],\n",
    "                    x[y.ravel()==0,1],x[y.ravel()==1,1], [], [],\n",
    "                    xlim=[min(x[:,0]),max(x[:,0])],ylim=[min(x[:,1]),max(x[:,1])], \n",
    "                    samplename=\"Whole sample\")\n",
    "ax1_ = ax1.plot(x1_circle,x2_circle,color='m',label=\"true frontier\")\n",
    "plt.legend(fancybox=True, framealpha=0.2, loc=\"lower left\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a neural network with an hidden layer for nonlinear logistic model \n",
    "## with pytorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The subsamples are drawn randomly without replacement: the test set is found first via sampling because smaller and then substracted from the whole sample to get the train set. This is equivalent to a function in **sklearn** with more available options, but writen in two rows here. Note this is with suffle as the order of the indexes is not kept. The indexes are the working backbone of the approach in large dataset because this is not possible to replicate or load the full dataset in the computer memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ids_train, ids_test, ids_all = utils.f_splitIndex(n)\n",
    "\n",
    "print(len(ids_all), len(ids_train), len(ids_test))\n",
    "print(set(ids_all)-set(ids_test)-set(ids_train)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader, TensorDataset\n",
    "dataset = TensorDataset( torch.Tensor(x), torch.Tensor(y) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl_train, dl_test, n, n_train, n_test = utils.f_splitDataset(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n, n_train, n_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Definition of the neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class GNLMRegression(nn.Module):\n",
    "    def __init__(self, name, layers):\n",
    "        super().__init__()\n",
    "        self.name = name\n",
    "        self.layers = layers\n",
    "        self.net = nn.Sequential(*layers)\n",
    "    def forward(self, x):\n",
    "        return self.net(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nb_nodes_in  = 2\n",
    "nb_nodes_out = 1\n",
    "nb_nodes_hid1 = 10\n",
    "\n",
    "layers = []\n",
    "layers.append(nn.Linear(nb_nodes_in,nb_nodes_hid1, bias=True))\n",
    "layers.append(nn.Tanh())\n",
    "layers.append(nn.Linear(nb_nodes_hid1, nb_nodes_out, bias=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Definition of the required object befores training the nn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def transform_yb(yb,name_model,yhatb=None,device=None):\n",
    "    return yb.ravel()\n",
    "\n",
    "def transform_yhatb(yhatb,name_model):\n",
    "    return yhatb.ravel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It has been added in the class, the object net which is required for the function forward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#not used after (included in the class)\n",
    "net = nn.Sequential(*layers)\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import copy\n",
    "model =  GNLMRegression(\"LMLP\",copy.deepcopy(layers))\n",
    "\n",
    "nbmax_epoqs=6000\n",
    "alpha_t= 1e-3\n",
    "debug_out=100\n",
    "   \n",
    "loss      = torch.nn.BCEWithLogitsLoss(reduction='sum')\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=alpha_t, momentum=0.0)\n",
    "monitor   = utils.MyMonitorTest(model,loss,dl_train,dl_test,nbmax_epoqs,debug_out)\n",
    "\n",
    "loss_train_s,tmax,monistopc  = utils.f_train_glmr(dl_train,model,optimizer,loss,monitor,\n",
    "                                                device=None,printed=2, \n",
    "                                                loss_yy_model = None,\n",
    "                                                transform_Xb=None,\n",
    "                                                transform_yb=transform_yb,\n",
    "                                                transform_yhatb=transform_yhatb,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "fig, ax = plt.subplots()\n",
    "plt = utils.f_draw_s([ range(len(loss_train_s)), monitor.step_test_s[monitor.loss_test_s>0].astype(int) ],\n",
    "               [ loss_train_s/n_train,monitor.loss_test_s[monitor.loss_test_s>0]/n_test],\n",
    "               [\"b-\", \"r-\"] ,\"t\",[ \"loss train\", \"loss test\"], \" \",ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(model,towdir(\"deepmodel_diskandnoise.pth\"))\n",
    "torch.save(model.state_dict(),towdir(\"deepmodelw_diskandnoise.pth\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "acc_train, yhat_train, y_train = utils.f_test_glmr(model,dl_train,True)\n",
    "acc_test, yhat_test, y_test = utils.f_test_glmr(model,dl_test, True)\n",
    "\n",
    "print(\"acc_train=\",utils.nprd(acc_train,4), \" acc_test=\",utils.nprd(acc_test,4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that there is also available the class **ModuleList** for dealing with list of laters, but this was not considered herein."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import deepglmlib.utils as utils\n",
    "import matplotlib.pyplot as plt\n",
    "fig, ax = plt.subplots()\n",
    "utils.f_plot_2d_boudary_MLP(ax,model,x,y,300)\n",
    "ax.plot(x1_circle,x2_circle,color='m',label=\"true frontier\")\n",
    "ax.legend(fancybox=True, framealpha=0.2, loc=\"lower left\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "myenv",
   "language": "python",
   "name": "myenv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}