From 71f36fbe79c03ab120cfb7d3ee5f015bfe5b6be5 Mon Sep 17 00:00:00 2001 From: Moistsoy <89026391+Moistsoy@users.noreply.github.com> Date: Sun, 3 Oct 2021 00:38:04 +0900 Subject: [PATCH 1/7] Update COSE474-03.md --- COSE474-03.md | 1 + 1 file changed, 1 insertion(+) diff --git a/COSE474-03.md b/COSE474-03.md index c8f3bdc..0386823 100644 --- a/COSE474-03.md +++ b/COSE474-03.md @@ -20,6 +20,7 @@ | 16 | Pilsang Kim | STAT | Hello World! | | 17 | Inyup Lee | CS | Hi, everyone | | 18 | Jaehu Lee | CS | Hello World! | +| 19 | Jiseok Ryu | Psychology | Hello World! | 2020 Fall == From e4b60395e5a3262520529b7a991394ecd0bdfc89 Mon Sep 17 00:00:00 2001 From: Moistsoy <89026391+Moistsoy@users.noreply.github.com> Date: Sun, 3 Oct 2021 03:10:15 +0900 Subject: [PATCH 2/7] Created using Colaboratory --- .../Data_Manipulation.ipynb | 2197 ++++++++++------- 1 file changed, 1275 insertions(+), 922 deletions(-) diff --git a/Ch04_The_Preliminaries_A_Crashcourse/Data_Manipulation.ipynb b/Ch04_The_Preliminaries_A_Crashcourse/Data_Manipulation.ipynb index 50d5d0d..6e963ba 100644 --- a/Ch04_The_Preliminaries_A_Crashcourse/Data_Manipulation.ipynb +++ b/Ch04_The_Preliminaries_A_Crashcourse/Data_Manipulation.ipynb @@ -1,926 +1,1279 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Data Manipulation\n", - "\n", - "It is impossible to get anything done if we cannot manipulate data. Generally, there are two important things we need to do with data: (i) acquire it and (ii) process it once it is inside the computer. There is no point in acquiring data if we do not even know how to store it, so let's get our hands dirty first by playing with synthetic data. We will start by introducing the tensor,\n", - "PyTorch's primary tool for storing and transforming data. If you have worked with NumPy before, you will notice that tensors are, by design, similar to NumPy's multi-dimensional array. Tensors support asynchronous computation on CPU, GPU and provide support for automatic differentiation." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Getting Started" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "import torch" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Tensors represent (possibly multi-dimensional) arrays of numerical values.\n", - "The simplest object we can create is a vector. To start, we can use `arange` to create a row vector with 12 consecutive integers." - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11.],\n", - " dtype=torch.float64)" - ] - }, - "execution_count": 2, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "x = torch.arange(12, dtype=torch.float64)\n", - "x" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "torch.Size([12])" - ] - }, - "execution_count": 3, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# We can get the tensor shape through the shape attribute.\n", - "x.shape" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": { - "scrolled": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "torch.Size([12])" - ] - }, - "execution_count": 4, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# .shape is an alias for .size(), and was added to more closely match numpy\n", - "x.size()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We use the `reshape` function to change the shape of one (possibly multi-dimensional) array, to another that contains the same number of elements.\n", - "For example, we can transform the shape of our line vector `x` to (3, 4), which contains the same values but interprets them as a matrix containing 3 rows and 4 columns. Note that although the shape has changed, the elements in `x` have not." - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([[ 0., 1., 2., 3.],\n", - " [ 4., 5., 6., 7.],\n", - " [ 8., 9., 10., 11.]], dtype=torch.float64)" - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "x = x.reshape((3, 4))\n", - "x" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Reshaping by manually specifying each of the dimensions can get annoying. Once we know one of the dimensions, why should we have to perform the division our selves to determine the other? For example, above, to get a matrix with 3 rows, we had to specify that it should have 4 columns (to account for the 12 elements). Fortunately, PyTorch can automatically work out one dimension given the other.\n", - "We can invoke this capability by placing `-1` for the dimension that we would like PyTorch to automatically infer. In our case, instead of\n", - "`x.reshape((3, 4))`, we could have equivalently used `x.reshape((-1, 4))` or `x.reshape((3, -1))`." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([[1.3733e-14, 6.4069e+02, 4.3066e+21],\n", - " [1.1824e+22, 4.3066e+21, 6.3828e+28]])" - ] - }, - "execution_count": 6, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "torch.FloatTensor(2, 3)" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([[ 0.0000e+00, -2.0000e+00, 0.0000e+00],\n", - " [-2.0000e+00, 4.2039e-45, 0.0000e+00]])" - ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "torch.Tensor(2, 3)" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([[ 0.0000e+00, -2.0000e+00, 4.3892e-11],\n", - " [-1.5849e+29, 2.8026e-45, 0.0000e+00]])" - ] - }, - "execution_count": 8, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "torch.empty(2, 3)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "torch.Tensor() is just an alias to torch.FloatTensor() which is the default type of tensor, when no dtype is specified during tensor construction.\n", - "\n", - "From the torch for numpy users notes, it seems that torch.Tensor() is a drop-in replacement of numpy.empty()\n", - "\n", - "So, in essence torch.FloatTensor() and torch.empty() does the same job.\n", - "\n", - "The `empty` method just grabs some memory and hands us back a matrix without setting the values of any of its entries. This is very efficient but it means that the entries might take any arbitrary values, including very big ones! Typically, we'll want our matrices initialized either with ones, zeros, some known constant or numbers randomly sampled from a known distribution.\n", - "\n", - "Perhaps most often, we want an array of all zeros. To create tensor with all elements set to 0 and a shape of (2, 3, 4) we can invoke:" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([[[0., 0., 0., 0.],\n", - " [0., 0., 0., 0.],\n", - " [0., 0., 0., 0.]],\n", - "\n", - " [[0., 0., 0., 0.],\n", - " [0., 0., 0., 0.],\n", - " [0., 0., 0., 0.]]])" - ] - }, - "execution_count": 9, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "torch.zeros((2, 3, 4))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can create tensors with each element set to 1 works via" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([[[1., 1., 1., 1.],\n", - " [1., 1., 1., 1.],\n", - " [1., 1., 1., 1.]],\n", - "\n", - " [[1., 1., 1., 1.],\n", - " [1., 1., 1., 1.],\n", - " [1., 1., 1., 1.]]])" - ] - }, - "execution_count": 10, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "torch.ones((2, 3, 4))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can also specify the value of each element in the desired NDArray by supplying a Python list containing the numerical values." - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([[2, 1, 4, 3],\n", - " [1, 2, 3, 4],\n", - " [4, 3, 2, 1]])" - ] - }, - "execution_count": 11, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "y = torch.tensor([[2, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]])\n", - "y" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In some cases, we will want to randomly sample the values of each element in the tensor according to some known probability distribution. This is especially common when we intend to use the tensor as a parameter in a neural network. The following snippet creates an tensor with a shape of (3,4). Each of its elements is randomly sampled in a normal distribution with zero mean and unit variance." - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([[-1.8349, 0.8108, -0.5054, -0.5164],\n", - " [ 0.2499, 1.5147, -0.9165, -0.4967],\n", - " [-0.3341, 0.2574, 1.2594, -0.2960]])" - ] - }, - "execution_count": 12, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "torch.randn(3, 4)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Operations\n", - "\n", - "Oftentimes, we want to apply functions to arrays. Some of the simplest and most useful functions are the element-wise functions. These operate by performing a single scalar operation on the corresponding elements of two arrays. We can create an element-wise function from any function that maps from the scalars to the scalars. In math notations we would denote such a function as $f: \\mathbb{R} \\rightarrow \\mathbb{R}$. Given any two vectors $\\mathbf{u}$ and $\\mathbf{v}$ *of the same shape*, and the function f,\n", - "we can produce a vector $\\mathbf{c} = F(\\mathbf{u},\\mathbf{v})$ by setting $c_i \\gets f(u_i, v_i)$ for all $i$. Here, we produced the vector-valued $F: \\mathbb{R}^d \\rightarrow \\mathbb{R}^d$ by *lifting* the scalar function to an element-wise vector operation. In PyTorch, the common standard arithmetic operators (+,-,/,\\*,\\*\\*) have all been *lifted* to element-wise operations for identically-shaped tensors of arbitrary shape. We can call element-wise operations on any two tensors of the same shape, including matrices." - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "x = tensor([1., 2., 4., 8.])\n", - "x + y tensor([ 3., 4., 6., 10.])\n", - "x - y tensor([-1., 0., 2., 6.])\n", - "x * y tensor([ 2., 4., 8., 16.])\n", - "x / y tensor([0.5000, 1.0000, 2.0000, 4.0000])\n" - ] - } - ], - "source": [ - "x = torch.tensor([1, 2, 4, 8], dtype=torch.float32)\n", - "y = torch.ones_like(x) * 2\n", - "print('x =', x)\n", - "print('x + y', x + y)\n", - "print('x - y', x - y)\n", - "print('x * y', x * y)\n", - "print('x / y', x / y)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Many more operations can be applied element-wise, such as exponentiation:" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([2.7183e+00, 7.3891e+00, 5.4598e+01, 2.9810e+03])" - ] - }, - "execution_count": 14, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "torch.exp(x)\n", - "# Note: torch.exp is not implemented for 'torch.LongTensor'." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In addition to computations by element, we can also perform matrix operations, like matrix multiplication using the `mm` or `matmul` function. Next, we will perform matrix multiplication of `x` and the transpose of `y`. We define `x` as a matrix of 3 rows and 4 columns, and `y` is transposed into a matrix of 4 rows and 3 columns. The two matrices are multiplied to obtain a matrix of 3 rows and 3 columns." - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "torch.float32\n", - "tensor([[2., 1., 4., 3.],\n", - " [1., 2., 3., 4.],\n", - " [4., 3., 2., 1.]])\n" - ] - }, - { - "data": { - "text/plain": [ - "tensor([[ 18., 20., 10.],\n", - " [ 58., 60., 50.],\n", - " [ 98., 100., 90.]])" - ] - }, - "execution_count": 15, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "x = torch.arange(12, dtype=torch.float32).reshape((3,4))\n", - "y = torch.tensor([[2, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]], dtype=torch.float32)\n", - "print(x.dtype)\n", - "print(y)\n", - "torch.mm(x, y.t())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Note that torch.dot() behaves differently to np.dot(). There's been some discussion about what would be desirable here. Specifically, torch.dot() treats both a and b as 1D vectors (irrespective of their original shape) and computes their inner product. " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can also merge multiple tensors. For that, we need to tell the system along which dimension to merge. The example below merges two matrices along dimension 0 (along rows) and dimension 1 (along columns) respectively." - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([[ 0., 1., 2., 3.],\n", - " [ 4., 5., 6., 7.],\n", - " [ 8., 9., 10., 11.],\n", - " [ 2., 1., 4., 3.],\n", - " [ 1., 2., 3., 4.],\n", - " [ 4., 3., 2., 1.]])" - ] - }, - "execution_count": 16, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "torch.cat((x, y), dim=0)" - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([[ 0., 1., 2., 3., 2., 1., 4., 3.],\n", - " [ 4., 5., 6., 7., 1., 2., 3., 4.],\n", - " [ 8., 9., 10., 11., 4., 3., 2., 1.]])" - ] - }, - "execution_count": 17, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "torch.cat((x, y), dim=1)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Sometimes, we may want to construct binary tensors via logical statements. Take `x == y` as an example. If `x` and `y` are equal for some entry, the new tensor has a value of 1 at the same position; otherwise it is 0." - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([[0, 1, 0, 1],\n", - " [0, 0, 0, 0],\n", - " [0, 0, 0, 0]], dtype=torch.uint8)" - ] - }, - "execution_count": 18, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "x == y" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Summing all the elements in the tensor yields an tensor with only one element." - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor(66.)" - ] - }, - "execution_count": 19, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "x.sum()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can transform the result into a scalar in Python using the `asscalar` function of `numpy`(from numpy v1.16 onwards it issues a deprication warning. We can ignore that at this moment). In the following example, the $\\ell_2$ norm of `x` yields a single element tensor. The final result is transformed into a scalar." - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "22.494443893432617" - ] - }, - "execution_count": 20, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import numpy as np\n", - "np.asscalar(x.norm())" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Broadcast Mechanism\n", - "\n", - "In the above section, we saw how to perform operations on two tensors of the same shape. When their shapes differ, a broadcasting mechanism may be triggered analogous to NumPy: first, copy the elements appropriately so that the two tensors have the same shape, and then carry out operations by element." - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "(tensor([[0.],\n", - " [1.],\n", - " [2.]]), tensor([[0., 1.]]))" - ] - }, - "execution_count": 21, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "a = torch.arange(3, dtype=torch.float).reshape((3, 1))\n", - "b = torch.arange(2, dtype=torch.float).reshape((1, 2))\n", - "a, b" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Since `a` and `b` are (3x1) and (1x2) matrices respectively, their shapes do not match up if we want to add them. PyTorch addresses this by 'broadcasting' the entries of both matrices into a larger (3x2) matrix as follows: for matrix `a` it replicates the columns, for matrix `b` it replicates the rows before adding up both element-wise." - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([[0., 1.],\n", - " [1., 2.],\n", - " [2., 3.]])" - ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "a + b" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Indexing and Slicing\n", - "\n", - "Just like in any other Python array, elements in a tensor can be accessed by its index. In good Python tradition the first element has index 0 and ranges are specified to include the first but not the last element. By this logic `1:3` selects the second and third element. Let's try this out by selecting the respective rows in a matrix." - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([[ 4., 5., 6., 7.],\n", - " [ 8., 9., 10., 11.]])" - ] - }, - "execution_count": 23, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "x[1:3]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Beyond reading, we can also write elements of a matrix." - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([[ 0., 1., 2., 3.],\n", - " [ 4., 5., 9., 7.],\n", - " [ 8., 9., 10., 11.]])" - ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "x[1, 2] = 9\n", - "x" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If we want to assign multiple elements the same value, we simply index all of them and then assign them the value. For instance, `[0:2, :]` accesses the first and second rows. While we discussed indexing for matrices, this obviously also works for vectors and for tensors of more than 2 dimensions." - ] - }, - { - "cell_type": "code", - "execution_count": 25, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "tensor([[12., 12., 12., 12.],\n", - " [12., 12., 12., 12.],\n", - " [ 8., 9., 10., 11.]])" - ] - }, - "execution_count": 25, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "x[0:2, :] = 12\n", - "x" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Saving Memory\n", - "\n", - "In the previous example, every time we ran an operation, we allocated new memory to host its results. For example, if we write `y = x + y`, we will dereference the matrix that `y` used to point to and instead point it at the newly allocated memory. In the following example we demonstrate this with Python's `id()` function, which gives us the exact address of the referenced object in memory. After running `y = y + x`, we will find that `id(y)` points to a different location. That is because Python first evaluates `y + x`, allocating new memory for the result and then subsequently redirects `y` to point at this new location in memory." - ] - }, - { - "cell_type": "code", - "execution_count": 26, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "False" - ] - }, - "execution_count": 26, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "before = id(y)\n", - "y = y + x\n", - "id(y) == before" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This might be undesirable for two reasons. First, we do not want to run around allocating memory unnecessarily all the time. In machine learning, we might have hundreds of megabytes of parameters and update all of them multiple times per second. Typically, we will want to perform these updates *in place*. Second, we might point at the same parameters from multiple variables. If we do not update in place, this could cause a memory leak, making it possible for us to inadvertently reference stale parameters.\n", - "\n", - "Fortunately, performing in-place operations in PyTorch is easy. We can assign the result of an operation to a previously allocated array with slice notation, e.g., `y[:] = `. To illustrate the behavior, we first clone the shape of a matrix using `zeros_like` to allocate a block of 0 entries." - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "id(z): 4548380712\n", - "id(z): 4548380712\n" - ] - } - ], - "source": [ - "z = torch.zeros_like(y)\n", - "print('id(z):', id(z))\n", - "z[:] = x + y\n", - "print('id(z):', id(z))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "While this looks pretty, `x+y` here will still allocate a temporary buffer to store the result of `x+y` before copying it to `z[:]`. To make even better use of memory, we can directly invoke the underlying `tensor` operation, in this case `add`, avoiding temporary buffers. We do this by specifying the `out` keyword argument, which every `tensor` operator supports:" - ] - }, - { - "cell_type": "code", - "execution_count": 28, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "True" - ] - }, - "execution_count": 28, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "before = id(z)\n", - "torch.add(x, y, out=z)\n", - "id(z) == before" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If the value of `x ` is not reused in subsequent computations, we can also use `x[:] = x + y` or `x += y` to reduce the memory overhead of the operation." - ] - }, - { - "cell_type": "code", - "execution_count": 29, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "True" - ] - }, - "execution_count": 29, - "metadata": {}, - "output_type": "execute_result" + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.8" + }, + "colab": { + "name": "Data_Manipulation.ipynb", + "provenance": [] } - ], - "source": [ - "before = id(x)\n", - "x += y\n", - "id(x) == before" - ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Mutual Transformation of PyTorch and NumPy\n", - "\n", - "Converting PyTorch Tensors to and from NumPy Arrays is easy. The converted arrays do *not* share memory. This minor inconvenience is actually quite important: when you perform operations on the CPU or one of the GPUs, you do not want PyTorch having to wait whether NumPy might want to be doing something else with the same chunk of memory. `.tensor` and `.numpy` do the trick." - ] - }, - { - "cell_type": "code", - "execution_count": 30, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "\n" - ] + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "3kFvwsOwtt8u" + }, + "source": [ + "# Data Manipulation\n", + "\n", + "It is impossible to get anything done if we cannot manipulate data. Generally, there are two important things we need to do with data: (i) acquire it and (ii) process it once it is inside the computer. There is no point in acquiring data if we do not even know how to store it, so let's get our hands dirty first by playing with synthetic data. We will start by introducing the tensor,\n", + "PyTorch's primary tool for storing and transforming data. If you have worked with NumPy before, you will notice that tensors are, by design, similar to NumPy's multi-dimensional array. Tensors support asynchronous computation on CPU, GPU and provide support for automatic differentiation." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "eyta6mOAtt80" + }, + "source": [ + "## Getting Started" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "Rn9hPIxAtt81" + }, + "source": [ + "import torch" + ], + "execution_count": 1, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "miAEYaQjtt83" + }, + "source": [ + "Tensors represent (possibly multi-dimensional) arrays of numerical values.\n", + "The simplest object we can create is a vector. To start, we can use `arange` to create a row vector with 12 consecutive integers." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "f4RY3QwHtt83", + "outputId": "2580df63-9104-42ea-cf25-3a8db4ab0086", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "x = torch.arange(12, dtype=torch.float64)\n", + "x" + ], + "execution_count": 2, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11.],\n", + " dtype=torch.float64)" + ] + }, + "metadata": {}, + "execution_count": 2 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "g8IgkvQwtt85", + "outputId": "27080c3c-b0fc-4acd-ab65-18dc1f47591e", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "# We can get the tensor shape through the shape attribute.\n", + "x.shape" + ], + "execution_count": 3, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "torch.Size([12])" + ] + }, + "metadata": {}, + "execution_count": 3 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "scrolled": true, + "id": "vYaIGyCott85", + "outputId": "ca79721d-90a9-45be-ed65-a513c5e3f9d6", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "# .shape is an alias for .size(), and was added to more closely match numpy\n", + "x.size()" + ], + "execution_count": 4, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "torch.Size([12])" + ] + }, + "metadata": {}, + "execution_count": 4 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c-t9-Jqctt87" + }, + "source": [ + "We use the `reshape` function to change the shape of one (possibly multi-dimensional) array, to another that contains the same number of elements.\n", + "For example, we can transform the shape of our line vector `x` to (3, 4), which contains the same values but interprets them as a matrix containing 3 rows and 4 columns. Note that although the shape has changed, the elements in `x` have not." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "Hi1ZVTY8tt88", + "outputId": "8a4089e2-b704-401c-b8b7-80a145b88fb0", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "x = x.reshape((3, 4))\n", + "x" + ], + "execution_count": 5, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[ 0., 1., 2., 3.],\n", + " [ 4., 5., 6., 7.],\n", + " [ 8., 9., 10., 11.]], dtype=torch.float64)" + ] + }, + "metadata": {}, + "execution_count": 5 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8Js7bMI9tt89" + }, + "source": [ + "Reshaping by manually specifying each of the dimensions can get annoying. Once we know one of the dimensions, why should we have to perform the division our selves to determine the other? For example, above, to get a matrix with 3 rows, we had to specify that it should have 4 columns (to account for the 12 elements). Fortunately, PyTorch can automatically work out one dimension given the other.\n", + "We can invoke this capability by placing `-1` for the dimension that we would like PyTorch to automatically infer. In our case, instead of\n", + "`x.reshape((3, 4))`, we could have equivalently used `x.reshape((-1, 4))` or `x.reshape((3, -1))`." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "c4kG5w9Ftt89", + "outputId": "e18b1e57-04d6-4a70-db12-dcc13497df30", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "torch.FloatTensor(2, 3)" + ], + "execution_count": 6, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[2.6519e-11, 3.0704e-41, 3.7835e-44],\n", + " [0.0000e+00, nan, 0.0000e+00]])" + ] + }, + "metadata": {}, + "execution_count": 6 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "lDMwcl0Qtt8-", + "outputId": "833d8dc4-bb96-4ebe-a0bf-8cd220a9299d", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "torch.Tensor(2, 3)" + ], + "execution_count": 7, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[2.6519e-11, 3.0704e-41, 2.3694e-38],\n", + " [0.0000e+00, nan, 0.0000e+00]])" + ] + }, + "metadata": {}, + "execution_count": 7 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "lmpl4ulktt8-", + "outputId": "12858e8d-0fdb-4e1a-b820-ff67dc100c75", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "torch.empty(2, 3)" + ], + "execution_count": 8, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[2.6519e-11, 3.0704e-41, 3.3631e-44],\n", + " [0.0000e+00, nan, 0.0000e+00]])" + ] + }, + "metadata": {}, + "execution_count": 8 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-3ZlDI53tt8-" + }, + "source": [ + "torch.Tensor() is just an alias to torch.FloatTensor() which is the default type of tensor, when no dtype is specified during tensor construction.\n", + "\n", + "From the torch for numpy users notes, it seems that torch.Tensor() is a drop-in replacement of numpy.empty()\n", + "\n", + "So, in essence torch.FloatTensor() and torch.empty() does the same job.\n", + "\n", + "The `empty` method just grabs some memory and hands us back a matrix without setting the values of any of its entries. This is very efficient but it means that the entries might take any arbitrary values, including very big ones! Typically, we'll want our matrices initialized either with ones, zeros, some known constant or numbers randomly sampled from a known distribution.\n", + "\n", + "Perhaps most often, we want an array of all zeros. To create tensor with all elements set to 0 and a shape of (2, 3, 4) we can invoke:" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "qTaM6HfTtt8_", + "outputId": "4ccb042d-c15e-4b69-a005-e42081f76500", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "torch.zeros((2, 3, 4))" + ], + "execution_count": 9, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[[0., 0., 0., 0.],\n", + " [0., 0., 0., 0.],\n", + " [0., 0., 0., 0.]],\n", + "\n", + " [[0., 0., 0., 0.],\n", + " [0., 0., 0., 0.],\n", + " [0., 0., 0., 0.]]])" + ] + }, + "metadata": {}, + "execution_count": 9 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JLohJ6-Ett9A" + }, + "source": [ + "We can create tensors with each element set to 1 works via" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "9U4aMt6ett9A", + "outputId": "1d61889b-129a-4238-c245-40591a562047", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "torch.ones((2, 3, 4))" + ], + "execution_count": 10, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[[1., 1., 1., 1.],\n", + " [1., 1., 1., 1.],\n", + " [1., 1., 1., 1.]],\n", + "\n", + " [[1., 1., 1., 1.],\n", + " [1., 1., 1., 1.],\n", + " [1., 1., 1., 1.]]])" + ] + }, + "metadata": {}, + "execution_count": 10 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2S8gkjsott9B" + }, + "source": [ + "We can also specify the value of each element in the desired NDArray by supplying a Python list containing the numerical values." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "o-trgxDmtt9B", + "outputId": "95fcad52-3395-4a67-9d29-3e6b9fbbd5c3", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "y = torch.tensor([[2, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]])\n", + "y" + ], + "execution_count": 11, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[2, 1, 4, 3],\n", + " [1, 2, 3, 4],\n", + " [4, 3, 2, 1]])" + ] + }, + "metadata": {}, + "execution_count": 11 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DJy6LWKstt9B" + }, + "source": [ + "In some cases, we will want to randomly sample the values of each element in the tensor according to some known probability distribution. This is especially common when we intend to use the tensor as a parameter in a neural network. The following snippet creates an tensor with a shape of (3,4). Each of its elements is randomly sampled in a normal distribution with zero mean and unit variance." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "Y552VuCwtt9B", + "outputId": "1fd2c818-1f00-40f7-b2ba-820f07f4c0ce", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "torch.randn(3, 4)" + ], + "execution_count": 12, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[ 0.0374, -0.2760, 0.7188, -1.1671],\n", + " [-0.7804, -1.6025, -0.2721, -0.0135],\n", + " [ 0.3318, -1.2199, 0.5424, -0.3309]])" + ] + }, + "metadata": {}, + "execution_count": 12 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QMOkL82htt9C" + }, + "source": [ + "## Operations\n", + "\n", + "Oftentimes, we want to apply functions to arrays. Some of the simplest and most useful functions are the element-wise functions. These operate by performing a single scalar operation on the corresponding elements of two arrays. We can create an element-wise function from any function that maps from the scalars to the scalars. In math notations we would denote such a function as $f: \\mathbb{R} \\rightarrow \\mathbb{R}$. Given any two vectors $\\mathbf{u}$ and $\\mathbf{v}$ *of the same shape*, and the function f,\n", + "we can produce a vector $\\mathbf{c} = F(\\mathbf{u},\\mathbf{v})$ by setting $c_i \\gets f(u_i, v_i)$ for all $i$. Here, we produced the vector-valued $F: \\mathbb{R}^d \\rightarrow \\mathbb{R}^d$ by *lifting* the scalar function to an element-wise vector operation. In PyTorch, the common standard arithmetic operators (+,-,/,\\*,\\*\\*) have all been *lifted* to element-wise operations for identically-shaped tensors of arbitrary shape. We can call element-wise operations on any two tensors of the same shape, including matrices." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "gd1mLZ78tt9C", + "outputId": "3b75c552-c4ab-45f9-b661-ec309f58eb98", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "x = torch.tensor([1, 2, 4, 8], dtype=torch.float32)\n", + "y = torch.ones_like(x) * 2\n", + "print('x =', x)\n", + "print('x + y', x + y)\n", + "print('x - y', x - y)\n", + "print('x * y', x * y)\n", + "print('x / y', x / y)" + ], + "execution_count": 13, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "x = tensor([1., 2., 4., 8.])\n", + "x + y tensor([ 3., 4., 6., 10.])\n", + "x - y tensor([-1., 0., 2., 6.])\n", + "x * y tensor([ 2., 4., 8., 16.])\n", + "x / y tensor([0.5000, 1.0000, 2.0000, 4.0000])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "YYqWBqk-tt9C" + }, + "source": [ + "Many more operations can be applied element-wise, such as exponentiation:" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "Rj7dE0e4tt9C", + "outputId": "1a205203-dba0-4393-b99d-415818072b84", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "torch.exp(x)\n", + "# Note: torch.exp is not implemented for 'torch.LongTensor'." + ], + "execution_count": 14, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([2.7183e+00, 7.3891e+00, 5.4598e+01, 2.9810e+03])" + ] + }, + "metadata": {}, + "execution_count": 14 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OwROBDvatt9D" + }, + "source": [ + "In addition to computations by element, we can also perform matrix operations, like matrix multiplication using the `mm` or `matmul` function. Next, we will perform matrix multiplication of `x` and the transpose of `y`. We define `x` as a matrix of 3 rows and 4 columns, and `y` is transposed into a matrix of 4 rows and 3 columns. The two matrices are multiplied to obtain a matrix of 3 rows and 3 columns." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ysU1rfVQtt9D", + "outputId": "367d06bb-b0a8-4336-b48b-bedefc30b6d1", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "x = torch.arange(12, dtype=torch.float32).reshape((3,4))\n", + "y = torch.tensor([[2, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]], dtype=torch.float32)\n", + "print(x.dtype)\n", + "print(y)\n", + "torch.mm(x, y.t())" + ], + "execution_count": 15, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "torch.float32\n", + "tensor([[2., 1., 4., 3.],\n", + " [1., 2., 3., 4.],\n", + " [4., 3., 2., 1.]])\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[ 18., 20., 10.],\n", + " [ 58., 60., 50.],\n", + " [ 98., 100., 90.]])" + ] + }, + "metadata": {}, + "execution_count": 15 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "EzrS1INmtt9D" + }, + "source": [ + "Note that torch.dot() behaves differently to np.dot(). There's been some discussion about what would be desirable here. Specifically, torch.dot() treats both a and b as 1D vectors (irrespective of their original shape) and computes their inner product. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fUD_tOQDtt9D" + }, + "source": [ + "We can also merge multiple tensors. For that, we need to tell the system along which dimension to merge. The example below merges two matrices along dimension 0 (along rows) and dimension 1 (along columns) respectively." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "jWkANgjXtt9D", + "outputId": "0ded388f-8cbd-487b-a1ea-f24845ae4f15", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "torch.cat((x, y), dim=0)" + ], + "execution_count": 16, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[ 0., 1., 2., 3.],\n", + " [ 4., 5., 6., 7.],\n", + " [ 8., 9., 10., 11.],\n", + " [ 2., 1., 4., 3.],\n", + " [ 1., 2., 3., 4.],\n", + " [ 4., 3., 2., 1.]])" + ] + }, + "metadata": {}, + "execution_count": 16 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "uutHkzUjtt9D", + "outputId": "c6a61794-639d-45ef-e0eb-0ccfc08c669b", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "torch.cat((x, y), dim=1)" + ], + "execution_count": 17, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[ 0., 1., 2., 3., 2., 1., 4., 3.],\n", + " [ 4., 5., 6., 7., 1., 2., 3., 4.],\n", + " [ 8., 9., 10., 11., 4., 3., 2., 1.]])" + ] + }, + "metadata": {}, + "execution_count": 17 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TPDVD8dAtt9D" + }, + "source": [ + "Sometimes, we may want to construct binary tensors via logical statements. Take `x == y` as an example. If `x` and `y` are equal for some entry, the new tensor has a value of 1 at the same position; otherwise it is 0." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ECN_vAVBtt9E", + "outputId": "d78441a5-a231-438e-c12e-4384533ad409", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "x == y" + ], + "execution_count": 18, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[False, True, False, True],\n", + " [False, False, False, False],\n", + " [False, False, False, False]])" + ] + }, + "metadata": {}, + "execution_count": 18 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "jO708QmqLyI9", + "outputId": "db81f519-8dde-44d4-b4e6-819882c68204", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "x < y" + ], + "execution_count": 19, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[ True, False, True, False],\n", + " [False, False, False, False],\n", + " [False, False, False, False]])" + ] + }, + "metadata": {}, + "execution_count": 19 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "Vm0VanTzL0HV", + "outputId": "b413b673-150d-4841-bc4f-da988ed8a76c", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "x > y" + ], + "execution_count": 20, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[False, False, False, False],\n", + " [ True, True, True, True],\n", + " [ True, True, True, True]])" + ] + }, + "metadata": {}, + "execution_count": 20 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "J35kciqmtt9E" + }, + "source": [ + "Summing all the elements in the tensor yields an tensor with only one element." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ZKzX-oaCtt9E", + "outputId": "be30f823-fe37-43ea-8f8a-131db0cf8f34", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "x.sum()" + ], + "execution_count": 21, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor(66.)" + ] + }, + "metadata": {}, + "execution_count": 21 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "GgLs3-KQtt9E" + }, + "source": [ + "We can transform the result into a scalar in Python using the `asscalar` function of `numpy`(from numpy v1.16 onwards it issues a deprication warning. We can ignore that at this moment). In the following example, the $\\ell_2$ norm of `x` yields a single element tensor. The final result is transformed into a scalar." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "VXaNZhtbtt9E", + "outputId": "35731066-db92-404d-9e4f-d9117be701cd", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "import numpy as np\n", + "np.asscalar(x.norm())" + ], + "execution_count": 22, + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:2: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead\n", + " \n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "22.494443893432617" + ] + }, + "metadata": {}, + "execution_count": 22 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "odgohox_tt9E" + }, + "source": [ + "## Broadcast Mechanism\n", + "\n", + "In the above section, we saw how to perform operations on two tensors of the same shape. When their shapes differ, a broadcasting mechanism may be triggered analogous to NumPy: first, copy the elements appropriately so that the two tensors have the same shape, and then carry out operations by element." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "fPoKku5ttt9E", + "outputId": "ccae0494-c3c8-4271-b9f9-bf714607b23b", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "a = torch.arange(6, dtype=torch.float).reshape((3, 1, 2))\n", + "b = torch.arange(6, dtype=torch.float).reshape((1, 3, 2))\n", + "a, b" + ], + "execution_count": 23, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "(tensor([[[0., 1.]],\n", + " \n", + " [[2., 3.]],\n", + " \n", + " [[4., 5.]]]), tensor([[[0., 1.],\n", + " [2., 3.],\n", + " [4., 5.]]]))" + ] + }, + "metadata": {}, + "execution_count": 23 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6AoY2kaStt9F" + }, + "source": [ + "Since `a` and `b` are (3x1) and (1x2) matrices respectively, their shapes do not match up if we want to add them. PyTorch addresses this by 'broadcasting' the entries of both matrices into a larger (3x2) matrix as follows: for matrix `a` it replicates the columns, for matrix `b` it replicates the rows before adding up both element-wise." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "35eZ_NLvtt9F", + "outputId": "9ea0c24a-6d53-4803-b272-32215b89e1dc", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "a + b" + ], + "execution_count": 24, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[[ 0., 2.],\n", + " [ 2., 4.],\n", + " [ 4., 6.]],\n", + "\n", + " [[ 2., 4.],\n", + " [ 4., 6.],\n", + " [ 6., 8.]],\n", + "\n", + " [[ 4., 6.],\n", + " [ 6., 8.],\n", + " [ 8., 10.]]])" + ] + }, + "metadata": {}, + "execution_count": 24 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "XLKjJpBott9F" + }, + "source": [ + "## Indexing and Slicing\n", + "\n", + "Just like in any other Python array, elements in a tensor can be accessed by its index. In good Python tradition the first element has index 0 and ranges are specified to include the first but not the last element. By this logic `1:3` selects the second and third element. Let's try this out by selecting the respective rows in a matrix." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "r3r9M8sstt9F", + "outputId": "f9815a94-94d1-41b3-d1e2-d4a851fefd5b", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "x[1:3]" + ], + "execution_count": 25, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[ 4., 5., 6., 7.],\n", + " [ 8., 9., 10., 11.]])" + ] + }, + "metadata": {}, + "execution_count": 25 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_B0mt0DYtt9F" + }, + "source": [ + "Beyond reading, we can also write elements of a matrix." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "7iUehTOxtt9F", + "outputId": "a9731bc0-1e60-4cad-a932-268e60d50fe7", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "x[1, 2] = 9\n", + "x" + ], + "execution_count": 26, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[ 0., 1., 2., 3.],\n", + " [ 4., 5., 9., 7.],\n", + " [ 8., 9., 10., 11.]])" + ] + }, + "metadata": {}, + "execution_count": 26 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "WLXrxtpltt9F" + }, + "source": [ + "If we want to assign multiple elements the same value, we simply index all of them and then assign them the value. For instance, `[0:2, :]` accesses the first and second rows. While we discussed indexing for matrices, this obviously also works for vectors and for tensors of more than 2 dimensions." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "NpOPh4xftt9G", + "outputId": "2aea6bcb-db99-44fb-fe5c-8fb7e3e52b13", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "x[0:2, :] = 12\n", + "x" + ], + "execution_count": 27, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "tensor([[12., 12., 12., 12.],\n", + " [12., 12., 12., 12.],\n", + " [ 8., 9., 10., 11.]])" + ] + }, + "metadata": {}, + "execution_count": 27 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7pe8JqtLtt9G" + }, + "source": [ + "## Saving Memory\n", + "\n", + "In the previous example, every time we ran an operation, we allocated new memory to host its results. For example, if we write `y = x + y`, we will dereference the matrix that `y` used to point to and instead point it at the newly allocated memory. In the following example we demonstrate this with Python's `id()` function, which gives us the exact address of the referenced object in memory. After running `y = y + x`, we will find that `id(y)` points to a different location. That is because Python first evaluates `y + x`, allocating new memory for the result and then subsequently redirects `y` to point at this new location in memory." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "qAxcxAiRtt9G", + "outputId": "f6cdb94c-5758-48e5-a91b-b8f7e604d0cb", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "before = id(y)\n", + "y = y + x\n", + "id(y) == before" + ], + "execution_count": 28, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "False" + ] + }, + "metadata": {}, + "execution_count": 28 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "CmeVoSMHtt9G" + }, + "source": [ + "This might be undesirable for two reasons. First, we do not want to run around allocating memory unnecessarily all the time. In machine learning, we might have hundreds of megabytes of parameters and update all of them multiple times per second. Typically, we will want to perform these updates *in place*. Second, we might point at the same parameters from multiple variables. If we do not update in place, this could cause a memory leak, making it possible for us to inadvertently reference stale parameters.\n", + "\n", + "Fortunately, performing in-place operations in PyTorch is easy. We can assign the result of an operation to a previously allocated array with slice notation, e.g., `y[:] = `. To illustrate the behavior, we first clone the shape of a matrix using `zeros_like` to allocate a block of 0 entries." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "2N0Du3y0tt9G", + "outputId": "73e41632-0d80-4250-891b-b3931e8a10be", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "z = torch.zeros_like(y)\n", + "print('id(z):', id(z))\n", + "z[:] = x + y\n", + "print('id(z):', id(z))" + ], + "execution_count": 29, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "id(z): 140701309674832\n", + "id(z): 140701309674832\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "R5oYULP9tt9G" + }, + "source": [ + "While this looks pretty, `x+y` here will still allocate a temporary buffer to store the result of `x+y` before copying it to `z[:]`. To make even better use of memory, we can directly invoke the underlying `tensor` operation, in this case `add`, avoiding temporary buffers. We do this by specifying the `out` keyword argument, which every `tensor` operator supports:" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "9DsVYxgTtt9G", + "outputId": "a9f2a270-fc62-401f-c185-485547ff1cc4", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "before = id(z)\n", + "torch.add(x, y, out=z)\n", + "id(z) == before" + ], + "execution_count": 30, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "True" + ] + }, + "metadata": {}, + "execution_count": 30 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BXKcHAsgtt9H" + }, + "source": [ + "If the value of `x ` is not reused in subsequent computations, we can also use `x[:] = x + y` or `x += y` to reduce the memory overhead of the operation." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "_R1hG073tt9H", + "outputId": "aeb9324c-8e55-401f-e383-2ef4cd7e1b5e", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "before = id(x)\n", + "x += y\n", + "id(x) == before" + ], + "execution_count": 31, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "True" + ] + }, + "metadata": {}, + "execution_count": 31 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "uufoIdfhtt9H" + }, + "source": [ + "## Mutual Transformation of PyTorch and NumPy\n", + "\n", + "Converting PyTorch Tensors to and from NumPy Arrays is easy. The converted arrays do *not* share memory. This minor inconvenience is actually quite important: when you perform operations on the CPU or one of the GPUs, you do not want PyTorch having to wait whether NumPy might want to be doing something else with the same chunk of memory. `.tensor` and `.numpy` do the trick." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "l8SkvcxUtt9H", + "outputId": "a073f69b-b010-4228-9cd8-28fe4d87e067", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "a = x.numpy()\n", + "print(type(a))\n", + "b = torch.tensor(a)\n", + "print(type(b))" + ], + "execution_count": 32, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "\n", + "\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "By1uV9SPtt9H" + }, + "source": [ + "## Exercises\n", + "\n", + "1. Run the code in this section. Change the conditional statement `x == y` in this section to `x < y` or `x > y`, and then see what kind of tensor you can get.\n", + "1. Replace the two tensors that operate by element in the broadcast mechanism with other shapes, e.g. three dimensional tensors. Is the result the same as expected?\n", + "1. Assume that we have three matrices `a`, `b` and `c`. Rewrite `c = torch.mm(a, b.t()) + c` in the most memory efficient manner." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MgJkkq1S4pyZ" + }, + "source": [ + "## Anwers\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "W1ABR5rb5Eye" + }, + "source": [ + " 1. Check the code above. " + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "kX1EC_sw5Ikz" + }, + "source": [ + "2. Check the code above. The result is the same as expected.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wnZ3oxpz5ND7" + }, + "source": [ + "3. \n", + "\n", + "```\n", + "x = torch.zeros_like(c)\n", + "torch.mm(a, b.t(), out=x)\n", + "c += x\n", + "```" + ] } - ], - "source": [ - "a = x.numpy()\n", - "print(type(a))\n", - "b = torch.tensor(a)\n", - "print(type(b))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Exercises\n", - "\n", - "1. Run the code in this section. Change the conditional statement `x == y` in this section to `x < y` or `x > y`, and then see what kind of tensor you can get.\n", - "1. Replace the two tensors that operate by element in the broadcast mechanism with other shapes, e.g. three dimensional tensors. Is the result the same as expected?\n", - "1. Assume that we have three matrices `a`, `b` and `c`. Rewrite `c = torch.mm(a, b.t()) + c` in the most memory efficient manner." - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.8" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} + ] +} \ No newline at end of file From 697d7ea18cf2485a08e3a08e5a6ecd29941aa485 Mon Sep 17 00:00:00 2001 From: Moistsoy <89026391+Moistsoy@users.noreply.github.com> Date: Sun, 3 Oct 2021 03:16:45 +0900 Subject: [PATCH 3/7] Created using Colaboratory --- .../Data_Manipulation.ipynb | 322 +++++++++--------- 1 file changed, 160 insertions(+), 162 deletions(-) diff --git a/Ch04_The_Preliminaries_A_Crashcourse/Data_Manipulation.ipynb b/Ch04_The_Preliminaries_A_Crashcourse/Data_Manipulation.ipynb index 6e963ba..2d1f0f4 100644 --- a/Ch04_The_Preliminaries_A_Crashcourse/Data_Manipulation.ipynb +++ b/Ch04_The_Preliminaries_A_Crashcourse/Data_Manipulation.ipynb @@ -70,11 +70,12 @@ { "cell_type": "code", "metadata": { - "id": "f4RY3QwHtt83", - "outputId": "2580df63-9104-42ea-cf25-3a8db4ab0086", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "f4RY3QwHtt83", + "outputId": "2580df63-9104-42ea-cf25-3a8db4ab0086" }, "source": [ "x = torch.arange(12, dtype=torch.float64)\n", @@ -98,11 +99,12 @@ { "cell_type": "code", "metadata": { - "id": "g8IgkvQwtt85", - "outputId": "27080c3c-b0fc-4acd-ab65-18dc1f47591e", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "g8IgkvQwtt85", + "outputId": "27080c3c-b0fc-4acd-ab65-18dc1f47591e" }, "source": [ "# We can get the tensor shape through the shape attribute.\n", @@ -126,11 +128,12 @@ "cell_type": "code", "metadata": { "scrolled": true, - "id": "vYaIGyCott85", - "outputId": "ca79721d-90a9-45be-ed65-a513c5e3f9d6", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "vYaIGyCott85", + "outputId": "ca79721d-90a9-45be-ed65-a513c5e3f9d6" }, "source": [ "# .shape is an alias for .size(), and was added to more closely match numpy\n", @@ -163,11 +166,12 @@ { "cell_type": "code", "metadata": { - "id": "Hi1ZVTY8tt88", - "outputId": "8a4089e2-b704-401c-b8b7-80a145b88fb0", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "Hi1ZVTY8tt88", + "outputId": "8a4089e2-b704-401c-b8b7-80a145b88fb0" }, "source": [ "x = x.reshape((3, 4))\n", @@ -203,11 +207,12 @@ { "cell_type": "code", "metadata": { - "id": "c4kG5w9Ftt89", - "outputId": "e18b1e57-04d6-4a70-db12-dcc13497df30", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "c4kG5w9Ftt89", + "outputId": "e18b1e57-04d6-4a70-db12-dcc13497df30" }, "source": [ "torch.FloatTensor(2, 3)" @@ -230,11 +235,12 @@ { "cell_type": "code", "metadata": { - "id": "lDMwcl0Qtt8-", - "outputId": "833d8dc4-bb96-4ebe-a0bf-8cd220a9299d", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "lDMwcl0Qtt8-", + "outputId": "833d8dc4-bb96-4ebe-a0bf-8cd220a9299d" }, "source": [ "torch.Tensor(2, 3)" @@ -257,11 +263,12 @@ { "cell_type": "code", "metadata": { - "id": "lmpl4ulktt8-", - "outputId": "12858e8d-0fdb-4e1a-b820-ff67dc100c75", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "lmpl4ulktt8-", + "outputId": "12858e8d-0fdb-4e1a-b820-ff67dc100c75" }, "source": [ "torch.empty(2, 3)" @@ -301,11 +308,12 @@ { "cell_type": "code", "metadata": { - "id": "qTaM6HfTtt8_", - "outputId": "4ccb042d-c15e-4b69-a005-e42081f76500", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "qTaM6HfTtt8_", + "outputId": "4ccb042d-c15e-4b69-a005-e42081f76500" }, "source": [ "torch.zeros((2, 3, 4))" @@ -342,11 +350,12 @@ { "cell_type": "code", "metadata": { - "id": "9U4aMt6ett9A", - "outputId": "1d61889b-129a-4238-c245-40591a562047", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "9U4aMt6ett9A", + "outputId": "1d61889b-129a-4238-c245-40591a562047" }, "source": [ "torch.ones((2, 3, 4))" @@ -383,11 +392,12 @@ { "cell_type": "code", "metadata": { - "id": "o-trgxDmtt9B", - "outputId": "95fcad52-3395-4a67-9d29-3e6b9fbbd5c3", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "o-trgxDmtt9B", + "outputId": "95fcad52-3395-4a67-9d29-3e6b9fbbd5c3" }, "source": [ "y = torch.tensor([[2, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]])\n", @@ -421,11 +431,12 @@ { "cell_type": "code", "metadata": { - "id": "Y552VuCwtt9B", - "outputId": "1fd2c818-1f00-40f7-b2ba-820f07f4c0ce", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "Y552VuCwtt9B", + "outputId": "1fd2c818-1f00-40f7-b2ba-820f07f4c0ce" }, "source": [ "torch.randn(3, 4)" @@ -461,11 +472,12 @@ { "cell_type": "code", "metadata": { - "id": "gd1mLZ78tt9C", - "outputId": "3b75c552-c4ab-45f9-b661-ec309f58eb98", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "gd1mLZ78tt9C", + "outputId": "3b75c552-c4ab-45f9-b661-ec309f58eb98" }, "source": [ "x = torch.tensor([1, 2, 4, 8], dtype=torch.float32)\n", @@ -503,11 +515,12 @@ { "cell_type": "code", "metadata": { - "id": "Rj7dE0e4tt9C", - "outputId": "1a205203-dba0-4393-b99d-415818072b84", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "Rj7dE0e4tt9C", + "outputId": "1a205203-dba0-4393-b99d-415818072b84" }, "source": [ "torch.exp(x)\n", @@ -539,11 +552,12 @@ { "cell_type": "code", "metadata": { - "id": "ysU1rfVQtt9D", - "outputId": "367d06bb-b0a8-4336-b48b-bedefc30b6d1", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "ysU1rfVQtt9D", + "outputId": "367d06bb-b0a8-4336-b48b-bedefc30b6d1" }, "source": [ "x = torch.arange(12, dtype=torch.float32).reshape((3,4))\n", @@ -599,11 +613,12 @@ { "cell_type": "code", "metadata": { - "id": "jWkANgjXtt9D", - "outputId": "0ded388f-8cbd-487b-a1ea-f24845ae4f15", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "jWkANgjXtt9D", + "outputId": "0ded388f-8cbd-487b-a1ea-f24845ae4f15" }, "source": [ "torch.cat((x, y), dim=0)" @@ -630,11 +645,12 @@ { "cell_type": "code", "metadata": { - "id": "uutHkzUjtt9D", - "outputId": "c6a61794-639d-45ef-e0eb-0ccfc08c669b", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "uutHkzUjtt9D", + "outputId": "c6a61794-639d-45ef-e0eb-0ccfc08c669b" }, "source": [ "torch.cat((x, y), dim=1)" @@ -667,11 +683,12 @@ { "cell_type": "code", "metadata": { - "id": "ECN_vAVBtt9E", - "outputId": "d78441a5-a231-438e-c12e-4384533ad409", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "ECN_vAVBtt9E", + "outputId": "d78441a5-a231-438e-c12e-4384533ad409" }, "source": [ "x == y" @@ -695,11 +712,12 @@ { "cell_type": "code", "metadata": { - "id": "jO708QmqLyI9", - "outputId": "db81f519-8dde-44d4-b4e6-819882c68204", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "jO708QmqLyI9", + "outputId": "db81f519-8dde-44d4-b4e6-819882c68204" }, "source": [ "x < y" @@ -723,11 +741,12 @@ { "cell_type": "code", "metadata": { - "id": "Vm0VanTzL0HV", - "outputId": "b413b673-150d-4841-bc4f-da988ed8a76c", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "Vm0VanTzL0HV", + "outputId": "b413b673-150d-4841-bc4f-da988ed8a76c" }, "source": [ "x > y" @@ -760,11 +779,12 @@ { "cell_type": "code", "metadata": { - "id": "ZKzX-oaCtt9E", - "outputId": "be30f823-fe37-43ea-8f8a-131db0cf8f34", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "ZKzX-oaCtt9E", + "outputId": "be30f823-fe37-43ea-8f8a-131db0cf8f34" }, "source": [ "x.sum()" @@ -795,11 +815,12 @@ { "cell_type": "code", "metadata": { - "id": "VXaNZhtbtt9E", - "outputId": "35731066-db92-404d-9e4f-d9117be701cd", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "VXaNZhtbtt9E", + "outputId": "35731066-db92-404d-9e4f-d9117be701cd" }, "source": [ "import numpy as np\n", @@ -841,11 +862,12 @@ { "cell_type": "code", "metadata": { - "id": "fPoKku5ttt9E", - "outputId": "ccae0494-c3c8-4271-b9f9-bf714607b23b", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "fPoKku5ttt9E", + "outputId": "ccae0494-c3c8-4271-b9f9-bf714607b23b" }, "source": [ "a = torch.arange(6, dtype=torch.float).reshape((3, 1, 2))\n", @@ -884,11 +906,12 @@ { "cell_type": "code", "metadata": { - "id": "35eZ_NLvtt9F", - "outputId": "9ea0c24a-6d53-4803-b272-32215b89e1dc", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "35eZ_NLvtt9F", + "outputId": "9ea0c24a-6d53-4803-b272-32215b89e1dc" }, "source": [ "a + b" @@ -931,11 +954,12 @@ { "cell_type": "code", "metadata": { - "id": "r3r9M8sstt9F", - "outputId": "f9815a94-94d1-41b3-d1e2-d4a851fefd5b", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "r3r9M8sstt9F", + "outputId": "f9815a94-94d1-41b3-d1e2-d4a851fefd5b" }, "source": [ "x[1:3]" @@ -967,11 +991,12 @@ { "cell_type": "code", "metadata": { - "id": "7iUehTOxtt9F", - "outputId": "a9731bc0-1e60-4cad-a932-268e60d50fe7", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "7iUehTOxtt9F", + "outputId": "a9731bc0-1e60-4cad-a932-268e60d50fe7" }, "source": [ "x[1, 2] = 9\n", @@ -1005,11 +1030,12 @@ { "cell_type": "code", "metadata": { - "id": "NpOPh4xftt9G", - "outputId": "2aea6bcb-db99-44fb-fe5c-8fb7e3e52b13", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "NpOPh4xftt9G", + "outputId": "2aea6bcb-db99-44fb-fe5c-8fb7e3e52b13" }, "source": [ "x[0:2, :] = 12\n", @@ -1045,11 +1071,12 @@ { "cell_type": "code", "metadata": { - "id": "qAxcxAiRtt9G", - "outputId": "f6cdb94c-5758-48e5-a91b-b8f7e604d0cb", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "qAxcxAiRtt9G", + "outputId": "f6cdb94c-5758-48e5-a91b-b8f7e604d0cb" }, "source": [ "before = id(y)\n", @@ -1084,11 +1111,12 @@ { "cell_type": "code", "metadata": { - "id": "2N0Du3y0tt9G", - "outputId": "73e41632-0d80-4250-891b-b3931e8a10be", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "2N0Du3y0tt9G", + "outputId": "73e41632-0d80-4250-891b-b3931e8a10be" }, "source": [ "z = torch.zeros_like(y)\n", @@ -1120,11 +1148,12 @@ { "cell_type": "code", "metadata": { - "id": "9DsVYxgTtt9G", - "outputId": "a9f2a270-fc62-401f-c185-485547ff1cc4", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "9DsVYxgTtt9G", + "outputId": "a9f2a270-fc62-401f-c185-485547ff1cc4" }, "source": [ "before = id(z)\n", @@ -1157,11 +1186,12 @@ { "cell_type": "code", "metadata": { - "id": "_R1hG073tt9H", - "outputId": "aeb9324c-8e55-401f-e383-2ef4cd7e1b5e", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "_R1hG073tt9H", + "outputId": "aeb9324c-8e55-401f-e383-2ef4cd7e1b5e" }, "source": [ "before = id(x)\n", @@ -1196,11 +1226,12 @@ { "cell_type": "code", "metadata": { - "id": "l8SkvcxUtt9H", - "outputId": "a073f69b-b010-4228-9cd8-28fe4d87e067", "colab": { - "base_uri": "https://localhost:8080/" - } + "base_uri": "https://localhost:8080/", + "height": 0 + }, + "id": "l8SkvcxUtt9H", + "outputId": "a073f69b-b010-4228-9cd8-28fe4d87e067" }, "source": [ "a = x.numpy()\n", @@ -1229,45 +1260,12 @@ "## Exercises\n", "\n", "1. Run the code in this section. Change the conditional statement `x == y` in this section to `x < y` or `x > y`, and then see what kind of tensor you can get.\n", - "1. Replace the two tensors that operate by element in the broadcast mechanism with other shapes, e.g. three dimensional tensors. Is the result the same as expected?\n", - "1. Assume that we have three matrices `a`, `b` and `c`. Rewrite `c = torch.mm(a, b.t()) + c` in the most memory efficient manner." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "MgJkkq1S4pyZ" - }, - "source": [ - "## Anwers\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "W1ABR5rb5Eye" - }, - "source": [ - " 1. Check the code above. " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "kX1EC_sw5Ikz" - }, - "source": [ - "2. Check the code above. The result is the same as expected.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "wnZ3oxpz5ND7" - }, - "source": [ - "3. \n", + "- Check the code above.\n", + "\n", + "2. Replace the two tensors that operate by element in the broadcast mechanism with other shapes, e.g. three dimensional tensors. Is the result the same as expected?\n", + "- Check the code above. The result is same as expected.\n", "\n", + "3. Assume that we have three matrices `a`, `b` and `c`. Rewrite `c = torch.mm(a, b.t()) + c` in the most memory efficient manner.\n", "```\n", "x = torch.zeros_like(c)\n", "torch.mm(a, b.t(), out=x)\n", From 13be2b81f3b5197a49eac4e95ebab2dd2918ba3d Mon Sep 17 00:00:00 2001 From: Moistsoy <89026391+Moistsoy@users.noreply.github.com> Date: Sun, 3 Oct 2021 04:09:06 +0900 Subject: [PATCH 4/7] Created using Colaboratory --- .../Automatic_Differentiation.ipynb | 798 ++++++++++-------- 1 file changed, 460 insertions(+), 338 deletions(-) diff --git a/Ch04_The_Preliminaries_A_Crashcourse/Automatic_Differentiation.ipynb b/Ch04_The_Preliminaries_A_Crashcourse/Automatic_Differentiation.ipynb index e0bfcc8..2f1175d 100644 --- a/Ch04_The_Preliminaries_A_Crashcourse/Automatic_Differentiation.ipynb +++ b/Ch04_The_Preliminaries_A_Crashcourse/Automatic_Differentiation.ipynb @@ -1,347 +1,469 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Automatic Differentiation\n", - "\n", - "In machine learning, we *train* models, updating them successively so that they get better and better as they see more and more data. Usually, *getting better* means minimizing a *loss function*, a score that answers the question \"how *bad* is our model?\" With neural networks, we typically choose loss functions that are differentiable with respect to our parameters.\n", - "Put simply, this means that for each of the model's parameters, we can determine how much *increasing* or *decreasing* it might affect the loss. While the calculations for taking these derivatives are straightforward, requiring only some basic calculus, for complex models, working out the updates by hand can be a pain (and often error-prone)." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The autograd package expedites this work by automatically calculating derivatives. And while many other libraries require that we compile a symbolic graph to take automatic derivatives, `autograd` allows us to take derivatives while writing ordinary imperative code. Every time we pass data through our model, `autograd` builds a graph on the fly, tracking which data combined through which operations to produce the output. This graph enables `autograd` to subsequently backpropagate gradients on command. Here *backpropagate* simply means to trace through the compute graph, filling in the partial derivatives with respect to each parameter." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "import torch\n", - "from torch.autograd import Variable" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## A Simple Example\n", - "\n", - "As a toy example, say that we are interested in differentiating the mapping $y = 2\\mathbf{x}^{\\top}\\mathbf{x}$ with respect to the column vector $\\mathbf{x}$. To start, let's create the variable `x` and assign it an initial value." - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "tensor([[0.],\n", - " [1.],\n", - " [2.],\n", - " [3.]], requires_grad=True)\n" - ] + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.1" + }, + "colab": { + "name": "Automatic_Differentiation.ipynb", + "provenance": [] } - ], - "source": [ - "x = Variable(torch.arange(4, dtype=torch.float32).reshape((4, 1)), requires_grad=True)\n", - "print(x)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Once we compute the gradient of ``y`` with respect to ``x``, we will need a place to store it. We can tell a tensor that we plan to store a gradient by the ``requires_grad=True`` keyword." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now we are going to compute ``y`` and PyTorch will generate a computation graph on the fly. Autograd is reverse automatic differentiation system. Conceptually, autograd records a graph recording all of the operations that created the data as you execute operations, giving you a directed acyclic graph whose leaves are the input tensors and roots are the output tensors. By tracing this graph from roots to leaves, you can automatically compute the gradients using the chain rule.\n", - "\n", - "Note that building the computation graph requires a nontrivial amount of computation. So PyTorch will *only* build the graph when explicitly told to do so. For a tensor to be “recordable”, it must be wrapped with torch.autograd.Variable. The Variable class provides almost the same API as Tensor, but augments it with the ability to interplay with torch.autograd.Function in order to be differentiated automatically. More precisely, a Variable records the history of operations on a Tensor." - ] }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ + "cells": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "tensor([[28.]], grad_fn=)\n" - ] - } - ], - "source": [ - "y = 2*torch.mm(x.t(),x)\n", - "print(y)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Since the shape of `x` is (4, 1), `y` is a scalar. Next, we can automatically find the gradient by calling the `backward` function. It should be noted that if `y` is not a scalar, PyTorch will first sum the elements in `y` to get the new variable by default, and then find the gradient of the variable with respect to `x`." - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "y.backward()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Since every Variable except for inputs is the result of an operation, each Variable has an associated grad_fn, which is the torch.autograd.Function that is used to compute the backward step. For inputs it is None:" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ + "cell_type": "markdown", + "metadata": { + "id": "pARL5s_sNHv2" + }, + "source": [ + "# Automatic Differentiation\n", + "\n", + "In machine learning, we *train* models, updating them successively so that they get better and better as they see more and more data. Usually, *getting better* means minimizing a *loss function*, a score that answers the question \"how *bad* is our model?\" With neural networks, we typically choose loss functions that are differentiable with respect to our parameters.\n", + "Put simply, this means that for each of the model's parameters, we can determine how much *increasing* or *decreasing* it might affect the loss. While the calculations for taking these derivatives are straightforward, requiring only some basic calculus, for complex models, working out the updates by hand can be a pain (and often error-prone)." + ] + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "x.grad: tensor([[ 0.],\n", - " [ 4.],\n", - " [ 8.],\n", - " [12.]])\n", - "x.grad_fn: None\n", - "y.grad_fn: \n" - ] - } - ], - "source": [ - "print(\"x.grad:\", x.grad)\n", - "print(\"x.grad_fn:\", x.grad_fn)\n", - "print(\"y.grad_fn:\", y.grad_fn)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The gradient of the function $y = 2\\mathbf{x}^{\\top}\\mathbf{x}$ with respect to $\\mathbf{x}$ should be $4\\mathbf{x}$. Now let's verify that the gradient produced is correct." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ + "cell_type": "markdown", + "metadata": { + "id": "rpyYGWxGNHv9" + }, + "source": [ + "The autograd package expedites this work by automatically calculating derivatives. And while many other libraries require that we compile a symbolic graph to take automatic derivatives, `autograd` allows us to take derivatives while writing ordinary imperative code. Every time we pass data through our model, `autograd` builds a graph on the fly, tracking which data combined through which operations to produce the output. This graph enables `autograd` to subsequently backpropagate gradients on command. Here *backpropagate* simply means to trace through the compute graph, filling in the partial derivatives with respect to each parameter." + ] + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "True\n", - "tensor([[ 0.],\n", - " [ 4.],\n", - " [ 8.],\n", - " [12.]])\n" - ] - } - ], - "source": [ - "print((x.grad - 4*x).norm().item() == 0)\n", - "print(x.grad)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Training Mode and Evaluation Mode\n", - "\n", - "`Model` will change the running mode to the evaluation mode on calling `model.eval()` or to the training mode on calling `model.train()`.\n", - "\n", - "In some cases, the same model behaves differently in the training and prediction modes (e.g. when using neural techniques such as dropout and batch normalization). In other cases, some models may store more auxiliary variables to make computing gradients easier. We will cover these differences in detail in later chapters. For now, you do not need to worry about them." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Computing the Gradient of Python Control Flow\n", - "\n", - "One benefit of using automatic differentiation is that even if the computational graph of the function contains Python's control flow (such as conditional and loop control), we may still be able to find the gradient of a variable. Consider the following program: It should be emphasized that the number of iterations of the loop (while loop) and the execution of the conditional judgment (if statement) depend on the value of the input `b`." - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [], - "source": [ - "def f(a):\n", - " b = a * 2\n", - " while b.norm().item() < 1000:\n", - " b = b * 2\n", - " if b.sum().item() > 0:\n", - " c = b\n", - " else:\n", - " c = 100 * b\n", - " return c" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Note that the number of iterations of the while loop and the execution of the conditional statement (if then else) depend on the value of `a`. To compute gradients, we need to `record` the calculation, and then call the `backward` function to calculate the gradient." - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [], - "source": [ - "a = torch.randn(size=(1,))\n", - "a.requires_grad=True\n", - "d = f(a)\n", - "d.backward()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's analyze the `f` function defined above. As you can see, it is piecewise linear in its input `a`. In other words, for any `a` there exists some constant such that for a given range `f(a) = g * a`. Consequently `d / a` allows us to verify that the gradient is correct:" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": {}, - "outputs": [ + "cell_type": "code", + "metadata": { + "id": "5XnuKK9iNHv-" + }, + "source": [ + "import torch\n", + "from torch.autograd import Variable" + ], + "execution_count": 1, + "outputs": [] + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "tensor([1], dtype=torch.uint8)\n" - ] - } - ], - "source": [ - "print(a.grad == (d / a))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Head gradients and the chain rule\n", - "\n", - "*Caution: This part is tricky and not necessary to understanding subsequent sections. That said, it is needed if you want to build new layers from scratch. You can skip this on a first read.*\n", - "\n", - "Sometimes when we call the backward method, e.g. `y.backward()`, where\n", - "`y` is a function of `x` we are just interested in the derivative of\n", - "`y` with respect to `x`. Mathematicians write this as\n", - "$\\frac{dy(x)}{dx}$. At other times, we may be interested in the\n", - "gradient of `z` with respect to `x`, where `z` is a function of `y`,\n", - "which in turn, is a function of `x`. That is, we are interested in\n", - "$\\frac{d}{dx} z(y(x))$. Recall that by the chain rule\n", - "\n", - "$$\\frac{d}{dx} z(y(x)) = \\frac{dz(y)}{dy} \\frac{dy(x)}{dx}.$$\n", - "\n", - "So, when ``y`` is part of a larger function ``z`` and we want ``x.grad`` to store $\\frac{dz}{dx}$, we can pass in the *head gradient* $\\frac{dz}{dy}$ as an input to ``backward()``. The default argument is ``torch.ones_like(y)``. See [Wikipedia](https://en.wikipedia.org/wiki/Chain_rule) for more details." - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": { - "scrolled": true - }, - "outputs": [ + "cell_type": "markdown", + "metadata": { + "id": "iOBsPSCzNHv_" + }, + "source": [ + "## A Simple Example\n", + "\n", + "As a toy example, say that we are interested in differentiating the mapping $y = 2\\mathbf{x}^{\\top}\\mathbf{x}$ with respect to the column vector $\\mathbf{x}$. To start, let's create the variable `x` and assign it an initial value." + ] + }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "tensor([[0.0000],\n", - " [4.0000],\n", - " [0.8000],\n", - " [0.1200]])\n" - ] + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "VM0iXJZtNHwA", + "outputId": "21f44e49-5465-4a0d-d345-feafc62b02ce" + }, + "source": [ + "x = Variable(torch.arange(4, dtype=torch.float32).reshape((4, 1)), requires_grad=True)\n", + "print(x)" + ], + "execution_count": 2, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "tensor([[0.],\n", + " [1.],\n", + " [2.],\n", + " [3.]], requires_grad=True)\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MODpmt4wNHwB" + }, + "source": [ + "Once we compute the gradient of ``y`` with respect to ``x``, we will need a place to store it. We can tell a tensor that we plan to store a gradient by the ``requires_grad=True`` keyword." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7YeCcrGTNHwC" + }, + "source": [ + "Now we are going to compute ``y`` and PyTorch will generate a computation graph on the fly. Autograd is reverse automatic differentiation system. Conceptually, autograd records a graph recording all of the operations that created the data as you execute operations, giving you a directed acyclic graph whose leaves are the input tensors and roots are the output tensors. By tracing this graph from roots to leaves, you can automatically compute the gradients using the chain rule.\n", + "\n", + "Note that building the computation graph requires a nontrivial amount of computation. So PyTorch will *only* build the graph when explicitly told to do so. For a tensor to be “recordable”, it must be wrapped with torch.autograd.Variable. The Variable class provides almost the same API as Tensor, but augments it with the ability to interplay with torch.autograd.Function in order to be differentiated automatically. More precisely, a Variable records the history of operations on a Tensor." + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "qlqJO2ANNHwC", + "outputId": "8bdc401a-01d2-4690-eefd-42dadec531f6" + }, + "source": [ + "y = 2*torch.mm(x.t(),x)\n", + "print(y)" + ], + "execution_count": 3, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "tensor([[28.]], grad_fn=)\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "055sbI9iNHwD" + }, + "source": [ + "Since the shape of `x` is (4, 1), `y` is a scalar. Next, we can automatically find the gradient by calling the `backward` function. It should be noted that if `y` is not a scalar, PyTorch will first sum the elements in `y` to get the new variable by default, and then find the gradient of the variable with respect to `x`." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "N_ehb6MENHwE" + }, + "source": [ + "y.backward()" + ], + "execution_count": 4, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ogzOdu12NHwF" + }, + "source": [ + "Since every Variable except for inputs is the result of an operation, each Variable has an associated grad_fn, which is the torch.autograd.Function that is used to compute the backward step. For inputs it is None:" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "YN8SfEMzNHwF", + "outputId": "af4975b8-ea31-439c-e407-007e4342f0f5" + }, + "source": [ + "print(\"x.grad:\", x.grad)\n", + "print(\"x.grad_fn:\", x.grad_fn)\n", + "print(\"y.grad_fn:\", y.grad_fn)" + ], + "execution_count": 5, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "x.grad: tensor([[ 0.],\n", + " [ 4.],\n", + " [ 8.],\n", + " [12.]])\n", + "x.grad_fn: None\n", + "y.grad_fn: \n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1a_GAR75NHwG" + }, + "source": [ + "The gradient of the function $y = 2\\mathbf{x}^{\\top}\\mathbf{x}$ with respect to $\\mathbf{x}$ should be $4\\mathbf{x}$. Now let's verify that the gradient produced is correct." + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "jHvO1YEJNHwG", + "outputId": "4cc25e9b-9505-49fa-dbe9-b62f7bf5bdfc" + }, + "source": [ + "print((x.grad - 4*x).norm().item() == 0)\n", + "print(x.grad)" + ], + "execution_count": 6, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "True\n", + "tensor([[ 0.],\n", + " [ 4.],\n", + " [ 8.],\n", + " [12.]])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HTOkervHNHwH" + }, + "source": [ + "## Training Mode and Evaluation Mode\n", + "\n", + "`Model` will change the running mode to the evaluation mode on calling `model.eval()` or to the training mode on calling `model.train()`.\n", + "\n", + "In some cases, the same model behaves differently in the training and prediction modes (e.g. when using neural techniques such as dropout and batch normalization). In other cases, some models may store more auxiliary variables to make computing gradients easier. We will cover these differences in detail in later chapters. For now, you do not need to worry about them." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "C4tYrWGcNHwH" + }, + "source": [ + "## Computing the Gradient of Python Control Flow\n", + "\n", + "One benefit of using automatic differentiation is that even if the computational graph of the function contains Python's control flow (such as conditional and loop control), we may still be able to find the gradient of a variable. Consider the following program: It should be emphasized that the number of iterations of the loop (while loop) and the execution of the conditional judgment (if statement) depend on the value of the input `b`." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "83S26YLWNHwI" + }, + "source": [ + "def f(a):\n", + " b = a * 2\n", + " while b.norm().item() < 1000:\n", + " b = b * 2\n", + " if b.sum().item() > 0:\n", + " c = b\n", + " else:\n", + " c = 100 * b\n", + " return c" + ], + "execution_count": 7, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Az0abh03NHwI" + }, + "source": [ + "Note that the number of iterations of the while loop and the execution of the conditional statement (if then else) depend on the value of `a`. To compute gradients, we need to `record` the calculation, and then call the `backward` function to calculate the gradient." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "Q-qRa2EkNHwI" + }, + "source": [ + "a = torch.randn(size=(1,))\n", + "a.requires_grad=True\n", + "d = f(a)\n", + "d.backward()" + ], + "execution_count": 8, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "4ghuYwNqSJ7t", + "outputId": "60982897-5e29-456f-f586-25f1165fd6f4" + }, + "source": [ + "a = torch.randn(size=(2,))\n", + "print(f(a))" + ], + "execution_count": 15, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "tensor([ -54457.0117, -175862.7812])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "iVamDxunNHwI" + }, + "source": [ + "Let's analyze the `f` function defined above. As you can see, it is piecewise linear in its input `a`. In other words, for any `a` there exists some constant such that for a given range `f(a) = g * a`. Consequently `d / a` allows us to verify that the gradient is correct:" + ] + }, + { + "cell_type": "code", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "BwXGi43jNHwJ", + "outputId": "98a97206-6c25-47c4-9880-86c280eb1930" + }, + "source": [ + "print(a.grad == (d / a))" + ], + "execution_count": 9, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "tensor([True])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "AEkeomgkNHwJ" + }, + "source": [ + "## Head gradients and the chain rule\n", + "\n", + "*Caution: This part is tricky and not necessary to understanding subsequent sections. That said, it is needed if you want to build new layers from scratch. You can skip this on a first read.*\n", + "\n", + "Sometimes when we call the backward method, e.g. `y.backward()`, where\n", + "`y` is a function of `x` we are just interested in the derivative of\n", + "`y` with respect to `x`. Mathematicians write this as\n", + "$\\frac{dy(x)}{dx}$. At other times, we may be interested in the\n", + "gradient of `z` with respect to `x`, where `z` is a function of `y`,\n", + "which in turn, is a function of `x`. That is, we are interested in\n", + "$\\frac{d}{dx} z(y(x))$. Recall that by the chain rule\n", + "\n", + "$$\\frac{d}{dx} z(y(x)) = \\frac{dz(y)}{dy} \\frac{dy(x)}{dx}.$$\n", + "\n", + "So, when ``y`` is part of a larger function ``z`` and we want ``x.grad`` to store $\\frac{dz}{dx}$, we can pass in the *head gradient* $\\frac{dz}{dy}$ as an input to ``backward()``. The default argument is ``torch.ones_like(y)``. See [Wikipedia](https://en.wikipedia.org/wiki/Chain_rule) for more details." + ] + }, + { + "cell_type": "code", + "metadata": { + "scrolled": true, + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "HZvGe9O-NHwJ", + "outputId": "d49aa149-3f78-46a9-d327-a4a047715bf6" + }, + "source": [ + "x = Variable(torch.tensor([[0.],[1.],[2.],[3.]]), requires_grad=True)\n", + "y = x * 2\n", + "z = y * x\n", + "\n", + "head_gradient = torch.tensor([[10], [1.], [.1], [.01]])\n", + "z.backward(head_gradient)\n", + "print(x.grad)" + ], + "execution_count": 10, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "tensor([[0.0000],\n", + " [4.0000],\n", + " [0.8000],\n", + " [0.1200]])\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8KD80FSoNHwK" + }, + "source": [ + "## Summary\n", + "\n", + "* PyTorch provides an `autograd` package to automate the derivation process.\n", + "* PyTorch's `autograd` package can be used to derive general imperative programs.\n", + "* The running modes of PyTorch include the training mode and the evaluation mode." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "s0vXkwaoNHwK" + }, + "source": [ + "## Exercises\n", + "\n", + "1. In the control flow example where we calculate the derivative of `d` with respect to `a`, what would happen if we changed the variable `a` to a random vector or matrix. At this point, the result of the calculation `f(a)` is no longer a scalar. What happens to the result? How do we analyze this?\n", + "- They result would be vector or matrix\n", + "\n", + "2. Redesign an example of finding the gradient of the control flow. Run and analyze the result.\n", + "-\n", + "\n", + "3. In a second-price auction (such as in eBay or in computational advertising), the winning bidder pays the second-highest price. Compute the gradient of the final price with respect to the winning bidder's bid using `autograd`. What does the result tell you about the mechanism? If you are curious to learn more about second-price auctions, check out this paper by [Edelman, Ostrovski and Schwartz, 2005](https://www.benedelman.org/publications/gsp-060801.pdf).\n", + "-\n", + "\n", + "4. Why is the second derivative much more expensive to compute than the first derivative?\n", + "- Because of the chain rule, we need to compute $N^2$ elements while we need to compute $N$ elements in the first derivative.\n", + "\n", + "5. Derive the head gradient relationship for the chain rule. If you get stuck, use the [\"Chain rule\" article on Wikipedia](https://en.wikipedia.org/wiki/Chain_rule).\n", + "- $\\text{head gradient} = \\frac{dz}{dy} = \\frac{dz}{dx}\\frac{dx}{dy}$\n", + "\n", + "6. Assume $f(x) = \\sin(x)$. Plot $f(x)$ and $\\frac{df(x)}{dx}$ on a graph, where you computed the latter without any symbolic calculations, i.e. without exploiting that $f'(x) = \\cos(x)$.\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "aMYeHC8lSD1m" + }, + "source": [ + "" + ], + "execution_count": null, + "outputs": [] } - ], - "source": [ - "x = Variable(torch.tensor([[0.],[1.],[2.],[3.]]), requires_grad=True)\n", - "y = x * 2\n", - "z = y * x\n", - "\n", - "head_gradient = torch.tensor([[10], [1.], [.1], [.01]])\n", - "z.backward(head_gradient)\n", - "print(x.grad)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Summary\n", - "\n", - "* PyTorch provides an `autograd` package to automate the derivation process.\n", - "* PyTorch's `autograd` package can be used to derive general imperative programs.\n", - "* The running modes of PyTorch include the training mode and the evaluation mode." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Exercises\n", - "\n", - "1. In the control flow example where we calculate the derivative of `d` with respect to `a`, what would happen if we changed the variable `a` to a random vector or matrix. At this point, the result of the calculation `f(a)` is no longer a scalar. What happens to the result? How do we analyze this?\n", - "1. Redesign an example of finding the gradient of the control flow. Run and analyze the result.\n", - "1. In a second-price auction (such as in eBay or in computational advertising), the winning bidder pays the second-highest price. Compute the gradient of the final price with respect to the winning bidder's bid using `autograd`. What does the result tell you about the mechanism? If you are curious to learn more about second-price auctions, check out this paper by [Edelman, Ostrovski and Schwartz, 2005](https://www.benedelman.org/publications/gsp-060801.pdf).\n", - "1. Why is the second derivative much more expensive to compute than the first derivative?\n", - "1. Derive the head gradient relationship for the chain rule. If you get stuck, use the [\"Chain rule\" article on Wikipedia](https://en.wikipedia.org/wiki/Chain_rule).\n", - "1. Assume $f(x) = \\sin(x)$. Plot $f(x)$ and $\\frac{df(x)}{dx}$ on a graph, where you computed the latter without any symbolic calculations, i.e. without exploiting that $f'(x) = \\cos(x)$.\n", - "\n" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.1" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} + ] +} \ No newline at end of file From 5bb4692e6e673ed657bfa8bcc85f4c42f33e550c Mon Sep 17 00:00:00 2001 From: Moistsoy <89026391+Moistsoy@users.noreply.github.com> Date: Sun, 3 Oct 2021 13:44:34 +0900 Subject: [PATCH 5/7] Created using Colaboratory --- .../Automatic_Differentiation.ipynb | 42 ++++--------------- 1 file changed, 9 insertions(+), 33 deletions(-) diff --git a/Ch04_The_Preliminaries_A_Crashcourse/Automatic_Differentiation.ipynb b/Ch04_The_Preliminaries_A_Crashcourse/Automatic_Differentiation.ipynb index 2f1175d..fba99c7 100644 --- a/Ch04_The_Preliminaries_A_Crashcourse/Automatic_Differentiation.ipynb +++ b/Ch04_The_Preliminaries_A_Crashcourse/Automatic_Differentiation.ipynb @@ -76,7 +76,7 @@ "base_uri": "https://localhost:8080/" }, "id": "VM0iXJZtNHwA", - "outputId": "21f44e49-5465-4a0d-d345-feafc62b02ce" + "outputId": "667d640c-a97e-487a-bb9e-b9910324bfb5" }, "source": [ "x = Variable(torch.arange(4, dtype=torch.float32).reshape((4, 1)), requires_grad=True)\n", @@ -123,7 +123,7 @@ "base_uri": "https://localhost:8080/" }, "id": "qlqJO2ANNHwC", - "outputId": "8bdc401a-01d2-4690-eefd-42dadec531f6" + "outputId": "f64abbc9-ec6b-4841-9146-fa4884360623" }, "source": [ "y = 2*torch.mm(x.t(),x)\n", @@ -176,7 +176,7 @@ "base_uri": "https://localhost:8080/" }, "id": "YN8SfEMzNHwF", - "outputId": "af4975b8-ea31-439c-e407-007e4342f0f5" + "outputId": "f29504cd-52c0-40d0-a310-cf13a99134f4" }, "source": [ "print(\"x.grad:\", x.grad)\n", @@ -194,7 +194,7 @@ " [ 8.],\n", " [12.]])\n", "x.grad_fn: None\n", - "y.grad_fn: \n" + "y.grad_fn: \n" ] } ] @@ -215,7 +215,7 @@ "base_uri": "https://localhost:8080/" }, "id": "jHvO1YEJNHwG", - "outputId": "4cc25e9b-9505-49fa-dbe9-b62f7bf5bdfc" + "outputId": "31a94d31-aca3-4a00-b04c-9ed7f4b9ccf1" }, "source": [ "print((x.grad - 4*x).norm().item() == 0)\n", @@ -302,30 +302,6 @@ "execution_count": 8, "outputs": [] }, - { - "cell_type": "code", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "4ghuYwNqSJ7t", - "outputId": "60982897-5e29-456f-f586-25f1165fd6f4" - }, - "source": [ - "a = torch.randn(size=(2,))\n", - "print(f(a))" - ], - "execution_count": 15, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "tensor([ -54457.0117, -175862.7812])\n" - ] - } - ] - }, { "cell_type": "markdown", "metadata": { @@ -342,18 +318,18 @@ "base_uri": "https://localhost:8080/" }, "id": "BwXGi43jNHwJ", - "outputId": "98a97206-6c25-47c4-9880-86c280eb1930" + "outputId": "b52efa80-abc2-47f4-e4bd-b6717acd3fbb" }, "source": [ "print(a.grad == (d / a))" ], - "execution_count": 9, + "execution_count": 30, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ - "tensor([True])\n" + "False\n" ] } ] @@ -436,7 +412,7 @@ "## Exercises\n", "\n", "1. In the control flow example where we calculate the derivative of `d` with respect to `a`, what would happen if we changed the variable `a` to a random vector or matrix. At this point, the result of the calculation `f(a)` is no longer a scalar. What happens to the result? How do we analyze this?\n", - "- They result would be vector or matrix\n", + "- They result would be vector or matrix. We can analyze it by externally compute gradient, such as autograd\n", "\n", "2. Redesign an example of finding the gradient of the control flow. Run and analyze the result.\n", "-\n", From 9fc4c15df718c3c40ed506d391937c8e9e4225b7 Mon Sep 17 00:00:00 2001 From: Moistsoy <89026391+Moistsoy@users.noreply.github.com> Date: Sun, 3 Oct 2021 17:45:21 +0900 Subject: [PATCH 6/7] Created using Colaboratory --- .../Automatic_Differentiation.ipynb | 144 +++++++++++++++--- 1 file changed, 124 insertions(+), 20 deletions(-) diff --git a/Ch04_The_Preliminaries_A_Crashcourse/Automatic_Differentiation.ipynb b/Ch04_The_Preliminaries_A_Crashcourse/Automatic_Differentiation.ipynb index fba99c7..3a438bd 100644 --- a/Ch04_The_Preliminaries_A_Crashcourse/Automatic_Differentiation.ipynb +++ b/Ch04_The_Preliminaries_A_Crashcourse/Automatic_Differentiation.ipynb @@ -73,10 +73,11 @@ "cell_type": "code", "metadata": { "colab": { - "base_uri": "https://localhost:8080/" + "base_uri": "https://localhost:8080/", + "height": 0 }, "id": "VM0iXJZtNHwA", - "outputId": "667d640c-a97e-487a-bb9e-b9910324bfb5" + "outputId": "d59e329f-5e69-4eb3-c061-51169a69a245" }, "source": [ "x = Variable(torch.arange(4, dtype=torch.float32).reshape((4, 1)), requires_grad=True)\n", @@ -120,10 +121,11 @@ "cell_type": "code", "metadata": { "colab": { - "base_uri": "https://localhost:8080/" + "base_uri": "https://localhost:8080/", + "height": 0 }, "id": "qlqJO2ANNHwC", - "outputId": "f64abbc9-ec6b-4841-9146-fa4884360623" + "outputId": "f8161cf1-9c10-40c7-a894-8cecee6fa75b" }, "source": [ "y = 2*torch.mm(x.t(),x)\n", @@ -173,10 +175,11 @@ "cell_type": "code", "metadata": { "colab": { - "base_uri": "https://localhost:8080/" + "base_uri": "https://localhost:8080/", + "height": 0 }, "id": "YN8SfEMzNHwF", - "outputId": "f29504cd-52c0-40d0-a310-cf13a99134f4" + "outputId": "f7a39e1a-c5fe-4ff3-c662-daf902a5b3d5" }, "source": [ "print(\"x.grad:\", x.grad)\n", @@ -194,7 +197,7 @@ " [ 8.],\n", " [12.]])\n", "x.grad_fn: None\n", - "y.grad_fn: \n" + "y.grad_fn: \n" ] } ] @@ -212,10 +215,11 @@ "cell_type": "code", "metadata": { "colab": { - "base_uri": "https://localhost:8080/" + "base_uri": "https://localhost:8080/", + "height": 0 }, "id": "jHvO1YEJNHwG", - "outputId": "31a94d31-aca3-4a00-b04c-9ed7f4b9ccf1" + "outputId": "d5dcdeab-c21e-4c5f-a6ce-d0109853b2dc" }, "source": [ "print((x.grad - 4*x).norm().item() == 0)\n", @@ -315,21 +319,22 @@ "cell_type": "code", "metadata": { "colab": { - "base_uri": "https://localhost:8080/" + "base_uri": "https://localhost:8080/", + "height": 0 }, "id": "BwXGi43jNHwJ", - "outputId": "b52efa80-abc2-47f4-e4bd-b6717acd3fbb" + "outputId": "2324c6d5-6709-45c7-9562-a54fa8bcfec0" }, "source": [ "print(a.grad == (d / a))" ], - "execution_count": 30, + "execution_count": 9, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ - "False\n" + "tensor([True])\n" ] } ] @@ -362,10 +367,11 @@ "metadata": { "scrolled": true, "colab": { - "base_uri": "https://localhost:8080/" + "base_uri": "https://localhost:8080/", + "height": 0 }, "id": "HZvGe9O-NHwJ", - "outputId": "d49aa149-3f78-46a9-d327-a4a047715bf6" + "outputId": "61e933e6-f4e9-4e74-c18d-1d46d9e396d9" }, "source": [ "x = Variable(torch.tensor([[0.],[1.],[2.],[3.]]), requires_grad=True)\n", @@ -412,10 +418,10 @@ "## Exercises\n", "\n", "1. In the control flow example where we calculate the derivative of `d` with respect to `a`, what would happen if we changed the variable `a` to a random vector or matrix. At this point, the result of the calculation `f(a)` is no longer a scalar. What happens to the result? How do we analyze this?\n", - "- They result would be vector or matrix. We can analyze it by externally compute gradient, such as autograd\n", + "- They result would be vector or matrix. We can analyze it by externally compute gradient.\n", "\n", "2. Redesign an example of finding the gradient of the control flow. Run and analyze the result.\n", - "-\n", + "- Check the code below\n", "\n", "3. In a second-price auction (such as in eBay or in computational advertising), the winning bidder pays the second-highest price. Compute the gradient of the final price with respect to the winning bidder's bid using `autograd`. What does the result tell you about the mechanism? If you are curious to learn more about second-price auctions, check out this paper by [Edelman, Ostrovski and Schwartz, 2005](https://www.benedelman.org/publications/gsp-060801.pdf).\n", "-\n", @@ -427,7 +433,7 @@ "- $\\text{head gradient} = \\frac{dz}{dy} = \\frac{dz}{dx}\\frac{dx}{dy}$\n", "\n", "6. Assume $f(x) = \\sin(x)$. Plot $f(x)$ and $\\frac{df(x)}{dx}$ on a graph, where you computed the latter without any symbolic calculations, i.e. without exploiting that $f'(x) = \\cos(x)$.\n", - "\n" + "- Check the code below\n" ] }, { @@ -436,10 +442,108 @@ "id": "aMYeHC8lSD1m" }, "source": [ - "" + "def f0(a):\n", + " b = a * a + a\n", + " if b.sum().item() > 0:\n", + " c = b\n", + " else:\n", + " c = 100 * b\n", + " return c\n" ], - "execution_count": null, + "execution_count": 11, "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "PMjCEITmOxfJ" + }, + "source": [ + "$d = 2a^2 + a\\\\\\text{a.grad} = 2a + 1$" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "LB9ZyUBSMpL4", + "outputId": "17b03c9e-0a38-4e4e-da7c-fac538fab76a", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 0 + } + }, + "source": [ + "a = torch.randn(size=(1,))\n", + "a.requires_grad=True\n", + "d = f0(a)\n", + "d.backward()\n", + "print(a)\n", + "print(d)\n", + "print(a.grad)" + ], + "execution_count": 12, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "tensor([0.3273], requires_grad=True)\n", + "tensor([0.4344], grad_fn=)\n", + "tensor([1.6546])\n" + ] + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "bwVLRMdXM0L_", + "outputId": "fecfa090-a0af-4043-ee68-f4f5a88dba86", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 334 + } + }, + "source": [ + "import numpy as np\n", + "from matplotlib import pyplot as plt \n", + "\n", + "x = np.linspace(-np.pi, np.pi, 1000)\n", + "X = torch.tensor(x, requires_grad=True)\n", + "y = torch.sin(X).sum().backward()\n", + "\n", + "plt.plot(X.detach().numpy(), np.sin(x))\n", + "plt.plot(X.detach().numpy(), X.grad)\n", + "plt.axis('equal')\n" + ], + "execution_count": 13, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "(-3.4557519189487724,\n", + " 3.4557519189487724,\n", + " -1.0999999381914298,\n", + " 1.0999987020200273)" + ] + }, + "metadata": {}, + "execution_count": 13 + }, + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3hU1brH8e87aZSEGkILTem9hKZ4BQFF9Ih0bCCi2BDbOfZ27P0oNkSKgCgdAQHpikgNvUPoIJDQCSFtZt0/9ngPV1NIZjJ7JvN+nmeeTNmZ9U7Kb/asvfZaYoxBKaVU4eewuwCllFK+oYGvlFJBQgNfKaWChAa+UkoFCQ18pZQKEqF2F5CT6OhoU716dbvLUEqpgLFu3bqTxphyWT3m14FfvXp14uPj7S5DKaUChogczO4x7dJRSqkgoYGvlFJBQgNfKaWChAa+UkoFCQ18pZQKEhr4SikVJDTwlVIqSHgc+CJSRUSWish2EdkmIo9nsY2IyDARSRCRzSLS3NN2lVJK5Y03TrzKBJ42xqwXkShgnYgsNMZsv2ybm4Fa7ktr4Cv3V6WUUj7i8R6+MeaYMWa9+/oFYAdQ+S+bdQPGGcsqoJSIVPS0baWUUlfOq334IlIdaAas/stDlYHDl90+wt/fFP58jsEiEi8i8UlJSd4sTymlgprXAl9EIoFpwBPGmPP5fR5jzAhjTJwxJq5cuSzn/1FKKZUPXgl8EQnDCvsJxpjpWWxyFKhy2e1Y931KKaV8xBujdAQYBewwxnyczWazgP7u0TptgHPGmGOetq2UUurKeWOUzrXAPcAWEdnovu8FoCqAMWY4MBfoCiQAKcBAL7SrlFIqDzwOfGPMckBy2cYAj3rallJKqfzTM22VUipIaOArpVSQ0MBXSqkgoYGvlFJBQgNfKaWChAa+UkoFCQ18pZQKEhr4SikVJDTwlVIqSGjgK6VUkNDAV0qpIKGBr5RSQUIDXymlgoQGvlJKBQkNfKWUChLeWuJwtIgkisjWbB5vLyLnRGSj+/KKN9pVSil15byx4hXAt8DnwLgctvnNGHOrl9pTSimVR17ZwzfGLANOe+O5lFJKFQxf9uG3FZFNIjJPRBpkt5GIDBaReBGJT0pK8mF5SilVuPkq8NcD1YwxTYDPgB+z29AYM8IYE2eMiStXrpyPylNKqcLPJ4FvjDlvjEl2X58LhIlItC/aVkopZfFJ4ItIBRER9/VW7nZP+aJtpZRSFq+M0hGRH4D2QLSIHAFeBcIAjDHDgV7AwyKSCVwC+hljjDfaVkopdWW8EvjGmDtyefxzrGGbSimlbKJn2iqlVJDQwFdKqSChga+UUkFCA18ppYKEBr5SSgUJDXyllAoSGvhKKRUkNPCVUipIaOArpVSQ0MBXSqkgoYGvlFJBQgNfKaWChAa+UkoFCQ18pZQKEhr4SikVJLwS+CIyWkQSRWRrNo+LiAwTkQQR2Swizb3RrlJKqSvnrT38b4EuOTx+M1DLfRkMfOWldpVSSl0hb614tUxEquewSTdgnHtZw1UiUkpEKhpjjnmjfaV8IjMd0pMhMw0coRASCiHhEFYMrCWblZsxhnSni7RMF2kZLlzGUCQshKJhIYSFCKI/L1t4JfCvQGXg8GW3j7jv08BX/sOZASd3w/EtkLgdzhyEc0esy6XT4EzP+vtCi0DxctalVBWIrm1dytWFmPrWG0MhlOl0kZCUzO4TySScuEBCUjLHz6Vy4nwaSclppGe6svy+EIcQHRlOhZJFqViiCFXLFqNO+SjqVoyiZkwkEaEhPn4lwcPv/hJFZDBWtw9Vq1a1uRpVqGWkwuHVcOA32P8b/LH+v6EeEmGFd8lYqNUJipWFiCgIj4LQcHA5rTcIZxqknIKLJyE5EY5vhR2zwbjDLqw4VG4OVVrD1R2sryFh9r1mD2Q6XcQfPMOqfadYe+A0Gw6dJSXdCYBDoFrZ4lQqVYRWNcoQExVBiaJhRIQ6iAh14HAIqRkuUjOcXEzLJOlCGsfPp7In8QJLdiaS7rR+XmEhQpPYUrSqUYZWNcrQ5qqyFAnTNwBvEauXxQtPZHXp/GSMaZjFY18DvxhjfnDf3gW0z61LJy4uzsTHx3ulPqUASD0HuxfAjlmQsAgyUkAcUKk5VGsLFZpAhUZQtmb+98wz0+D0fuuTwpE1cHiNdd04IaKkFfx1b4W6XSG8uHdfn5elpGeyeEcii3acYOnORM6nZiIC9SqUoGX10jSrWpo6FaKoEV0838Gc4XRx4ORFdh6/wNaj51i9/zRbjp7D6TIUDQvhulrR3NigAp3qxVCqWLiXX2HhIyLrjDFxWT7mo8C/BRgCdAVaA8OMMa1ye04NfOUVLhfsWwrrx8GuudZefGQFqHcr1OwM1a6BIiUKtobU87D/V9g9H/YshOTj1t5/3VugSV+46gZw+McoaZfLsObAaaatO8LcLce4mO6kTPFwbqgbQ6d6MVxTM5oSRQr2U8rFtEziD55h8Y4TLNh2guPnUwkPcdCpfgy9W1ThulrRhIb4x8/L3xR44IvID0B7IBo4AbwKhAEYY4aLdYTmc6yRPCnAQGNMrkmuga88cvEUxI+ygv7cYShaGhr3hYY9oXKcfQHrcsGhlbBlMmz7EVLPQuka0OoBaHoXFC1lS1nJaZlMiT/MtysOcPBUCsXDQ+jaqCI9msfSqkYZQhz2HGg1xrDl6Dl+3PAHP248yumL6cRERXBHq6rc07Ya0ZERttTlr3yyh18QNPBVvpzaCyu/gI3fQ+YluKo9NB9g7U2H+lk4ZKbBzp9gzTfWm0BYMSv0r33cOobgA3+cvcSo5fuZvPYwF9IyaVGtNHe3qUqXBhUpGu5f/efpmS6W7Exk0tpDLN2VRHiog57NKzOoXQ1qxkTZXZ5f0MBXweHUXvjlHdg6zRo22bgPtH0MYuraXdmVObYJVo+AzZOs203vhOuegtLVC6a5c5f4culeJq49hDFwS+OKDLy2Bk2r2PMJI6/2JiUzavl+pq07QrrTxS2NKvJEp9rUjIm0uzRbaeCrwu38H/Dre7B+vLUH3+oBaPMIRFWwu7L8OXsYfv8U1o+1RgO1GADtX4DIcl55+lPJaXy2JIHvVx/CYOgdV4VHO9SkcqmiXnl+Xzt9MZ1Ry/cx5vcDpGY4ub1ZZZ7oWJuqZYvZXZotNPBV4ZR+EZZ9CKu+tIIxbiBc90+IKm93Zd5x/g/47SOIH2N19Vz3pPVGFpa/YE7PdDF2xQGGLd5DSoaTPnGxPNqhJrGlC0cwnkpOY/ivexm38iDGwKDravBoh5pERvjd6PMCpYGvChdjYNsMWPASnD8KjfrADS8WWNeH7U7ugYWvWCOMSsRCl7eh3m1XfHavMYaF20/w9twdHDiVQoc65XjxlvqFtuvjxPlU3vt5J9PXHyUmKoJnu9Sle7PKOGw66OxrGviq8EjaBXOetk6WqtAYun4IVVvbXZVv7P8Nfn4eTmyB2l2g6wdQKueTE4+fS+XlmVtZuP0ENWMieemWerSvE+Ojgu214dAZXpu9nU2Hz9K8aine69mYWuUL/4FdDXwV+JyZsGIY/PKu1aXR8RVocS84/GsUSYFzZsLq4bD0Let2++etbp6/nCTmchkmrDnEe/N2kuly8WSn2tzXrgZhQTZ23eUyTN9wlLfmbCc5LZNHO9TkkfY1CQ8tvD8HDXwV2E5sgx8fgWMboX43a68+Mjj2UrN19jDM/RfsnmedU9D9a4iuCcC+pGSembqZ+INnuLZmWd7u3ohqZf37jN6CdjI5jddnb2fWpj+oXT6S93s1CZjRSHmlga8Ck8sJyz+GX96DIiXhlo+gwe12V+U/jIFt0+GnpyAzDdP5dX4wN/LGnJ2Ehzp46ZZ69GoRqzNTXmbxjhO89ONWEi+kMfSGWjza4epCd8auBr4KPOeOwPTBcPB3aNDD2qsvXtbuqvzT+WOkz3iU8P2L+c3ZkCmVn+eFOzpRoWQRuyvzS+cuZfDKzK3M3PgHzauW4pO+zQrVEM6cAr9wvbWpwmHHbPjqWutEpO5fQ+8xGvY5+OVYCNcceoiXnffTJnwvn557jAonltldlt8qWTSMT/s149N+TdmTmMzNny5j6rojdpflExr4yn9kXIKfnoRJd1tDLB9cBk362V2V33K6DB/O38W9Y9YSHVWEux59lbCHlyMlKsH3vWHhq9YUzipL3ZpWZt7j19Ggckn+OWUT/5qyidQMp91lFSjt0lH+4cxBK+iPb4ZrhsINL1vzzqssnUxOY+gPG1ix9xR94mJ5vVvD/05PnHHJGr65bgxUaQO9Rlnz+qssOV2GTxbt5rMlCdSrWIKv7mpO9ejAPcitffjKvyUshmmDrFkke4yAOjktj6zWHjjNkO/XczYlgzdub0ifuGwmWdsyFWY/bi3D2PtbuOp6n9YZaJbuTOSJSRtxuQwf9mnCTQ0Cc2oO7cNX/skYa+qA73pCVCUYvFTDPhfjVx6g34hVFA0LYcYj12Yf9gCNesHgX60hrOO7w+qvrZ+5ylKHujHMGdqOq8oV58Hx63jv5524XIXr56WBr+yRdsHqwln8OjTsAfcvhLJX212V38pwunhxxhZenrmN9rXLMeuxdtSvdAWLtkTXhPsXWWfmznsGZg6xpmRWWYotXYzJD7XljlZV+eqXvQwev47ktEy7y/IaDXzle+eOwOgusGse3PQO9Bzl90v92enMxXTuGbWaCasP8dD1VzOif1zeVpyKiIK+38H1z8LG7+DbW+DC8YIrOMBFhIbwdveGvN6tAUt3JdLzyxUcPp1id1le4ZXAF5EuIrJLRBJE5LksHr9XRJJEZKP7cr832lUB6Og6+OYGOHsI7poCbR+54knAgtHuExfo9sXvrD90lv/0bcJzN9fN38pTDgd0eAH6jIMT22FEe2vYq8qSiNC/bXXGDmzFsXOXuO3z5azad8rusjzmceCLSAjwBXAzUB+4Q0TqZ7HpJGNMU/dlpKftqgC07UcYcwuEFoFBC6FmR7sr8mu/7Eqkx5crSEl3MnFwG7o388JIm/rdYNACkBAYfbO1oLvKVrta0cwc0o7SxcO5e+RqJscftrskj3hjD78VkGCM2WeMSQcmAt288LyqsPjz4OyUAVCxMTywJHBWobLJpLWHGDQ2nqplijFryLU0r1rae09eoaHVrx9dE37oC2t1/ysnNaKLM+ORa2l7dVmembqZTxbtxp9HN+bEG4FfGbj8be+I+76/6ikim0VkqohkO7RARAaLSLyIxCclJXmhPGUrZybMHmodnG3UG/rPguLRdlflt4wx/Gfhbp6dtoVrri7L5IfaUqkgVqIqURHunQs1O1vTTS94yRoWq7JUsmgYo+9tSc/msXyyaA/PTttMhjPwfl6+Omg7G6hujGkMLATGZrehMWaEMSbOGBNXrpx3lnRTNsm4BJPvgfXjrJWoenwDYTq/S3YynC6em7aFTxfvoWfzWEbf27JgV2uKiIR+30PL+2HFZzD1XshILbj2AlxYiIMPezdm6A01mRx/hPvHxnMxwEbweCPwjwKX77HHuu/7P8aYU8aYP8eCjQRaeKFd5c9STsO4262ROF0/hI4v68HZHFxMy+SBcfFMij/MYzfU5MPejX0zd31IqPX7ufFN2D7TOici9VzBtxugRISnbqzDuz0asTzhJH1HrCTxQuC8SXrjL2otUEtEaohIONAPmHX5BiJS8bKbtwE7vNCu8lfnjsKYm+GP9dbEZ60esLsiv3YyOY1+I1axbHcSb3dvxNM31vHtlMYicM1j0GMkHF5lDdtMTvRd+wGoX6uqjBwQx76ki/T4cgUHTl60u6Qr4nHgG2MygSHAfKwgn2yM2SYir4vIbe7NhorINhHZBAwF7vW0XeWnEnfCqButBbjvngYNuttdkV/74+wl+gxfyZ7EC3zTP447W+e8ZGGBatwb7pgIJxNg9E1w5oB9tQSADnVimDi4DSnpTnoNX8mOY+ftLilXOpeO8p7Da2FCLwiNgLumWiNyVLb2JSVzz6g1nL+UweiBLWlZvYzdJVkOr4EJva3hs/dMh/IN7K7IryUkXuCeUWu4mJbJmIEtaVHN3t+jzqWjCt7BlTD+diha2hrnrWGfo21/nKPP1ytJzXDyw+A2/hP2AFVawX0/W109Y262frcqWzVjopjyUFvKRkZw98g1LNvtv6MLNfCV5w4sd0+AVhEGzrPmslfZij9wmn4jVhEW4mDyQ21pWLmk3SX9XUw96427eDlr4rWExXZX5NdiSxdj8oNtqR5dnEFj1zJ3yzG7S8qSBr7yzL5f4Lte1nzr986xxnerbC3bncQ9o9YQHRnBlIfacnW5SLtLyl6pqjDwZ2tSux/6WSOuVLbKRUUwcXAbmsSWYsj365m45pDdJf2NBr7Kv4TF8H1fKFPDCvuo8nZX5NfmbTnGoLFrqR5dnMkPtiW2dACsoxpZDgbMtvrxJ90N22bYXZFfK1k0jPGDWnNdrXI8N30L3/6+3+6S/h8NfJU/uxfAD3dA2Vow4CcrGFS2Zm48yqPfr6dR5ZJMfKAN5aIi7C7pyhUrA/1nQuU4mHofbJpod0V+rWh4CN/0j+PG+uV5bfZ2Rizba3dJ/0cDX+Xdrnkw6S5rPpwBs3SB8VxMW3eEJydtpGX1Mowf1JqSxfIwtbG/KFLSGrFTvR3MeAjix9hdkV8LD3XwxV3NuaVxRd6eu5PPl+yxuyQACvC8bVUo7ZgNUwZChUZWABT14qRehdDktYd5dvpm2l5VlpED4igWHsD/cuHF4c7JMLk//PQEZKZCm4ftrspvhYU4+LRvU8JDHHy4YDfpTsOTnWr59qS6vwjgvz7lc9t+tNaerdTMOqmqiB+OLvEj368+xAsztnBdrWi+6R/330XGA1lYUeg7AabdBz8/Z82XdN1Tdlflt0JDHHzYuwlhIcKwxXtIz3TxbBcfn0l9eT22tKoCz5apMH0wxLa0Fi4pcgXL6wWx8SsPWMsR1inH8LtbFI6w/1NoOPT6Fn58CBb/G1yZcP0zdlflt0Icwrs9GhMe6mD4r3tJz3Tx8q31bAl9DXyVu02TrH/uqm2tj/QRfjyU0A+M+X0//569nU71YvjiruZEhBaisP9TSCh0/9paSGXpW9aaB+2ftbsqv+VwCG90a0hYiIPRv+8n3enk9dsa4sjP6mUe0MBXOdswAWY+CjWus+ZZ0bVnczTyt328OWcHN9Yvz+d3Nic8tBCPi3CEwO1fWmfk/vI2GBd0eN7uqvyWiPDKrfUJD3Xw9a/7yHQa3u7eyKehr4GvsrduLMx+HK5qb82bHh4A48ZtNPzXvbw7byddG1Xg037NfDO9sd0cIdDtCxAH/PouYKD98zoVdjZEhOe61CXM4eDzpQkAPg19DXyVtbWjYM5TULOTdZBOFy7J0edL9vDhgt38o0kl/tOnCaHBEPZ/coTAbZ8DAr++Z3XvdHhBQz8bIsLTN9bGITBsSQLGwDs9fBP6Gvjq71aPgHn/gtpdoM84a/ZLla1PFu3mk0V76N6sMh/0ahxcYf8nhwNu+8wK+WXvAwY6vKihnw0R4cnOtUGs0TsGw7s9Ghd46Gvgq/9v5Rcw/wWoeyv0GmONyFBZMsbwn0V7GLZ4D71axPJez8aE+PggnF9xOOAfw9yh/4HVp3+DrnSWHRHhqc61EeDTxXswBt7rWbCh75XAF5EuwKdACDDSGPPuXx6PAMZhLW14CuhrjDngjbaVF/3+KSx8BerdBr1GQ0gAnhHqI8YYPl64m8+WJNA3rorPPpL7PYcDbv3U6tP/7SOre6fjKxr6OXiyc23AHfpQoDsOHge+iIQAXwCdgSPAWhGZZYzZftlmg4AzxpiaItIPeA/o62nbyouWfQhL3oAGPaDHCA37HBhj+GjBbj5fmkC/llV8PtLC7zkccMt/AIHlH1t7+p1e09DPwZOdayMCnyyy9vTf71Uwoe+NPfxWQIIxZh+AiEwEugGXB3434DX39anA5yIipgCW28pwunh68iZua1KJTvV19sYr8st71rC6Rn3g9q+sMdYqS8YYPlywiy+W7uWOVlV463YN+yw5HHDLx9ae/u+fAAY6/VtDPwdPdKqNIPxn0W4Mhg96NfF66Hvj6FJl4PBlt4+478tyG/cauOeALGfcEpHBIhIvIvFJSXlfOSY1w8nBUxd5eMI6Fmw7nufvDyrGwJK3rLBvehd0H65hnwNjDB/M/zPsq2rY58bhgFs+gpb3u7sLX7b+5lS2Hu9Ui6c61+bImUukZ7q8/vx+N5zAGDPCGBNnjIkrVy7vU+5GFQlj/P2taVCpJI9MWM/PW/1z5RnbGWOdFr/sfWje3xpW5yiEZ4R6iTGG9+fv4stf9nJn66q8dbvvz5IMSCLQ9UNoNRhWfAbzX9TQz8XQjrX4blBrioZ7///RG4F/FKhy2e1Y931ZbiMioUBJrIO3BaJEkTDGD2pF49iSPPr9Br9dbsw2xlh7W8v/A3H3WQfZHH733u83jDG8+/NOvvplL3e1rsqb3TTs80QEbn4fWj8Mq76wJl3T0M9RQZ2h7Y1nXQvUEpEaIhIO9ANm/WWbWcAA9/VewJKC6L+/XFSRMMYNak2zKqV47IcNzN70R0E2FziMgZ+ft/a2Wg22+lk17LNljOHdeTv5+td93N2mKm/qnn3+iECXd6DtEFg9HOb+S0PfBh532BpjMkVkCDAfa1jmaGPMNhF5HYg3xswCRgHjRSQBOI31plDgIiNC+fa+Vtw3Zi2PT9yAyxi6Nf3r4YUg4nLBvGdg7TfQ5hG46W09iJYDYwzvzNvJiGX7uKdNNV7v1sDWucwDngjc+KZ1IHfFMDBO6PqR7nD4kFeO0Blj5gJz/3LfK5ddTwV6e6OtvIqMCGXMwJbc9+1anpy0EZcxdG8Wa0cp9nK5rKkS1o2Ba4ZC59c17HNgjOHtuTv45rf99G9bjX/fpmHvFSLW354j1Bqy6crULkUfCoohGcXdoT/o23iemrwJlwt6tgii0He5YPZQ2DAerntaz37MhTGGN+fsYNTy/QxoW43XNOy9S8Q6GcsRYp2R63LBbcN00IAPBEXgAxQLD2X0vS15YFw8/5y6Cacx9Imrkvs3BjqXE2YOgU3fw/XP6kyGuTDG8MZPOxj9+37uvaY6r/6jvoZ9QRBxz7UTYs2yaZzWrJsa+gUqaAIfrNXkRw6I44Fx8Tw7bTPGGPq2rGp3WQXHmQEzHoSt06x/Ll2VKEfGGF7/aTtjfj/AwGur88qtGvYFSsSaP9/hXkTF5dQT/wpY0P1ki4SF8E3/OB4cv45np23B6YI7WxfC0M9Mt9Yd3THb6jO99nG7K/JrLpfhpZlb+X71Ie67toZtS9AFpeufsQ7kLnnD2tPvPkJDv4AE5U+1SFgIX9/Tgoe/W8cLM7bgNIZ72lSzuyzvyUiFyf1hz3z3+OcH7a7Irzldhmembmba+iM80v5q/nWTfYtMB63/+ad1IHfRq9aefs+ROp9TAQjKwAcr9Iff04JHJ6zn5R+3Yoyhf9vqdpflufQUmHQX7F0Kt34CcQPtrsivZThdPDlpIz9tPsZTnWvz2A01Nezt0u4Jq3tnwUvWhGs6Y6vXBfVYqIjQEL68qwWd65fnlZnbGPnbPrtL8kxaMnzfxwr7bl9o2OciLdPJoxPW89PmYzx/c12GdqylYW+3ax6Dm96BHbNg8gDITLO7okIlqAMfrFOYv7izObc0qsibc3bw8cLdFPBJwAUj9Rx81wMOroAe30Czu+yuyK+lZjh5cPw6Fmw/wWv/qM+D119td0nqT20fsebf2TXH2oFJv2h3RYVG0HbpXC481MGwO5pRLDyEYYv3kJyaGVgH7S6dgfE94Phm6D0G6nezuyK/lpKeyf1j41m57xTv9GjEHa0K4UH7QNfqAQgvDjMfhXG3w11ToGgpu6sKeBr4biEO4b2ejYksEsro3/dzMS2Tt3s08v8l6y4ch/Hd4VQC9P0O6txsd0V+7XxqBoO+Xcu6g2f4qHcTejQPohPwAk3TO63QnzoIvr0V7pkBkXmfQVf9V9B36VzO4RBeubU+QzvWYlL8YYZO3FAgc1J7zel9MOpGOHvI2gPSsM9R4oVU+n69ig2HzjLsjmYa9oGgfje4c6K1QzOmC5w7YndFAU0D/y/+XFj4ha51mbP5GIPHx5Oa4bS7rL87sQ1Gd4G0CzBgFlzV3u6K/Nrh0yn0Hr6SAycvMnJAHLc2rmR3SepK1exk7d0nJ1p/86f22l1RwNLAz8bg/7mat7s34tfdSfQfvYYLqRl2l/Rfh1bDmJut09IHzoPKLeyuyK/tPH6enl+t4GxKBt/d35r2dWLsLknlVbW2MGA2ZKRYoX98q90VBSQN/Bzc2boqn/RtyvqDZ+jz9SoSz6faXRIkLILxt0OxaBg0H2Lq2l2RX1t38DR9hq9EBKY81JYW1UrbXZLKr0pNrR0cRyh82xUOLLe7ooCjgZ+Lbk0rM3JAHAdPXaT7lytISEy2r5gtU+H7flD2arjvZyilo0tysnRnIneNXE3ZyAimPnQNtctH2V2S8lS5OtaOTmR5a7DCthl2VxRQPAp8ESkjIgtFZI/7a5a7TyLiFJGN7stfV8Pye+3rxDBpcFvSMp30Gr6CdQdP+7YAY6zlCKcNgiqt4N45EKndEjmZEn+YB8bFc3W5SKY81JYqZYrZXZLyllJV4b75UKk5TBkIq76yu6KA4eke/nPAYmNMLWCx+3ZWLhljmrovt3nYpi0axZZk+sPXUqpoGHd+s5r52477pmFnprVwyaLXoGFP6+BVkZK+aTsAGWP4eOFu/jV1M22uKsvEwW2IjoywuyzlbcXKQP8foe4t1hq5C16y5tVXOfI08LsBY93XxwK3e/h8fq1q2WJMe/ga6lYswcPfreO7VQcLtsG0ZJh4J8SPhnZPQo+REKrhlZ30TBdPT97EsMV76N0iljEDWxJVROdiKbTCikKfcdDyAWuN5un361QMufA08MsbY465rx8HymezXRERiReRVSKS45uCiAx2bxuflJTkYXneVzYygh8esEZ6vPTjVt78aTtOVwFMxXDhBHx7CyQstBYa7/SaLgOXg3MpGQwYvYbpG47ydOfavN+rMWEh+kn1NvkAABINSURBVPMq9Bwh0PUD6Piqte7D+O5w8ZTdVfktyW3eGBFZBFTI4qEXgbHGmFKXbXvGGPO3fnwRqWyMOSoiVwFLgI7GmFwH08bFxZn4+PjcNrNFptPFm3N28O2KA3SoU45hdzTz3t7k8S3ww52QchJ6jYE6XbzzvIXU4dMpDPx2LQdPXeT9Xo2Dc81iBZunWFMxlKgId0wK2hFsIrLOGBOX1WO57gIZYzoZYxpmcZkJnBCRiu5GKgKJ2TzHUffXfcAvQLN8vha/ERri4LXbGvDm7Q1ZtuckPb9aweHTKZ4/8faZ1tmzrkwYOFfDPhcrEk5y2+fLSTyfytj7WmnYB7PGva0BDekpMKoz7Flod0V+x9PPvLOAAe7rA4CZf91AREqLSIT7ejRwLbDdw3b9xt1tqjHuvlYcP5dKty9+Z+2BfI7gcbngl3ethUti6sPgpVAp4N8XC4wxhrErDnDP6DWUjYxg5pB2XHN1tN1lKbtVaWn975SuZs20ufILa5SbAjwP/HeBziKyB+jkvo2IxInISPc29YB4EdkELAXeNcYUmsAHuLZmND8++ucInlVMWH0wb1Msp1+EKQPgl3egyR3WXkpUVr1oCqx57J+btoVXZ22jQ51yzHjkGmpEF7e7LOUvSsZawzbrdIX5L8CsIdYqcCr3Pnw7+XMfflbOpWQwdOIGft2dRI/mlXnr9kYUDQ/J+ZtOJlh79Uk7oPMb0PZRa3FnlaUT51N5ZMJ61h08w5AONXmqc20c/j6jqbKHy2XtRC17Hyo2sUb0lK5ud1UFLqc+fA18L3O5DMOW7OHTxXuoUz6K4Xe3oHp2e59bp8OsxyAkHHp+Y00SpbL1254knpi4kZR0Jx/0bqwToKkrs2seTH8QBGtxoNo32V1RgfLooK3KG4dDeKJTbcbc25Lj51P5x+fLWfDXk7Qy02Duv2DqQKu//qHfNOxz4HQZPl6wi/6j11A2MpzZj12rYa+uXJ2b4cFfoZS7X3/xG9ZC6UFI9/AL0OHTKTwyYT1bjp6jf9tqvNC1HkUuHIKp98Ef66HNo9b4+tBwu0v1W4kXUnn8h42s3HeKXi1ieb1bA4qF67o9Kh8yLlk7WhvGQ9VroMfXhXI+Ku3SsVFappMPft7FyOX7eKz0Kp7IHE2IIxS6fQ71A3KWCZ+Zv+04L0zfwsX0TN7o1pDecVXsLkkVBpsmwpx/gjjg1o+hUS+7K/Iq7dKxUURoCC91iCG+5rc8fekz1qZVY1LLibjq/sPu0vzWhdQM/jllEw+OX0f5EkWYNaSdhr3ynib9rG7UcnWsCQmnPwip5+2uyif0s3FB2zUPZg0lOvUsF69/lZEHrmXRopNM27uK93o21uGEf7Fq3ymenryJY+cuMaRDTYZ2rEV4qO6XKC8rU8OaW3/ZB9YongPL4db/QO0b7a6sQGmXTkG5cBzmPQvbf4SYBtBjBFRoiDGGKeuO8OZP20nLdPFU59oMaleD0CCf9+VcSgbv/ryTH9YconrZYnzUp6kuVqJ84/Baa6x+0k5o1Ae6vAvFy9pdVb5pH74vuVywYRwseAUyL8H/PAPXPv63A7Mnzqfy8o9bWbD9BI1jS/J290Y0rBx80x4bY5i9+Rivz97O6YtpDGpXgyc719YDs8q3MtPgt4+sS5GS0OU9q28/AM+J0cD3lSPx8PPzcGQNVGsH//gEomtlu7kxhjlbjvHarG2cuphOv5ZV+OeNdSgbJPO37zlxgTfm7GDZ7qSgftNTfuTENpg5xBpFV/UauPld66StAKKBX9DOHYXF/4bNk6B4DHR8BZredcXTGZ+7lMGwxXsYu+IARcNDeKJTbe5pU63Q9l2fSk7jk0V7+H7NIYqFh/BU59r0b1udED1jVvkDlxPWj4Mlb0DKaWgxAG54GYoHxlxNGvgFJeU0rPzcWmLN5bSmRbjuKYjI39qpCYkX+Pfs7fy25yRVyhTliY61ub1Z5UIThBfTMhm78gBfLd1LSoaTu1tX5fFOtSlTXM9DUH7o0hn49X1YMwJCi0LbR6DNI1C0VO7fayMNfG+7dMaahW/VcEhPhoY9rAUYSlfz+KmNMfyyK4kPF+xi2x/nqRkTyZOdatOlYYWADf7ktEzGrTzAyN/2c/piOjfUjeGFrvWoGRNpd2lK5S5pFyx9y5q6vEhJuGYotH4w3zt2BU0D31vOHIA131gf99LOQ/1ucP1zUL6+15tyuQw/bzvOxwt3k5CYTLWyxbi/XQ16taiS+4RsfiLxfCoTVh9i7MoDnE3J4Pra5RjasZaOvlGB6Y+NsPRt2DMfIkpaXT2tH4KSle2u7P/RwPeEywUHl1sf63bOsc7Oq98N2j0FFRoWePNOl2H+tuN8vWwfmw6fpXSxMPq1qkqfuCp+OYbfGMOGw2cZu+IAc7ccI8Np6Fg3hsc61qJpFf/+KKzUFTmyDlYMgx2zrDxo0ANa3AvVrvGLUT0a+Plxep91CvamH+DsIShaGloMhJb32/KObowh/uAZvlm2j8U7E3G6DK2ql6F3XCw3NaxACZsX6z5yJoWZG/9gxoajJCQmExURSq+4WPq3re6Xb0xKeezMQVg9HNaPh/QLULqGNVijSV9b5+gpsMAXkd7Aa1iLnLQyxmSZziLSBfgUCAFGGmPevZLn92ngu1xwbKN1ZuyuuXBiKyBwVXtoeifUvRXCi/mmllycOJ/K9PVHmRJ/mH0nLxIWIrS9OpqbGpSnY93yVChZpMBrMMaw49gFluw8wZKdiaw/dBaAltVL071ZLLc1rURkhI6lV0Eg/SLsmA0bvoMDv1n3VWxiZUadrlC+gU/3/Asy8OsBLuBr4J9ZBb6IhAC7gc7AEWAtcMeVrHpVoIGfeh4Sd1hj5g+ugEMrrYOx4oAqbaBuV+ujmp/1z13OGMP6Q2dZsO04P287zsFT1pq61csWo81VZWlZvQwNKpfg6nKRhHl4Ju+ldCe7T1wg/uAZ1h88w9oDp0m8kAZAk9iSdK5fnm5NK1OljH+8KSpli9P7ra6enXPg8BrAQLGyVndPtXZQuTnE1CvQA74F3qUjIr+QfeC3BV4zxtzkvv08gDHmndyeN9+Bf3gtZKRAZqo1JWraeTh/DC4cg3NHrFOozx3+7/ZlrrJOsqjeDmrdGJCnVRtj2H0imeUJJ1m59xRr9p/ifGomAGEhQs2YKKqWKUrFkkWpXKooZSPDKRYeQrHwUCJCHThdhnSni7RMF2cuppN0IY2k5DQOn04hISmZI2cu/d/SoJVLFaVFtdK0qxlN+7rliIkq+E8USgWcCycgYSEc+N06Dnj20H8fK10domtDicrWJaoCRERCWHEIKwphxSC2Rb6azSnwffGZuzJwWbpyBGhdoC2O/Yc1rcFfFYuGEpWgahuIcS8+UqlZoVg/VkSoUyGKOhWiGNSuBk6XYW9SMjuOnWfHsQvsOn6efUkXWb7nJBfTr2zxhxJFQqlUqihNYkvRq3kVapePpFnV0j7pMlIq4EWVh2Z3WxewdjaPbYbEbdYZvacS4Og6SDn19+8tHgP/2uP1knINfBFZBGSViC8aY2Z6uyARGQwMBqhaNZ8HPu6cCI5Q62SJsCLWx6fICkG10EiIQ6hdPora5aPo1vS/9xtjOJ+aydmUdC6mOUlJzyQt00VYiIOwECEsxEHp4uGULR5OkbDAGP6pVEAoGWtd6nb9//dnpELycetYQMYl66ujYP73cg18Y4yna+8dBS6fzDzWfV927Y0ARoDVpZOvFq9qn69vCwYiQsmiYZQsau+oHqWUW1gRny2u7ovJWtYCtUSkhoiEA/2AWT5oVyml1GU8CnwR6S4iR4C2wBwRme++v5KIzAUwxmQCQ4D5wA5gsjFmm2dlK6WUyiuPDtoaY2YAM7K4/w+g62W35wJzPWlLKaWUZwrn/LtKKaX+RgNfKaWChAa+UkoFCQ18pZQKEhr4SikVJDTwlVIqSGjgK6VUkNDAV0qpIKGBr5RSQUIDXymlgoQGvlJKBQkNfKWUChIa+EopFSQ08JVSKkho4CulVJDQwFdKqSDh6YpXvUVkm4i4RCQuh+0OiMgWEdkoIvGetKmUUip/PFrxCtgK9AC+voJtOxhjTnrYnlJKqXzydInDHQAi4p1qlFJKFRhf9eEbYIGIrBORwTltKCKDRSReROKTkpJ8VJ5SShV+ue7hi8gioEIWD71ojJl5he20M8YcFZEYYKGI7DTGLMtqQ2PMCGAEQFxcnLnC51dKKZWLXAPfGNPJ00aMMUfdXxNFZAbQCsgy8JVSShWMAu/SEZHiIhL153XgRqyDvUoppXzI02GZ3UXkCNAWmCMi8933VxKRue7NygPLRWQTsAaYY4z52ZN2lVJK5Z2no3RmADOyuP8PoKv7+j6giSftKKWU8pyeaauUUkFCA18ppYKEBr5SSgUJDXyllAoSGvhKKRUkNPCVUipIaOArpVSQ0MBXSqkgoYGvlFJBQgNfKaWChAa+UkoFCQ18pZQKEhr4SikVJDTwlVIqSGjgK6VUkPB0AZQPRGSniGwWkRkiUiqb7bqIyC4RSRCR5zxpUymlVP54uoe/EGhojGkM7Aae/+sGIhICfAHcDNQH7hCR+h62q5RSKo88CnxjzAJjTKb75iogNovNWgEJxph9xph0YCLQzZN2lVJK5Z03+/DvA+ZlcX9l4PBlt4+478uSiAwWkXgRiU9KSvJieUopFdxyXdNWRBYBFbJ46EVjzEz3Ni8CmcAETwsyxowARgDExcUZT59PKaWUJdfAN8Z0yulxEbkXuBXoaIzJKqCPAlUuux3rvk8ppZQPeTpKpwvwDHCbMSYlm83WArVEpIaIhAP9gFmetKuUUirvPO3D/xyIAhaKyEYRGQ4gIpVEZC6A+6DuEGA+sAOYbIzZ5mG7Siml8ijXLp2cGGNqZnP/H0DXy27PBeZ60pZSSinP6Jm2SikVJDTwlVIqSGjgK6VUkNDAV0qpIKGBr5RSQUIDXymlgoRkfXKsfxCRJOBgATx1NHCyAJ7XVwK9fgj816D12y/QX0NB1V/NGFMuqwf8OvALiojEG2Pi7K4jvwK9fgj816D12y/QX4Md9WuXjlJKBQkNfKWUChLBGvgj7C7AQ4FePwT+a9D67Rfor8Hn9QdlH75SSgWjYN3DV0qpoKOBr5RSQSIoA19E3hCRze45/BeISCW7a8orEflARHa6X8cMESlld015ISK9RWSbiLhEJGCG1olIFxHZJSIJIvKc3fXklYiMFpFEEdlqdy35ISJVRGSpiGx3//08bndNeSUiRURkjYhscr+Gf/us7WDswxeREsaY8+7rQ4H6xpiHbC4rT0TkRmCJMSZTRN4DMMY8a3NZV0xE6gEu4Gvgn8aYeJtLypWIhAC7gc7AEazV3O4wxmy3tbA8EJH/AZKBccaYhnbXk1ciUhGoaIxZLyJRwDrg9gD7HQhQ3BiTLCJhwHLgcWPMqoJuOyj38P8Me7fiQMC96xljFrhXEwNYhbVWcMAwxuwwxuyyu448agUkGGP2GWPSgYlAN5tryhNjzDLgtN115Jcx5pgxZr37+gWsVfQq21tV3hhLsvtmmPvikwwKysAHEJG3ROQwcBfwit31eOg+YJ7dRQSBysDhy24fIcDCpjARkepAM2C1vZXknYiEiMhGIBFYaIzxyWsotIEvIotEZGsWl24AxpgXjTFVgAlYa+76ndxeg3ubF4FMrNfhV66kfqXyQ0QigWnAE3/5xB4QjDFOY0xTrE/mrUTEJ91rHq1p68+MMZ2ucNMJWOvtvlqA5eRLbq9BRO4FbgU6Gj88GJOH30GgOApUuex2rPs+5UPufu9pwARjzHS76/GEMeasiCwFugAFfiC90O7h50REal12sxuw065a8ktEugDPALcZY1LsridIrAVqiUgNEQkH+gGzbK4pqLgPeI4CdhhjPra7nvwQkXJ/jqoTkaJYgwB8kkHBOkpnGlAHa5TIQeAhY0xA7amJSAIQAZxy37UqkEYaiUh34DOgHHAW2GiMucneqnInIl2BT4AQYLQx5i2bS8oTEfkBaI81Ne8J4FVjzChbi8oDEWkH/AZswfr/BXjBGDPXvqryRkQaA2Ox/oYcwGRjzOs+aTsYA18ppYJRUHbpKKVUMNLAV0qpIKGBr5RSQUIDXymlgoQGvlJKBQkNfKWUChIa+EopFST+F/uayngDA1iSAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + } + } + ] } ] } \ No newline at end of file From afee37b50d7ee8226f05524d71fa90abd3815863 Mon Sep 17 00:00:00 2001 From: Moistsoy <89026391+Moistsoy@users.noreply.github.com> Date: Sun, 3 Oct 2021 21:31:15 +0900 Subject: [PATCH 7/7] Created using Colaboratory --- ...e_Classification_Data(Fashion-MNIST).ipynb | 1790 +++++------------ 1 file changed, 499 insertions(+), 1291 deletions(-) diff --git a/Ch05_Linear_Neural_Networks/Image_Classification_Data(Fashion-MNIST).ipynb b/Ch05_Linear_Neural_Networks/Image_Classification_Data(Fashion-MNIST).ipynb index f64fe0a..f53d46f 100644 --- a/Ch05_Linear_Neural_Networks/Image_Classification_Data(Fashion-MNIST).ipynb +++ b/Ch05_Linear_Neural_Networks/Image_Classification_Data(Fashion-MNIST).ipynb @@ -1,1332 +1,540 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Image Classification Data (Fashion-MNIST)\n", - "\n", - "Before we implement softmax regression ourselves, let's pick a real dataset to work with. To make things visually compelling, we will pick an image classification dataset. The most commonly used image classification data set is the [MNIST](http://yann.lecun.com/exdb/mnist/) handwritten digit recognition data set, proposed by LeCun, Cortes and Burges in the 1990s. However, even simple models achieve classification accuracy over 95% on MNIST, so it is hard to spot the differences between better models and weaker ones. In order to get a better intuition, we will use the qualitatively similar, but comparatively complex [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist) dataset, proposed by [Xiao, Rasul and Vollgraf](https://arxiv.org/abs/1708.07747) in 2017." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Getting the Data\n", - "\n", - "First, import the packages or modules required in this section.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "%matplotlib inline\n", - "import sys\n", - "sys.path.insert(0, '..')\n", - "import d2l\n", - "\n", - "import torch\n", - "import torchvision\n", - "from torchvision import transforms\n", - "from torch.utils.data import DataLoader\n", - "import time" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Conveniently, PyTorch's `torchvision.datasets` package provides easy access to a number of benchmark vision datasets for testing our models.\n", - "The first time we invoke `data.vision.FashionMNIST(train=True)`\n", - "to collect the training data,\n", - "PyTorch will automatically retrieve the dataset via our Internet connection.\n", - "Subsequently, PyTorch will use the already-downloaded local copy.\n", - "We specify whether we are requesting the training set or the test set\n", - "by setting the value of the parameter `train` to `True` or `False`, respectively.\n", - "Recall that we will only be using the training data for training,\n", - "holding out the test set for a final evaluation of our model." - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [], - "source": [ - "# By default pytorch torchvision datasets are of type PIL.\n", - "# Define a transform \"trans\" to change the PIL to Tensor format.\n", - "trans = transforms.ToTensor() " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The `ToTensor` Transform also moves the image channel from the last dimension to the first dimension to facilitate the convolutional neural network calculations introduced later." - ] + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + }, + "colab": { + "name": "Image_Classification_Data(Fashion-MNIST).ipynb", + "provenance": [] + }, + "language_info": { + "name": "python" + } }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ + "cells": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz to ./FashionMNIST/raw/train-images-idx3-ubyte.gz\n" - ] + "cell_type": "markdown", + "metadata": { + "id": "m8nl8kOUe-wB" + }, + "source": [ + "# Image Classification Data (Fashion-MNIST)\n", + "\n", + "Before we implement softmax regression ourselves, let's pick a real dataset to work with. To make things visually compelling, we will pick an image classification dataset. The most commonly used image classification data set is the [MNIST](http://yann.lecun.com/exdb/mnist/) handwritten digit recognition data set, proposed by LeCun, Cortes and Burges in the 1990s. However, even simple models achieve classification accuracy over 95% on MNIST, so it is hard to spot the differences between better models and weaker ones. In order to get a better intuition, we will use the qualitatively similar, but comparatively complex [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist) dataset, proposed by [Xiao, Rasul and Vollgraf](https://arxiv.org/abs/1708.07747) in 2017." + ] }, { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "55be4164ba0b495098cd99457b97cfaa", - "version_major": 2, - "version_minor": 0 + "cell_type": "markdown", + "metadata": { + "id": "I0Q4Ew9ne-wD" }, - "text/plain": [ - "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))" + "source": [ + "## Getting the Data\n", + "\n", + "First, import the packages or modules required in this section.\n", + "\n" ] - }, - "metadata": {}, - "output_type": "display_data" }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "Extracting ./FashionMNIST/raw/train-images-idx3-ubyte.gz to ./FashionMNIST/raw\n", - "Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz to ./FashionMNIST/raw/train-labels-idx1-ubyte.gz\n" - ] + "cell_type": "code", + "metadata": { + "id": "Yrc5aq4b-oCg" + }, + "source": [ + "!pip install d2l" + ], + "execution_count": 1, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "uHnsGsXae-wE" + }, + "source": [ + "%matplotlib inline\n", + "import sys\n", + "sys.path.insert(0, '..')\n", + "import d2l\n", + "from IPython import display\n", + "from matplotlib import pyplot as plt\n", + "import torch\n", + "import torchvision\n", + "from torchvision import transforms\n", + "from torch.utils.data import DataLoader\n", + "\n", + "import time" + ], + "execution_count": 2, + "outputs": [] }, { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "ba74efac1d584f4ba6686e3b7c40abee", - "version_major": 2, - "version_minor": 0 + "cell_type": "markdown", + "metadata": { + "id": "R8H2S-RSe-wF" }, - "text/plain": [ - "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))" + "source": [ + "Conveniently, PyTorch's `torchvision.datasets` package provides easy access to a number of benchmark vision datasets for testing our models.\n", + "The first time we invoke `data.vision.FashionMNIST(train=True)`\n", + "to collect the training data,\n", + "PyTorch will automatically retrieve the dataset via our Internet connection.\n", + "Subsequently, PyTorch will use the already-downloaded local copy.\n", + "We specify whether we are requesting the training set or the test set\n", + "by setting the value of the parameter `train` to `True` or `False`, respectively.\n", + "Recall that we will only be using the training data for training,\n", + "holding out the test set for a final evaluation of our model." ] - }, - "metadata": {}, - "output_type": "display_data" }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "Extracting ./FashionMNIST/raw/train-labels-idx1-ubyte.gz to ./FashionMNIST/raw\n", - "Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz to ./FashionMNIST/raw/t10k-images-idx3-ubyte.gz\n", - "\n" - ] + "cell_type": "code", + "metadata": { + "id": "DIP-pcvJe-wF" + }, + "source": [ + "# By default pytorch torchvision datasets are of type PIL.\n", + "# Define a transform \"trans\" to change the PIL to Tensor format.\n", + "trans = transforms.ToTensor() " + ], + "execution_count": 3, + "outputs": [] }, { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "cfd4d76a7b16448596dc1d30e0289a87", - "version_major": 2, - "version_minor": 0 + "cell_type": "markdown", + "metadata": { + "id": "McDcTejYe-wG" }, - "text/plain": [ - "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))" + "source": [ + "The `ToTensor` Transform also moves the image channel from the last dimension to the first dimension to facilitate the convolutional neural network calculations introduced later." ] - }, - "metadata": {}, - "output_type": "display_data" }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "Extracting ./FashionMNIST/raw/t10k-images-idx3-ubyte.gz to ./FashionMNIST/raw\n", - "Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz to ./FashionMNIST/raw/t10k-labels-idx1-ubyte.gz\n" - ] + "cell_type": "code", + "metadata": { + "id": "Zz0pkE_Pe-wG" + }, + "source": [ + "mnist_train = torchvision.datasets.FashionMNIST(root=\"./\", train=True, transform=trans, target_transform=None, download=True)\n", + "mnist_test = torchvision.datasets.FashionMNIST(root=\"./\", train=False, transform=trans, target_transform=None, download=True)" + ], + "execution_count": null, + "outputs": [] }, { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "4fbefcdb6c9646f880a9d8486bfe3ff5", - "version_major": 2, - "version_minor": 0 + "cell_type": "markdown", + "metadata": { + "id": "bsOEIIzfe-wH" }, - "text/plain": [ - "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))" + "source": [ + "The number of images for each category in the training set and the testing set is 6,000 and 1,000, respectively. Since there are 10 categories, the numbers of examples in the training set and the test set are 60,000 and 10,000, respectively." ] - }, - "metadata": {}, - "output_type": "display_data" }, { - "name": "stdout", - "output_type": "stream", - "text": [ - "Extracting ./FashionMNIST/raw/t10k-labels-idx1-ubyte.gz to ./FashionMNIST/raw\n", - "Processing...\n", - "Done!\n" - ] + "cell_type": "code", + "metadata": { + "id": "JBOUknvye-wH", + "outputId": "bbd26b10-fb4d-49e8-de77-49880688f980", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "len(mnist_train), len(mnist_test)" + ], + "execution_count": 5, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "(60000, 10000)" + ] + }, + "metadata": {}, + "execution_count": 5 + } + ] }, { - "name": "stderr", - "output_type": "stream", - "text": [ - "/home/jeon/anaconda3/envs/ai_safe/lib/python3.6/site-packages/torchvision/datasets/mnist.py:469: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)\n", - " return torch.from_numpy(parsed.astype(m[2], copy=False)).view(*s)\n" - ] - } - ], - "source": [ - "mnist_train = torchvision.datasets.FashionMNIST(root=\"./\", train=True, transform=trans, target_transform=None, download=True)\n", - "mnist_test = torchvision.datasets.FashionMNIST(root=\"./\", train=False, transform=trans, target_transform=None, download=True)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The number of images for each category in the training set and the testing set is 6,000 and 1,000, respectively. Since there are 10 categories, the numbers of examples in the training set and the test set are 60,000 and 10,000, respectively." - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [ + "cell_type": "markdown", + "metadata": { + "id": "fAnZ1O1be-wI" + }, + "source": [ + "We can access any example by indexing into the dataset using square brackets `[]`. In the following code, we access the image and label corresponding to the first example." + ] + }, { - "data": { - "text/plain": [ - "(60000, 10000)" + "cell_type": "code", + "metadata": { + "id": "rT37CGmQe-wI" + }, + "source": [ + "feature, label = mnist_train[0]" + ], + "execution_count": 6, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "u5_KbsUBe-wI" + }, + "source": [ + "Our example, stored here in the variable `feature` corresponds to an image with a height and width of 28 pixels. PyTorch automatically scales it into a tensor with each pixel value between 0 and 1. It is stored in a 3D Tensor. Its first dimension is the number of channels. Since the data set is a grayscale image, the number of channels is 1. When we encounter color, images, we'll have 3 channels for red, green, and blue. To keep things simple, we will record the shape of the image with the height and width of $h$ and $w$ pixels, respectively, as $h \\times w$ or `(h, w)`." ] - }, - "execution_count": 4, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "len(mnist_train), len(mnist_test)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can access any example by indexing into the dataset using square brackets `[]`. In the following code, we access the image and label corresponding to the first example." - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "feature, label = mnist_train[0]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Our example, stored here in the variable `feature` corresponds to an image with a height and width of 28 pixels. PyTorch automatically scales it into a tensor with each pixel value between 0 and 1. It is stored in a 3D Tensor. Its first dimension is the number of channels. Since the data set is a grayscale image, the number of channels is 1. When we encounter color, images, we'll have 3 channels for red, green, and blue. To keep things simple, we will record the shape of the image with the height and width of $h$ and $w$ pixels, respectively, as $h \\times w$ or `(h, w)`." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ + }, { - "data": { - "text/plain": [ - "(torch.Size([1, 28, 28]), torch.float32)" + "cell_type": "code", + "metadata": { + "id": "lMCbmWOEe-wJ", + "outputId": "61ac3a1b-d9a2-497e-b45c-27b1a82b2225", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "feature.shape, feature.dtype" + ], + "execution_count": 7, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "(torch.Size([1, 28, 28]), torch.float32)" + ] + }, + "metadata": {}, + "execution_count": 7 + } ] - }, - "execution_count": 6, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "feature.shape, feature.dtype" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The label of each image is represented as a scalar in PyTorch. Its type is a 64-bit integer." - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [ + }, { - "data": { - "text/plain": [ - "(9, int)" + "cell_type": "markdown", + "metadata": { + "id": "SxC-gLi9e-wJ" + }, + "source": [ + "The label of each image is represented as a scalar in PyTorch. Its type is a 64-bit integer." ] - }, - "execution_count": 8, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "label, type(label)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "There are 10 categories in Fashion-MNIST: t-shirt, trousers, pullover, dress, coat, sandal, shirt, sneaker, bag and ankle boot. The following function can convert a numeric label into a corresponding text label." - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": {}, - "outputs": [], - "source": [ - "# This function has been saved in the d2l package for future use\n", - "def get_fashion_mnist_labels(labels):\n", - " text_labels = ['t-shirt', 'trouser', 'pullover', 'dress', 'coat',\n", - " 'sandal', 'shirt', 'sneaker', 'bag', 'ankle boot']\n", - " return [text_labels[int(i)] for i in labels]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The following defines a function that can draw multiple images and corresponding labels in a single line." - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [], - "source": [ - "# This function has been saved in the d2l package for future use\n", - "def show_fashion_mnist(images, labels):\n", - " d2l.use_svg_display()\n", - " # Here _ means that we ignore (not use) variables\n", - " _, figs = d2l.plt.subplots(1, len(images), figsize=(12, 12))\n", - " for f, img, lbl in zip(figs, images, labels):\n", - " f.imshow(img.reshape((28, 28)).numpy())\n", - " f.set_title(lbl)\n", - " f.axes.get_xaxis().set_visible(False)\n", - " f.axes.get_yaxis().set_visible(False)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next, let's take a look at the image contents and text labels for the first nine examples in the training data set.\n", - "\n", - "Note: PyTorch DataLoader objects don't support regular array slicing. You can instead iterate through." - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [ + }, { - "data": { - "image/svg+xml": [ - "\n", - "\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "\n" + "cell_type": "code", + "metadata": { + "id": "wHWoYIVae-wJ", + "outputId": "c3e1077e-3c26-4327-e590-0d400ea95f75", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "label, type(label)" ], - "text/plain": [ - "
" + "execution_count": 8, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "(9, int)" + ] + }, + "metadata": {}, + "execution_count": 8 + } ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "X=[]\n", - "y=[]\n", - "for idx, data in enumerate(mnist_train):\n", - " if(idx>=0 and idx<10):\n", - " X.append(data[0])\n", - " y.append(data[1])\n", - " if (idx>=10):\n", - " break\n", - "# X, y = mnist_train[0:9]\n", - "show_fashion_mnist(X, get_fashion_mnist_labels(y))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Reading a Minibatch\n", - "\n", - "To make our life easier when reading from the training and test sets we use a `DataLoader` rather than creating one from scratch, as we did in `chapter_linear_scratch`. Recall that a data loader reads a mini-batch of data with an example number of `batch_size` each time.\n", - "\n", - "In practice, reading data can often be a significant performance bottleneck for training, especially when the model is simple or when the computer is fast. A handy feature of PyTorch's `DataLoader` is the ability to use multiple processes to speed up data reading. For instance, we can set aside 4 processes to read the data (via `num_workers`).\n", - "\n", - "We've already applied required transformations before." - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": {}, - "outputs": [], - "source": [ - "batch_size = 256\n", - "if sys.platform.startswith('win'):\n", - " # set 0 for windows\n", - " # 0 means no additional processes are needed to speed up the reading of data\n", - " num_workers = 0\n", - "else:\n", - " num_workers = 4\n", - "\n", - "train_iter = DataLoader(mnist_train, batch_size, shuffle=True, num_workers=num_workers)\n", - "test_iter = DataLoader(mnist_test, batch_size, shuffle=False, num_workers=num_workers)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The logic that we will use to obtain and read the Fashion-MNIST data set is\n", - "encapsulated in the `d2l.load_data_fashion_mnist` function, which we will use in\n", - "later chapters. This function will return two variables, `train_iter` and\n", - "`test_iter`. As the content of this book continues to deepen, we will further\n", - "improve this function.\n", - "\n", - "Let's look at the time it takes to read the training data." - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "metadata": {}, - "outputs": [ + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hjXRqLE8e-wJ" + }, + "source": [ + "There are 10 categories in Fashion-MNIST: t-shirt, trousers, pullover, dress, coat, sandal, shirt, sneaker, bag and ankle boot. The following function can convert a numeric label into a corresponding text label." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "-QrG4ztYe-wK" + }, + "source": [ + "# This function has been saved in the d2l package for future use\n", + "def get_fashion_mnist_labels(labels):\n", + " text_labels = ['t-shirt', 'trouser', 'pullover', 'dress', 'coat',\n", + " 'sandal', 'shirt', 'sneaker', 'bag', 'ankle boot']\n", + " return [text_labels[int(i)] for i in labels]" + ], + "execution_count": 9, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xpRaSnLBe-wK" + }, + "source": [ + "The following defines a function that can draw multiple images and corresponding labels in a single line." + ] + }, { - "data": { - "text/plain": [ - "'0.99 sec'" + "cell_type": "code", + "metadata": { + "id": "OKRDNxOre-wK" + }, + "source": [ + "# This function has been saved in the d2l package for future use\n", + "def show_fashion_mnist(images, labels):\n", + " display.set_matplotlib_formats('svg')\n", + " # Here _ means that we ignore (not use) variables\n", + " _, figs = plt.subplots(1, len(images), figsize=(12, 12))\n", + " for f, img, lbl in zip(figs, images, labels):\n", + " f.imshow(img.reshape((28, 28)).numpy())\n", + " f.set_title(lbl)\n", + " f.axes.get_xaxis().set_visible(False)\n", + " f.axes.get_yaxis().set_visible(False)" + ], + "execution_count": 10, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "V0r_8Zz0e-wK" + }, + "source": [ + "Next, let's take a look at the image contents and text labels for the first nine examples in the training data set.\n", + "\n", + "Note: PyTorch DataLoader objects don't support regular array slicing. You can instead iterate through." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "lcrs8wyPe-wK", + "outputId": "e8f6574c-1704-4bd5-bb79-37ddf7391d54", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 142 + } + }, + "source": [ + "X=[]\n", + "y=[]\n", + "for idx, data in enumerate(mnist_train):\n", + " if(idx>=0 and idx<10):\n", + " X.append(data[0])\n", + " y.append(data[1])\n", + " if (idx>=10):\n", + " break\n", + "# X, y = mnist_train[0:9]\n", + "show_fashion_mnist(X, get_fashion_mnist_labels(y))" + ], + "execution_count": 11, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/svg+xml": "\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" + }, + "metadata": { + "needs_background": "light" + } + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "iVpiwmFoe-wL" + }, + "source": [ + "## Reading a Minibatch\n", + "\n", + "To make our life easier when reading from the training and test sets we use a `DataLoader` rather than creating one from scratch, as we did in `chapter_linear_scratch`. Recall that a data loader reads a mini-batch of data with an example number of `batch_size` each time.\n", + "\n", + "In practice, reading data can often be a significant performance bottleneck for training, especially when the model is simple or when the computer is fast. A handy feature of PyTorch's `DataLoader` is the ability to use multiple processes to speed up data reading. For instance, we can set aside 4 processes to read the data (via `num_workers`).\n", + "\n", + "We've already applied required transformations before." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ASaTFlP5e-wL", + "outputId": "d1f6236f-4782-4dab-b606-5a77657b9cbe", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "batch_size = 256\n", + "if sys.platform.startswith('win'):\n", + " # set 0 for windows\n", + " # 0 means no additional processes are needed to speed up the reading of data\n", + " num_workers = 0\n", + "else:\n", + " num_workers = 4\n", + "\n", + "train_iter = DataLoader(mnist_train, batch_size, shuffle=True, num_workers=num_workers)\n", + "test_iter = DataLoader(mnist_test, batch_size, shuffle=False, num_workers=num_workers)" + ], + "execution_count": 12, + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:481: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n", + " cpuset_checked))\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rOsZShjae-wL" + }, + "source": [ + "The logic that we will use to obtain and read the Fashion-MNIST data set is\n", + "encapsulated in the `d2l.load_data_fashion_mnist` function, which we will use in\n", + "later chapters. This function will return two variables, `train_iter` and\n", + "`test_iter`. As the content of this book continues to deepen, we will further\n", + "improve this function.\n", + "\n", + "Let's look at the time it takes to read the training data." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "T_hpAoOMe-wL", + "outputId": "621f0cdc-30cd-47cf-8079-c679c9a5a10a", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 90 + } + }, + "source": [ + "start = time.time()\n", + "for X, y in train_iter:\n", + " continue\n", + "'%.2f sec' % (time.time() - start)" + ], + "execution_count": 13, + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:481: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n", + " cpuset_checked))\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + }, + "text/plain": [ + "'6.91 sec'" + ] + }, + "metadata": {}, + "execution_count": 13 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "XDlnwXgce-wM" + }, + "source": [ + "## Summary\n", + "\n", + "* Fashion-MNIST is an apparel classification data set containing 10 categories, which we will use to test the performance of different algorithms in later chapters.\n", + "* We store the shape of image using height and width of $h$ and $w$ pixels, respectively, as $h \\times w$ or `(h, w)`.\n", + "* Data iterators are a key component for efficient performance. Use existing ones if available." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Spvu1DmEe-wM" + }, + "source": [ + "## Exercises\n", + "\n", + "1. Does reducing `batch_size` (for instance, to 1) affect read performance?\n", + "\n", + "- batch_size = 256\n", + " 6.44 sec\n", + "- batch_size = 128\n", + " 7.15 sec\n", + "- batch_size = 256\n", + " 7.99 sec\n", + "- batch_size = 1\n", + " 153.98 sec\n", + "- performance gets worse when reducing batch_size\n", + "\n", + "2. For non-Windows users, try modifying `num_workers` to see how it affects read performance.\n", + "- Check the code below\n", + "\n", + "3. Use the PyTorch documentation to see which other datasets are available in `torchvision.datasets`.\n", + "\n", + "4. Use the PyTorch documentation to see which other transformations are available in `torchvision.transforms`." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "JLTIRfS8Hcbu", + "outputId": "d36b59a7-d410-4b4c-9d11-487469d9299e", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "source": [ + "batch_size = 256\n", + "record = []\n", + "\n", + "for workers in [1, 2]:\n", + " train_iter = DataLoader(mnist_train, batch_size, shuffle=True, num_workers=num_workers)\n", + " \n", + " start = time.time()\n", + " for x, y in train_iter: continue\n", + " record.append(time.time()-start)\n", + "\n", + "print(f'workers: 1 time: {record[0]:.3f}\\nworkers: 2 time: {record[1]:.3f}')\n" + ], + "execution_count": 14, + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:481: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n", + " cpuset_checked))\n" + ] + }, + { + "output_type": "stream", + "name": "stdout", + "text": [ + "workers: 1 time: 6.852\n", + "workers: 2 time: 6.570\n" + ] + } ] - }, - "execution_count": 13, - "metadata": {}, - "output_type": "execute_result" } - ], - "source": [ - "start = time.time()\n", - "for X, y in train_iter:\n", - " continue\n", - "'%.2f sec' % (time.time() - start)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Summary\n", - "\n", - "* Fashion-MNIST is an apparel classification data set containing 10 categories, which we will use to test the performance of different algorithms in later chapters.\n", - "* We store the shape of image using height and width of $h$ and $w$ pixels, respectively, as $h \\times w$ or `(h, w)`.\n", - "* Data iterators are a key component for efficient performance. Use existing ones if available." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Exercises\n", - "\n", - "1. Does reducing `batch_size` (for instance, to 1) affect read performance?\n", - "1. For non-Windows users, try modifying `num_workers` to see how it affects read performance.\n", - "1. Use the PyTorch documentation to see which other datasets are available in `torchvision.datasets`.\n", - "1. Use the PyTorch documentation to see which other transformations are available in `torchvision.transforms`." - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "ai_safe", - "language": "python", - "name": "ai_safe" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.12" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} + ] +} \ No newline at end of file