{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\\* This tutorial was originally from the class CS131 at Stanford." ] }, { "cell_type": "markdown", "metadata": { "id": "RFF8vimgZ0t7" }, "source": [ "# Numpy/Python Review\n", "\n", "\n", "### Python\n", "\n", "Easy to pick, programmer-friendly language.\n", "\n", "\"Python is an interpreted language, which can save you considerable time during program development because no compilation and linking is necessary. The interpreter can be used interactively, which makes it easy to experiment with features of the language, to write throw-away programs, or to test functions during bottom-up program development. It is also a handy desk calculator.\" - [Python Tutorial](https://docs.python.org/3/tutorial/appetite.html)\n", "\n", "\n", "Basics:\n", "* Coming from C/C++? Use indentations for code blocks instead of braces.\n", "* Python is dynamically typed, no need to declare variables along with their data types.\n", "* Python is interpretable, and code can be excuted line-by-line - use this to debug!\n", "\n", "\n", "Tips:\n", "* [Python reference docs](https://docs.python.org/3/reference/index.html) are great! Official documentation is your best friend.\n", "* Python tutorial available on [course webpage](http://vision.stanford.edu/teaching/cs131_fall2223/).\n", "\n", "\n", "References:\n", "1. [Wikipedia](https://en.wikipedia.org/wiki/Python_(programming_language))\n", "2. [Official Python Tutorial](https://docs.python.org/3/tutorial/index.html)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "81ZiNt13Z0t-" }, "outputs": [], "source": [ "# help?\n", "\n", "help(print)\n", "# help(range)\n", "# help(len)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "LgnAKvpwZ0t_" }, "outputs": [], "source": [ "dir(list)[::-1]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$Question:$ In scientific computing, we would need vector and matrices operations very frequently. How does it look like using python list?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "a = [2, 0, 2, 4]\n", "b = [0, 4, 0, 5]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "a + b # Nope! " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Loop \n", "c = [] \n", "for i in range(len(a)):\n", " c.append(a[i]+b[i])\n", "c" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Loop (one line)\n", "c = [a[i]+b[i] for i in range(len(a))]\n", "c" ] }, { "cell_type": "markdown", "metadata": { "id": "2p37LKqHZ0uA" }, "source": [ "### Numpy\n", "\n", "**So you want to learn about numpy?**\n", "\n", "Numpy is an important library for anyone wanting to work with large scale datasets in Python. Numpy arrays provide value because they are easier and faster to manipulate than traditional python lists.\n", "\n", "### Why use numpy arrays instead of Python lists?\n", "\n", "1. Saves you time as a programmer: when using lists, you have to use loops to apply a function to each element of an array. In numpy, all common mathematical computations can be vectorized. This leads to a much faster runtime, as well as fewer lines of code.\n", "\n", "2. Under the hood, numpy arrays use less memory, and are constrained to a single data type, which enables faster execution." ] }, { "cell_type": "markdown", "metadata": { "id": "pACcX_DmZ0uB" }, "source": [ "### Let's get started!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "UQNige54Z0uC" }, "outputs": [], "source": [ "#It's a common convention to abbreviate numpy as np\n", "import numpy as np" ] }, { "cell_type": "markdown", "metadata": { "id": "iU4hfXObZ0uD" }, "source": [ "### Creating a Numpy Array\n", "There are many ways to create a numpy array. In general, the options are\n", "- converting other Python data structures to np.array format\n", "- numpy functions that create new arrays (i.e. np.ones, np.zeros, np.arange)\n", "- reading files from disk (in this class, mostly images)" ] }, { "cell_type": "markdown", "metadata": { "id": "YDVwh20iZ0uD" }, "source": [ "Note that unlike lists, you can't create an empty numpy array" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "jmJ-JosVZ0uE" }, "outputs": [], "source": [ "# create a 1 dimensional array from a list: [1,2,3,4,5]\n", "print(\"1 dimensional: \")\n", "x = np.array([1,2,3,4,5])\n", "print(\"Shape: \", x.shape)\n", "x" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "DpDlwAi2Z0uF" }, "outputs": [], "source": [ "# Create a 2-d array from multiple lists:\n", " # first row = [1,2,3,4,5]\n", " # second row = [6,7,8,9,10]\n", "print(\"2 dimensional: \")\n", "y = np.array([[1,2,3,4,5], [6,7,8,9,10]])\n", "print(\"Shape: \", y.shape)\n", "y" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "peVa5-abZ0uF" }, "outputs": [], "source": [ "# you can create an array of any dimensions using np.zeros\n", "z = np.zeros((3, 3))\n", "z" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "W06e6z3IZ0uF" }, "outputs": [], "source": [ "#other handy things\n", "identity = np.identity(3) # also np.eye()\n", "print(identity)\n", "\n", "ones = np.ones((2,2))\n", "print(ones)" ] }, { "cell_type": "markdown", "metadata": { "id": "ZGOBzZpkZ0uG" }, "source": [ "### Array attributes" ] }, { "cell_type": "markdown", "metadata": { "id": "LkJqiq8sZ0uG" }, "source": [ "Every numpy array is a grid of elements of the same type. Numpy provides a large set of numeric datatypes that you can use to construct arrays. Numpy tries to guess a datatype when you create an array, but functions that construct arrays usually also include an optional argument to explicitly specify the datatype. Take a look at the attributes associated with a numpy array, and then we'll explore how numpy arrays determine their type." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "K7rKzbEUZ0uH" }, "outputs": [], "source": [ "print(identity.shape) #returns tuple of dimensions\n", "print(identity.size) # returns total number of elements\n", "print(identity.dtype) #the default dtype is float64" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "YOoZeVpxZ0uH" }, "outputs": [], "source": [ "x = np.array([1, 2]) # Let numpy choose the datatype\n", "y = np.array([1, 2.0]) # Let numpy choose the datatype\n", "z = np.array([1, 2], dtype=np.int64) # Force a particular datatype\n", "\n", "print(x.dtype, y.dtype, z.dtype)" ] }, { "cell_type": "markdown", "metadata": { "id": "jvR21nr3Z0uH" }, "source": [ "### Accessing elements" ] }, { "cell_type": "markdown", "metadata": { "id": "lEUVeOUbZ0uI" }, "source": [ "Numpy offers several ways to index into arrays. When arrays are one dimensional, indexing works just like lists. When arrays are 2 or more dimensional, you specify an index for each dimension:\n", "\n", "`value = array[row_index, col_index]`" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "zTZ51y4IZ0uI" }, "outputs": [], "source": [ "# Create a 2-dimensional array (matrix)\n", "# [[ 1 2 3]\n", "# [ 4 5 6]]\n", "a = np.array([[1,2,3],[4,5,6]])\n", "print(a)\n", "\n", "# Access the 3 with array indexing\n", "a_3 = a[0,2]\n", "print(\"expecting 3, got: \", a_3)\n", "\n", "# Access the 4 with array indexing\n", "a_4 = a[1,0]\n", "print(\"expecting 4, got: \", a_4)" ] }, { "cell_type": "markdown", "metadata": { "id": "vpb3XNoeZ0uI" }, "source": [ "Slicing: Similar to Python lists, numpy arrays can be sliced. Since arrays may be multidimensional, you must specify a slice for each dimension of the array. Recall how slicing works in lists:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "QgHd7WOTZ0uJ" }, "outputs": [], "source": [ "# Recall list slicing:\n", "l = [0,1,2,3,4]\n", "\n", "print(l)\n", "print(\"All elements of l: \", l[:])\n", "print(\"All after 2nd element: \", l[2:])\n", "print(\"All before 2nd element: \", l[:2])\n", "print(\"All between 1st and 3rd element exclusive: \", l[1:3])" ] }, { "cell_type": "markdown", "metadata": { "id": "fKjjcnx_Z0uJ" }, "source": [ "Now let's try slicing in two dimensions!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "6sy59D9Ee3Uy" }, "outputs": [], "source": [ "print(a)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "UNk3HpUPZ0uJ" }, "outputs": [], "source": [ "print(\"a = \\n\", a)\n", "\n", "# Access the 0th row of a:\n", "a_row0 = a[0]\n", "print(\"0th row expecting [1,2,3], got: \", a_row0)\n", "\n", "# Access the 0th column of a\n", "a_col0 = a[:, 0]\n", "print(\"0th column expecting [1,4], got: \", a_col0)" ] }, { "cell_type": "markdown", "metadata": { "id": "ufebVPLX3JUN" }, "source": [ "Exercise:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "N-4ubj6yZ0uJ" }, "outputs": [], "source": [ "# TODO: Create the following rank 2 array with shape (3, 4)\n", "# [[ 1, 2, 3, 4]\n", "# [ 5, 6, 7, 8]\n", "# [ 9, 10, 11, 12]\n", "# [13, 14, 15, 16]]\n", "\n", "# HINT: if you want to take a fancy shortcut, use np.arange() to create this matrix\n", "\n", "b =\n", "print(b)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "j4AIVLTwZ0uK" }, "outputs": [], "source": [ "# TODO: use slicing to pull out the middle two rows\n", "b_rows =\n", "print(\"middle two rows: \\n\", b_rows)\n", "\n", "# TODO: use slicing to pull out the middle two columns\n", "b_cols =\n", "print(\"middle two columns: \\n\", b_cols)\n", "\n", "# TODO: use slicing to pull out only the middle two rows and middle two columns,\n", "# meaning a (2,2) array:\n", "# [[6 7]\n", "# [10 11]]\n", "b_center =\n", "print(\"center two columns and two rows: \\n\", b_center)\n" ] }, { "cell_type": "markdown", "metadata": { "id": "n8YE7OppZ0uK" }, "source": [ "A slice of an array is a view into the same data, so modifying it will modify the original array." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "fNxbNqPZZ0uK" }, "outputs": [], "source": [ "# [[ 1 2 3]\n", "# [ 4 5 6]]\n", "a = np.array([[1,2,3],[4,5,6]])\n", "print(\"original a = \\n\", a)\n", "\n", "a_row0 = a[0,:]\n", "a_row0[1] = 100 # a_row0[1] is the same piece of data as a[0, 1]\n", "\n", "print(\"new a = \\n\", a)" ] }, { "cell_type": "markdown", "metadata": { "id": "OE4SLYosZ0uK" }, "source": [ "### Matrix Operations\n", "Numpy makes it easy to do matrix operations on arrays, like multiply, invert, etc. Basic mathematical functions operate elementwise on arrays, and are available both as operator overloads and as functions in the numpy module" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "D8jvpeSKZ0uK" }, "outputs": [], "source": [ "x = np.array([[1,2],[3,4]], dtype=np.float64)\n", "y = np.array([[5,6],[7,8]], dtype=np.float64)\n", "\n", "print(\"x = \\n\", x)\n", "print(x.dtype)\n", "print(\"y = \\n\", y)\n", "print(y.dtype)\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "x+y" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "WzBoRomIZ0uL" }, "outputs": [], "source": [ "#all arithmetic operations are applied to a matrix element-wise\n", "print(\"Original x: \")\n", "print(x)\n", "print(\"\\nx+1\")\n", "print(x + 1)\n", "print(\"\\nx*4.5\")\n", "print(4.5*x)\n", "print(\"\\nx/2.0\")\n", "print(x/2.0)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "jM_bJDgMZ0uL" }, "outputs": [], "source": [ "# Elementwise sum; both produce the array\n", "print(\"x = \\n\", x)\n", "print(\"y = \\n\", y)\n", "print(\"\\nx+y:\")\n", "print(x + y)\n", "print(np.add(x, y))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "zVp3eRPcZ0uL" }, "outputs": [], "source": [ "# Elementwise difference; both produce the array\n", "print(\"x = \\n\", x)\n", "print(\"y = \\n\", y)\n", "print(\"\\nx-y:\")\n", "print(x - y)\n", "print(np.subtract(x, y))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "mK0tulGUZ0uL" }, "outputs": [], "source": [ "# Elementwise product; both produce the array\n", "print(\"x = \\n\", x)\n", "print(\"y = \\n\", y)\n", "print(\"\\nx*y:\")\n", "print(x * y)\n", "print(np.multiply(x, y))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ZtWIsJa7Z0uM" }, "outputs": [], "source": [ "# Elementwise division; both produce the array\n", "# [[ 0.2 0.33333333]\n", "# [ 0.42857143 0.5 ]]\n", "print(\"x = \\n\", x)\n", "print(\"y = \\n\", y)\n", "print(\"\\nx/y:\")\n", "print(x / y)\n", "print(np.divide(x, y))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "SFW3RvgFZ0uM" }, "outputs": [], "source": [ "# Elementwise square root; produces the array\n", "# [[ 1. 1.41421356]\n", "# [ 1.73205081 2. ]]\n", "print(\"x = \\n\", x)\n", "print(\"\\nsqrt(x):\")\n", "print(np.sqrt(x))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "fcnbm2VFZ0uM" }, "outputs": [], "source": [ "# Elementwise power; produces the array\n", "# [[ 1. 1.41421356]\n", "# [ 1.73205081 2. ]]\n", "print(\"x = \\n\", x)\n", "print(\"\\nx^2:\")\n", "print(np.power(x,2))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "7D2n5oSCZ0uM" }, "outputs": [], "source": [ "# Will this multiplication operator do elementwise multiplication or matrix multiplication?\n", "x*y" ] }, { "cell_type": "markdown", "metadata": { "id": "JsyKKnw1Z0uN" }, "source": [ "Exercise:\n", "\n", "Let's practice these operations!\n", "\n", "Given arrays x and y, write array z as a function of x and y: z[0] = x[0] - y[0], and z[1] = x[1] - y[1]. Use numpy operations!\n", "\n", "Hint: think about how the 2 equations may be written with vector operations (such as those above), and that will translate cleanly to numpy" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ocLn2n_DZ0uN" }, "outputs": [], "source": [ "# TODO\n", "x = np.array([1,3])\n", "y = np.array([2,5])\n", "z = np.array([-1,-2])\n", "\n", "z_computed =\n", "print(\"Your answer: \", z_computed)\n", "print(\"Expecting: \", z)\n", "if z.all() == z_computed.all():\n", " print(\"Correct!\")\n", "else:\n", " print(\"Try thinking of a numpy function above with x and y that gets you equivalent to z\")" ] }, { "cell_type": "markdown", "metadata": { "id": "H1q5zF8RZ0uN" }, "source": [ "Now, try using the numpy functions to calculate x = ((a+b)*c) - d\n", "\n", "If you get stuck, try following the order of operations with the parenthesis and compute a+b first and verify by hand, and then proceed to (a+b)*c, and so on." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "V7adzxtQZ0uN" }, "outputs": [], "source": [ "\n", "a = np.array([1,3])\n", "b = np.array([1,1])\n", "c = np.array([2,1/2])\n", "d = np.array([1,1])\n", "x = np.array([3,1])\n", "\n", "# TODO\n", "#answer =\n", "\n", "print(\"Your answer: \", answer)\n", "print(\"Expecting: \", x)\n", "if x.all() == answer.all():\n", " print(\"Correct!\")\n", "else:\n", " print(\"Try thinking of a which numpy functions above with a,b, and c gets you equivalent to x\")" ] }, { "cell_type": "markdown", "metadata": { "id": "GIIomipyZ0uO" }, "source": [ "### Axis-based operations" ] }, { "cell_type": "markdown", "metadata": { "id": "1ddmrUDKZ0uO" }, "source": [ "Now that we've covered 2d indexing, there's some important numpy functions that operate on a particular axis, so let's get hands on with sum()! Many numpy operations including sum, max, argmax, mean, standard deviation operate over a chosen axis of your array.\n", "\n", "Sum can sum all elements of the 2d array, or it can sum across each row, or sum down each column.\n", "\n", "Axis 0 corresponds to columns, and axis 1 corresponds to rows, just like indexing. When you apply an operation like sum over axis 0, that means axis 0 is being squashed into length 1 by the sum, max, etc. operation." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "0hxbJYS1Z0uO" }, "outputs": [], "source": [ "#numpy arrays have builtin methods to calculate summary statistics such as std. dev., mean, min, max, sum\n", "y = np.random.random(9).reshape(3,3)\n", "print(\"std:\", y.std())\n", "print(\"mean:\", y.mean())\n", "print(\"sum:\", y.sum())\n", "\n", "print()\n", "#you can also do these operations with respect to a particular axis\n", "print(\"std (rows):\", y.std(axis = 1))\n", "print(\"mean (rows):\", y.mean(axis = 1))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "jBKes_QWZ0uO" }, "outputs": [], "source": [ "# Sum of an array along various axes\n", "x = np.array([[1,2],[3,4]])\n", "print(x)\n", "print()\n", "print(np.sum(x)) # Compute sum of all elements; prints \"10\"\n", "print()\n", "print(np.sum(x, axis=0)) # Compute sum of each column; prints \"[4 6]\"\n", "print()\n", "print(np.sum(x, axis=1)) # Compute sum of each row; prints \"[3 7]\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "qpogwF8GZ0uO" }, "outputs": [], "source": [ "# Sum of a vector doesn't need to specify axis since there's only one axis\n", "x = np.array([1,2,3])\n", "print(x)\n", "print(np.sum(x))" ] }, { "cell_type": "markdown", "metadata": { "id": "4_0F6HJbZ0uP" }, "source": [ "Exercise:\n", "Your turn!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "dYtCDi-yZ0uP" }, "outputs": [], "source": [ "# Get hands-on with sum! Guess what the output will be and then run it to confirm\n", "x = np.array([[1,2,3],[4,5,6],[3,2,1]])\n", "print(\"x = \\n\", x)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "nFnjfRWTZ0uP" }, "outputs": [], "source": [ "#np.sum(x)\n", "# TODO:\n", "guess =\n", "print(\"Your guess for np.sum(x): \", guess)\n", "print(\"Actual = \", np.sum(x))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "5J5qI2kYZ0uP" }, "outputs": [], "source": [ "#np.sum(x, axis=1)\n", "# TODO:\n", "guess =\n", "print(\"Your guess for np.sum(x, axis=1): \", guess)\n", "print(\"Actual = \", np.sum(x, axis=1))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Dp5knbxlZ0uP", "scrolled": true }, "outputs": [], "source": [ "#np.sum(x, axis=0)\n", "# TODO:\n", "guess =\n", "print(\"Your guess for np.sum(x, axis=0): \", guess)\n", "print(\"Actual = \", np.sum(x, axis=0))" ] }, { "cell_type": "markdown", "metadata": { "id": "LB_N-_P2Z0uQ" }, "source": [ "# Demo\n", "Imagine I want to apply the function $f(x) = x^2 + x -6$ to every emelent of an array" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "y9w_AnmIZ0uQ" }, "outputs": [], "source": [ "#First let's try it with python lists\n", "import random\n", "random.seed(1)\n", "h = 1000000\n", "\n", "def func(x):\n", " return x**2 + x - 6\n", "\n", "first_list = [random.randint(-10,10) for i in range(h)]\n", "print(len(first_list))\n", "first_list[:10]\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "PyVF4XuEZ0uQ" }, "outputs": [], "source": [ "%%time\n", "for i in range(h):\n", " first_list[i] = func(first_list[i])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "TiKgYuL3Z0uQ" }, "outputs": [], "source": [ "first_list[:10]" ] }, { "cell_type": "markdown", "metadata": { "id": "S9pqYfElZ0uQ" }, "source": [ "### Now let's do the same thing with numpy arrays" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "BfOrrp8rZ0uQ" }, "outputs": [], "source": [ "#By default, rand returns a rnadom number in the range [0,1) - so we normalize to get the values we want\n", "array = np.random.rand(h)\n", "array = array*20 - 10\n", "array = array.astype(np.int64)\n", "print(array[:10])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "r6zCQcolZ0uR" }, "outputs": [], "source": [ "%%time\n", "array = array**2 + array - 6" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "eh1Ic74vZ0uR", "scrolled": true }, "outputs": [], "source": [ "#Check that it worked\n", "print(array[:10])" ] }, { "cell_type": "markdown", "metadata": { "id": "KsB-TbIWZ0uR" }, "source": [ "### It's way faster!" ] }, { "cell_type": "markdown", "metadata": { "id": "n6fMMSrzZ0uR" }, "source": [ "### Some other useful manipulations\n", "A complete list of possible array manipulations can be found here:\n", "https://numpy.org/devdocs/reference/routines.array-manipulation.html" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Wu5e1Rx-Z0uR" }, "outputs": [], "source": [ "x = [[1, 2], [3, 4]]\n", "y = [[4, 5], [6, 7]]\n", "np.concatenate((x,y), axis = 0)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "eh3LmaU7Z0uS" }, "outputs": [], "source": [ "result = np.concatenate((x,y), axis = 1)\n", "result" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "7KluHa41Z0uS" }, "outputs": [], "source": [ "#not an inplace operation, if you want it to persist you need to reassign the variable using =\n", "np.transpose(result)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ocx7S_E-Z0uS" }, "outputs": [], "source": [ "#to invert a matrix use np.linalg.inv\n", "x = np.array([[1,1], [1,0]])\n", "np.linalg.inv(x)" ] }, { "cell_type": "markdown", "metadata": { "id": "GyyfWg8aZ0uS" }, "source": [ "### Numpy for Linear Algebra" ] }, { "cell_type": "markdown", "metadata": { "id": "3HeDLpXrZ0uS" }, "source": [ "#### Dot products" ] }, { "cell_type": "markdown", "metadata": { "id": "-lIm3KaTZ0uS" }, "source": [ "We'll first define vectors v and w. Recall that the dot product of two vectors is calculating by multiplying each corresponding value and then summing the result. Let's start by computing the dot product with a for loop" ] }, { "cell_type": "markdown", "metadata": { "id": "8egbFWD7BwMj" }, "source": [ "Exercise" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "1PRBYBL2Z0uS" }, "outputs": [], "source": [ "v = np.array([1,2,3])\n", "w = np.array([4,5,6])\n", "\n", "print(\"v = \\n\", v)\n", "print(\"w = \\n\", w)\n", "print(\"expected dot product = 32\")\n", "\n", "# TODO: compute dot product of v and w with a for loop\n", "# note: length of a 1 dimensional array = len(array) or array.shape[0]\n", "\n", "forloop_dotproduct = 0\n", "print(forloop_dotproduct)\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "5ntFwZBwZ0uT" }, "outputs": [], "source": [ "# TODO: compute dot product of v and w with numpy functions\n", "# hint: addition and summation are the key operations in a dot product\n", "\n", "np_dotproduct =\n", "print(np_dotproduct)\n" ] }, { "cell_type": "markdown", "metadata": { "id": "BT4IwH3lZ0uT" }, "source": [ "#### Matrix-vector product" ] }, { "cell_type": "markdown", "metadata": { "id": "s12VsKhVZ0uT" }, "source": [ "We'll first define matrix M and vector v. Recall that matrix multiplication between a matrix and a vector involves computing a dot product between each row of the matrix M and the vector v.\n", "\n", "In machine learning, M is typically a matrix of learned parameters where each row corresponds to one of the output predictions, which is a weighted sum of the input features from v. The features v change for every new example but the weights M are applied to every example in training and testing data.\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "KGjvsG2qZ0uT" }, "outputs": [], "source": [ "M = np.array([[1,2,3],[3,2,1]])\n", "v = np.array([4,1,2])\n", "\n", "print(\"M = \\n\", M)\n", "print(\"v = \\n\", v)\n", "print(\"Result = \\n[12, 16]\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "2s6-TKgCZ0uT" }, "outputs": [], "source": [ "# TODO: get each row of M using array slicing\n", "# M_row0 = M[?]\n", "# M_row1 = M[?]\n", "\n", "# TODO: compute the dot product of each row of M with v\n", "# row0_dotproduct = np.dot(?)\n", "# row1_dotproduct = np.dot(?)\n", "\n", "# answer =\n", "print(\"np.dot matrix product = \\n\", answer)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "HNkw4FzJZ0uU" }, "outputs": [], "source": [ "# Use numpy's built in functionality to get the same result\n" ] }, { "cell_type": "markdown", "metadata": { "id": "9i-1cXmPZ0uU" }, "source": [ "#### Matrix-Matrix multiplication" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Q0FPnO5IZ0uU" }, "outputs": [], "source": [ "A = np.array([[1,2],[3,4]])\n", "B = np.array([[4,3],[2,1]])\n", "\n", "print(\"A = \\n\", A)\n", "print(\"B = \\n\", B)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "V6u3ezKPZ0uU" }, "outputs": [], "source": [ "upper_left =\n", "print(\"upper left = \", upper_left)\n", "\n", "upper_right =\n", "print(\"upper right = \", upper_right)\n", "\n", "lower_left =\n", "print(\"lower left = \", lower_left)\n", "\n", "lower_right =\n", "print(\"lower right = \", lower_right)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "rfmA7QyWZ0uU" }, "outputs": [], "source": [ "# Numpy matrix products:\n" ] }, { "cell_type": "markdown", "metadata": { "id": "8CyuIE-LZ0uU" }, "source": [ "### Broadcasting\n", "The term broadcasting refers to how numpy treats doing arithmetic operations on arrays of different shapes.\n", "When operating on two arrays, NumPy compares their shapes element-wise. Two dimensions are compatible when\n", "* they are equal, or\n", "* one of them is 1\n", "\n", "If the dimensional are different, but one of them is 1, then numpy will apply the operation to each column on that axis" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "XTcIRwyVZ0uV" }, "outputs": [], "source": [ "x = np.array(range(5))\n", "x = x.reshape(5, 1)\n", "print(\"x is a column vector\")\n", "print()\n", "print(x.shape)\n", "print(x)\n", "\n", "y = np.ones((5))\n", "print(\"y is a row vector\")\n", "print(y.shape)\n", "print(y)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ASLyXUbBZ0uV" }, "outputs": [], "source": [ "print(x+y)" ] }, { "cell_type": "markdown", "metadata": { "id": "-tip0ZO-Z0uV" }, "source": [ "### Real world example of why broadcasting is useful" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "QoviBJx0Z0uV" }, "outputs": [], "source": [ "# Suppose we have a matrix where we want to normalize the rows to have mean zero\n", "matrix = 10*np.random.rand(4,5)\n", "matrix" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "V9kKMy5wZ0uV" }, "outputs": [], "source": [ "row_means = matrix.mean(axis = 1).reshape((4,1))\n", "row_means" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ju9HFQIoZ0uV" }, "outputs": [], "source": [ "matrix = matrix - row_means\n", "print(matrix)\n", "matrix.mean(axis = 1)" ] }, { "cell_type": "markdown", "metadata": { "id": "OcP0nEvCZ0uW" }, "source": [ "### Using boolean masks\n", "In numpy, when you compare two arrays, you get a boolean mask as the output. You can then use this boolean mask to grab specific values in an array." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ID7haMT-Z0uW" }, "outputs": [], "source": [ "array = np.array(range(20)).reshape((4,5))\n", "output = array > 10\n", "output" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "pmPbfr0FZ0uW" }, "outputs": [], "source": [ "array[output]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "j_oNmg9WZ0uW" }, "outputs": [], "source": [ "#you can combine boolean statements for more complicated operations\n", "mask = (array < 5) | (array > 15)\n", "mask" ] }, { "cell_type": "markdown", "metadata": { "id": "HQx8vY6HZ0uW" }, "source": [ "## Practice boolean mask problem\n", "Given a matrix, change all of the negative values to zero" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "AY3AeV1lZ0uW" }, "outputs": [], "source": [ "matrix = 2*np.random.rand(5, 5) - 1\n", "print(matrix)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "9SkAsQ6pZ0uW" }, "outputs": [], "source": [ "### SOLUTION ###\n", "mask = matrix < 0\n", "print(mask)\n", "matrix[mask] = 0\n", "print(matrix)" ] }, { "cell_type": "markdown", "metadata": { "id": "dk3vUuFyZ0uX" }, "source": [ "### Reshaping" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "gy1stX_JZ0uX" }, "outputs": [], "source": [ "#when your reshape, by default you fill the new array by rows\n", "x = np.linspace(1, 12, 6)\n", "print(x)\n", "x = x.reshape((3,2)) #does not reshape in place!\n", "x" ] }, { "cell_type": "markdown", "metadata": { "id": "KCogQC5CZ0uX" }, "source": [ "# Making Plots in Jupyter Notebook\n", "A Matplotlib figure can be categorized into several parts as below:\n", " \n", "**Figure:** It is a whole figure which may contain one or more than one axes (plots). You can think of a Figure as a canvas which contains plots.\n", "\n", "**Axes:** It is what we generally think of as a plot. A Figure can contain many Axes. It contains two or three (in the case of 3D) Axis objects. Each Axes has a title, an x-label and a y-label.\n", "\n", "**Axis:** They are the number line like objects and take care of generating the graph limits." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "V-NjYYVtZ0uX" }, "outputs": [], "source": [ "import matplotlib.pyplot as plt" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "yrqGqValZ0uX" }, "outputs": [], "source": [ "x = np.arange(10)**2\n", "print(x)\n", "plt.plot(x)\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "OIKyvWPfZ0uX" }, "outputs": [], "source": [ "plt.figure(figsize = (15,15))\n", "plt.plot(x)\n", "plt.title(\"This is a graph\")\n", "plt.xlabel(\"this is the x label\")\n", "plt.ylabel(\"this is the y label\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "UqXN-V20Z0uY" }, "source": [ "# Part 2" ] }, { "cell_type": "markdown", "metadata": { "id": "S9UfCBueZ0uY" }, "source": [ "### Data Structures in Python\n", "You may have noticed that it can be tricky to manipulate arrays in python because by default, python passed objects as references. In this section, we'll explain this behavior, and giving some debugging tools to use when confronted with related issues." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "uST0YY8-Z0uY" }, "outputs": [], "source": [ "array = np.linspace(1, 10, 10)\n", "array" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "gc881PSYZ0uY" }, "outputs": [], "source": [ "dup = array\n", "dup" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Yn57lSHMZ0uY" }, "outputs": [], "source": [ "array[0] = 100\n", "dup" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ITiar0u3gllF" }, "outputs": [], "source": [ "help(id)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "yw7bIIEOZ0uY" }, "outputs": [], "source": [ "print(id(array))\n", "print(id(dup))" ] }, { "cell_type": "markdown", "metadata": { "id": "sWknE4UEZ0uZ" }, "source": [ "### Notice that the dup and array point to the same object!\n", "How would we fix this?\n", "\n", "### Use the copy library, or np.array.copy\n", "IMPORTANT: Using the slicing syntax [:] doesn't always work" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "y6j7-E_7Z0uZ" }, "outputs": [], "source": [ "#slicing\n", "array = np.linspace(1, 10, 10)\n", "dup = array[:]\n", "print(id(array))\n", "print(id(dup))\n", "array[0] = 100\n", "dup" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "gSxwkLlNZ0uZ" }, "outputs": [], "source": [ "#using copy\n", "import copy\n", "array = np.linspace(1, 10, 10)\n", "dup = copy.deepcopy(array)\n", "print(id(array))\n", "print(id(dup))\n", "array[0] = 100\n", "dup" ] }, { "cell_type": "markdown", "metadata": { "id": "N407VV6GZ0uZ" }, "source": [ "Beware of copy vs. deepcopy!\n", "\n", "https://docs.python.org/3.6/library/copy.html\n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "bjQzugV3Z0uZ" }, "outputs": [], "source": [ "#numpy arrays also have a builtin copy function\n", "array = np.linspace(1, 10, 10)\n", "dup = array.copy()\n", "print(id(array))\n", "print(id(dup))\n", "array[0] = 100\n", "dup" ] }, { "cell_type": "markdown", "metadata": { "id": "y4DSSXPRZ0uZ" }, "source": [ "### Let's try it with multidimensional arrays" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "sVIeJpDRZ0ua" }, "outputs": [], "source": [ "def display(img):\n", " # Show image\n", " plt.figure(figsize = (5,5))\n", " plt.imshow(img)\n", " plt.axis('off')\n", " plt.show()\n", "\n", "def load(image_path):\n", " out = io.imread(image_path)\n", "\n", " # Let's convert the image to be between the correct range.\n", " out = out.astype(np.float64) / 255\n", " return out" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "VuTsJfLFZ0ua" }, "outputs": [], "source": [ "from skimage import io\n", "img = load('image1.jpg')\n", "display(img)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "4upgptz8Z0ua" }, "outputs": [], "source": [ "def rgb_exclusion(image, channel):\n", " out = image\n", " if channel == 'R':\n", " out[:, :, 0] = 0\n", " elif channel == 'G':\n", " out[:, :, 1] = 0\n", " elif channel == 'B':\n", " out[:, :, 2] = 0\n", "\n", " return out\n", "\n", "no_green = rgb_exclusion(img, 'G')\n", "display(no_green)\n", "\n", "display(img)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "B2zeVvvHZ0ua" }, "outputs": [], "source": [ "img = load('image1.jpg')\n", "display(img)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "BDfmiZT2Z0ua" }, "outputs": [], "source": [ "#TODO: How to fix?\n", "def rgb_exclusion(image, channel):\n", " out = image.copy()\n", " if channel == 'R':\n", " out[:, :, 0] = 0\n", " elif channel == 'G':\n", " out[:, :, 1] = 0\n", " elif channel == 'B':\n", " out[:, :, 2] = 0\n", "\n", " return out\n", "\n", "no_green = rgb_exclusion(img, 'G')\n", "display(no_green)\n", "\n", "display(img)" ] }, { "cell_type": "markdown", "metadata": { "id": "HbWixCAbZ0ub" }, "source": [ "### Sorting" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "6EOcgXINZ0ub" }, "outputs": [], "source": [ "test = np.random.random(10)\n", "print(test)\n", "print(np.sort(test))\n", "print(np.argsort(test))" ] }, { "cell_type": "markdown", "metadata": { "id": "JyXN3PC2Z0ub" }, "source": [ "### Linear Algebra\n", "We can use the np.linalg module to do a lot of linear algebra stuff withpretty clean syntax.\n", "\n", "For example, say we wanted to solve the linear system $$ Ax = b$$." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "psUBf8wNZ0ub" }, "outputs": [], "source": [ "A = np.array([[1, 1], [2, 1]])\n", "b = np.array([[1], [0]])\n", "#This function takes parameters A, b, and returns x such that Ax =b.\n", "x = np.linalg.solve(A, b)\n", "x" ] }, { "cell_type": "markdown", "metadata": { "id": "_Rr5lVJDZ0ub" }, "source": [ "### How about more complicated stuff?\n", "Imagine trying to find a line of best fit.\n", "\n", "Linear regression finds the \"line of best fit\" by minimizing the residual sum of squares.\n", "\n", "If we have n datapoints $\\{(x_1, y_1), ... ,(x_n, y_n)\\}$, the objective function takes the form\n", "$$loss(X) = \\Sigma_{i = 1}^n (y_i - f(x_i))^2$$ where $$f(x_i) = \\theta_0 + \\theta_1 x_1 + ... +\\theta_n x_n$$\n", "\n", "It turns out the parameters such that the loss function is minimized are given by the closed form solution\n", "\n", "$$\\theta = (X^T X)^{-1} X^T y$$\n", "\n", "Let's see it in action!!!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "kYLe0hoOZ0ub" }, "outputs": [], "source": [ "x = np.concatenate((np.linspace(1, 5, 10).reshape(10, 1), np.ones(10).reshape(10, 1)), axis = 1)\n", "y = x[:,0].copy() + 2*np.random.rand(10) - 0.5\n", "plt.scatter(x[:,0], y)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "aRFBUDWQZ0uc" }, "outputs": [], "source": [ "theta = np.linalg.lstsq(x, y, rcond=None)[0]\n", "print(theta)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "nEXh6NPDZ0uc" }, "outputs": [], "source": [ "plt.scatter(x[:,0], y)\n", "plt.plot(x[:,0], x[:,0]*theta[0] + theta[1])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "JjvhDbfXZ0uc" }, "outputs": [], "source": [ "theta = np.linalg.inv(x.T.dot(x)).dot(x.T).dot(y)\n", "print(theta)" ] }, { "cell_type": "markdown", "metadata": { "id": "meT4vAEQZ0uc" }, "source": [ "### Vectorizing equations\n", "In a couple of homeworks, you'll gain experience formulating equations in matrix format. This can be really tough, so let's go through an example for how to think about it!" ] }, { "cell_type": "markdown", "metadata": { "id": "PpEjslLSZ0uc" }, "source": [ "Suppose we didn't know that linear regression had a closed form solution. We could also solve the problem above using gradient descent.\n", "\n", "The gradient descent update rule looks like this:\n", "$$\\theta_{t+1} =\\theta_t - \\alpha \\nabla_{\\theta} L(\\theta, X)$$\n", "\n", "So we need to find the gradient with respect to $\\theta$. Recall that a gradient is a vector of partial derivatives. So let's start by finding just one partial derivative.\n", "\n", "$$\\frac{\\partial}{\\partial \\theta_j}L(\\theta, X) = \\Sigma_{i = 1}^n 2(y_i - f(x_i))(-x_i[j])$$ where $$f(x_i) = \\theta_0 + \\theta_1 x_1 + ... +\\theta_n x_n$$\n", "\n", "Now the task is to get this into matrix format! Our theta vector is $\\theta = [\\theta_0,\\theta_1] \\in R^2$. Notice that our residuals can be written as a vector, $y - f(\\theta, X) \\in R^n$.\n", "\n", "In matrix multiplication, we dot the row of the first matrix with the columns of the second matrix. A sum (like above) can oftern be expressed as a row times a column.\n", "\n", "$$\\theta_{t+1} =\\theta_t - \\alpha X^T (y - f(X, \\theta))$$\n", "\n", "\n", "\n", "\n", " " ] }, { "cell_type": "markdown", "metadata": { "id": "ZE6YAQ78Z0uc" }, "source": [ "### Additional Resources\n", "https://www.youtube.com/watch?v=8Mpc9ukltVA\n", "\n", "https://jakevdp.github.io/PythonDataScienceHandbook/02.01-understanding-data-types.html\n", "\n", "https://towardsdatascience.com/matplotlib-tutorial-learn-basics-of-pythons-powerful-plotting-library-b5d1b8f67596\n", "\n", "https://scipy-cookbook.readthedocs.io/items/ViewsVsCopies.html\n", "\n", "https://www.geeksforgeeks.org/copy-python-deep-copy-shallow-copy/" ] } ], "metadata": { "colab": { "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.12" } }, "nbformat": 4, "nbformat_minor": 1 }