{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stochastic Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Stochastic Gradient Descent (SGD)</b>, a simple modification applied to the <b>gradient descent</b> algorithm that we saw from previous week's section. SGD also computes the gradient but updates the weight matrix W on small batches of training data, instead of the entire training set itself.\n",
    "\n",
    "While this leads to “noiser” weight updates, it also allows us to take more steps along the gradient (1 step for each batch vs. 1 step per epoch), ultimately leading to faster convergence and no negative affects to loss and classification accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a quick <u>comparison table</u> between Stochastic Gradient Descent and gradient descent:\n",
    "\n",
    "| Algorithm | Time per iter | Total time to convergence<br>for large data in theory | Total time to convergence<br>for large data in practice | Sensitivity to params |\n",
    "|------------------------|---------------------|-------------------------------------------------------|---------------------------------------------------------|-----------------------|\n",
    "| Gradient<br>Descent | Slow for large data | Slower | Usually slower | Moderate |\n",
    "| Stochastic<br>Gradient | Always fast | Faster | Usually faster | Very high |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's consider an example where we illustrate the use of stochastic gradient descent for linear regression. \n",
    "\n",
    "Our goal is to find the equation of the straight line $h_\\theta(x) = \\theta_0 + \\theta_1 x$ that best fits our data points. The function that we are trying to minimize in this case is:\n",
    "\n",
    "$J(\\theta_0,\\theta_1) = {1 \\over 2m} \\sum\\limits_{i=1}^m (h_\\theta(x_i)-y_i)^2$\n",
    "\n",
    "In this case, our gradient will be defined in two dimensions:\n",
    "\n",
    "$\\frac{\\partial}{\\partial \\theta_0} J(\\theta_0,\\theta_1) = \\frac{1}{m}  \\sum\\limits_{i=1}^m (h_\\theta(x_i)-y_i)$\n",
    "\n",
    "$\\frac{\\partial}{\\partial \\theta_1} J(\\theta_0,\\theta_1) = \\frac{1}{m}  \\sum\\limits_{i=1}^m ((h_\\theta(x_i)-y_i) \\cdot x_i)$\n",
    "\n",
    "Then we followed this algorithm (where $\\alpha$ was a non-adapting stepsize):\n",
    "\n",
    "&nbsp;&nbsp;&nbsp; 1: &nbsp; Choose initial guess $x_0$ <br>\n",
    "&nbsp;&nbsp;&nbsp;    2: &nbsp; <b>for</b> k = 0, 1, 2, ... <b>do</b> <br>\n",
    "&nbsp;&nbsp;&nbsp;    3:   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $s_k$ = -$\\nabla f(x_k)$ <br>\n",
    "&nbsp;&nbsp;&nbsp;    4:   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $x_{k+1} = x_k + \\alpha s_k$ <br>\n",
    "&nbsp;&nbsp;&nbsp;    5: &nbsp;  <b>end for</b>\n",
    "\n",
    "Here is the stochastic gradient descent algorithm for simple linear regression, where m is the size of the data set:\n",
    "\n",
    "&nbsp;&nbsp;&nbsp; 1: &nbsp; Randomly shuffle the data set <br>\n",
    "&nbsp;&nbsp;&nbsp;    2: &nbsp; <b>for</b> k = 0, 1, 2, ... <b>do</b> <br>\n",
    "&nbsp;&nbsp;&nbsp;    3: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <b>for</b> i = 1 to m <b>do</b> <br>\n",
    "&nbsp;&nbsp;&nbsp;    4:   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $\\begin{bmatrix}\n",
    " \\theta_0 \\\\ \n",
    " \\theta_1 \\\\ \n",
    " \\end{bmatrix}=\\begin{bmatrix}\n",
    " \\theta_0 \\\\ \n",
    " \\theta_1 \\\\ \n",
    " \\end{bmatrix}-\\alpha\\begin{bmatrix}\n",
    " 2(h_\\theta(x_i)-y_i) \\\\ \n",
    " 2x_i(h_\\theta(x_i)-y_i) \\\\ \n",
    " \\end{bmatrix}$ <br>\n",
    "&nbsp;&nbsp;&nbsp;    5: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <b>end for</b> <br> \n",
    "&nbsp;&nbsp;&nbsp;    6: &nbsp;  <b>end for</b>\n",
    "\n",
    "Typically, with stochastic gradient descent, you will run through the entire data set 1 to 10 times (see value for k in line 2 of the pseudocode above), depending on how fast the data is converging and how large the data set is.\n",
    "\n",
    "In the example below, we'll create a set of 50,000 points around the line 𝑦 = 7+ 3𝑥 + 𝜖 , for values of x between 0 and 100:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "f = lambda x: 7 + x * 3 + np.random.randn(len(x)) * 10\n",
    "x = np.random.random(50000) * 100\n",
    "y = f(x) \n",
    "m = len(y)\n",
    "batch = 100 # batch size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's randomly shuffle around our dataset. Note that in this example, this step isn't strictly necessary since the data is already in a random order. However, that obviously may not always be the case:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll setup our h function and our cost function, which we will use to check how the value is improving."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "h = lambda theta_0, theta_1, x: theta_0 + theta_1 * x\n",
    "cost = lambda theta_0, theta_1, x_i, y_i: 0.5 * (h(theta_0, theta_1, x_i) - y_i) ** 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll run our stochastic gradient descent algorithm. To see it's progress, we'll take a cost measurement at every step. Every 10,000 steps, we'll get an average cost from the last 10,000 steps and then append that to our cost_list variable. We will run through the entire list 10 times here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Local minimum occurs where:\n",
      "theta_0 = 7.07789735537464\n",
      "theta_1 = 3.0368460823859147\n"
     ]
    }
   ],
   "source": [
    "theta_old = np.array([0.,0.])\n",
    "theta_new = np.array([1.,1.]) # The algorithm starts at [1,1]\n",
    "lr = 0.00005 # learning rate\n",
    "batch = 1 # batch size\n",
    "\n",
    "iter_num = 0\n",
    "s_k = np.array([float(\"inf\"), float(\"inf\")])\n",
    "sum_cost = 0\n",
    "cost_list = []\n",
    "\n",
    "for i in range(m * 20):\n",
    "    iter_num += 1\n",
    "    # random choice of matching x and y\n",
    "    idx = np.random.randint(0, m, size=batch)\n",
    "    x_i = x[idx]  # random choice of x\n",
    "    y_i = y[idx]  # random choice of y\n",
    "    \n",
    "    theta_old = theta_new\n",
    "    pred_y = h(theta_old[0], theta_old[1], x_i)\n",
    "    s_k[0] = pred_y - y_i\n",
    "    s_k[1] = (pred_y - y_i) * x_i\n",
    "    s_k = (-1) * s_k\n",
    "    theta_new = theta_old + lr * s_k\n",
    "    \n",
    "    # saved for plot\n",
    "    sum_cost += cost(theta_old[0], theta_old[1], x_i, y_i)\n",
    "    if (i + 1) % 10000 == 0:\n",
    "        cost_list.append(sum_cost / 10000.0)\n",
    "        sum_cost = 0   \n",
    "            \n",
    "print(\"Local minimum occurs where:\")\n",
    "print(\"theta_0 =\", theta_new[0])\n",
    "print(\"theta_1 =\", theta_new[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, our values for $\\theta_0$ and $\\theta_1$ are close to their true values of 7 and 3.\n",
    "\n",
    "Now, we plot our cost versus the number of iterations. As you can see, the cost goes down quickly at first, but starts to level off as we go through more iterations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAABNq0lEQVR4nO29eZhcV3Xu/a6a5+qu6kEttdStWdaMJE94NhiDDRhIQgZICPBdAwlk4EJCCLm5Scj3EcKTBC4QYgwhNwwJGBs7MXjANh4Ay5ZsTZZkSW5NPY/VNXXN+/vjnH3q1NRdVV2np1q/59Gj7qrqqn2qTr1n7XetvTYJIcAwDMM0D6bFHgDDMAyzsLDwMwzDNBks/AzDME0GCz/DMEyTwcLPMAzTZFgWewDV0NbWJnp7exd7GAzDMMuKw4cPjwsh2otvXxbC39vbi0OHDi32MBiGYZYVRHSx3O1s9TAMwzQZLPwMwzBNBgs/wzBMk8HCzzAM02Sw8DMMwzQZLPwMwzBNBgs/wzBMk7Gihf+JUyP46s/OLfYwGIZhlhQrWvifPjOGe57pW+xhMAzDLCkMFX4iaiGi+4joNBGdIqJriejv1d+PEdEDRNRi1Os7rGYk0lmjnp5hGGZZYnTE/0UAjwghtgHYA+AUgMcB7BRC7AZwBsCfGfXiDosJiXQOvMsYwzBMHsOEn4h8AG4E8A0AEEKkhBAhIcRjQoiM+rDnAXQbNQa71QwASGZyRr0EwzDMssPIiH8DgDEA/0pELxPRvUTkLnrMBwD8xKgBOKTwp1n4GYZhJEYKvwXAPgD/LIR4HYAYgE/JO4nozwFkAHyn3B8T0d1EdIiIDo2NjdU1AIdVObxEhn1+hmEYiZHC3w+gXwhxUP39PigXAhDR+wC8FcB7RAUDXghxjxDigBDiQHt7STvpqnBYlIifE7wMwzB5DBN+IcQwgMtEtFW96Q0AThLRmwH8KYC3CyHiRr0+kLd6Emz1MAzDaBi9EcvHAHyHiGwA+gC8H8CLAOwAHiciAHheCPFhI15cs3o44mcYhtEwVPiFEEcAHCi6eZORr6knH/Gz8DMMw0hW9MrdfHKXrR6GYRjJihZ+Oyd3GYZhSljRws8eP8MwTCkrWvhlxM8LuBiGYfKsaOHXkru8gIthGEZjhQs/Wz0MwzDFrHDh5wVcDMMwxaxo4beaTTCbiCN+hmEYHSta+IF8T36GYRhGYeULv9XMyV2GYRgdzSH8bPUwDMNorHjht1tNXMfPMAyjY8ULv8PCET/DMIyelS/8VhN7/AzDMDqaQPjNXNXDMAyjo0mEnyN+hmEYSRMIv4mFn2EYRsfKF34LWz0MwzB6Vrzw261mJDm5yzAMo7HihV+xejjiZxiGkTSB8HNyl2EYRs/KF36LGZmcQCbLUT/DMAzQDMIvN2PJsPAzDMMATSH8ct9dtnsYhmGAphB+jvgZhmH0NIHwy+0XOeJnGIYBmkD47RYWfoZhGD0rXvg1q4dr+RmGYQA0hfBzcpdhGEZP0wg/9+RnGIZRaALhZ6uHYRhGz8oXfk7uMgzDFLDyhV8r5+SIn2EYBmgK4ZdWD0f8DMMwQFMIPyd3GYZh9Kx44bdbOLnLMAyjx1DhJ6IWIrqPiE4T0SkiupaIAkT0OBGdVf9vNXgMsFtMXMfPMAyjYnTE/0UAjwghtgHYA+AUgE8BeEIIsRnAE+rvhsKbsTAMw+QxTPiJyAfgRgDfAAAhREoIEQJwF4B/Ux/2bwDeYdQYJLz9IsMwTB4jI/4NAMYA/CsRvUxE9xKRG0CnEGIIANT/O8r9MRHdTUSHiOjQ2NjYvAbisJo5ucswDKNipPBbAOwD8M9CiNcBiKEGW0cIcY8Q4oAQ4kB7e/u8BmK3mNjqYRiGUTFS+PsB9AshDqq/3wflQjBCRF0AoP4/auAYAEiPn60ehmEYwEDhF0IMA7hMRFvVm94A4CSAhwC8T73tfQAeNGoMEoeFk7sMwzASi8HP/zEA3yEiG4A+AO+HcrH5PhF9EMAlAL9m8Bhgt5oQTmSMfhmGYZhlgaHCL4Q4AuBAmbveYOTrFuOwmjEWSS7kSzIMwyxZVvzKXYDr+BmGYfQ0h/BbuI6fYRhG0hzCz3X8DMMwGk0i/FzHzzAMI2kS4Vfq+IUQiz0UhmGYRadphB8Akhn2+RmGYZpC+GVP/iQneBmGYZpD+HkXLoZhmDzNJfyc4GUYhmkW4eftFxmGYSTNIfwWjvgZhmEkzSH8bPUwDMNoNInwq1YPl3MyDMM0i/BzxM8wDCNpEuGXyV0WfoZhmKYQfrua3OUFXAzDME0i/LyAi2EYJk+TCD9bPQzDMJImEX6Z3GWrh2EYpimE32o2wWwijvgZhmHQJMIP8PaLDMMwkuYRfqsZSU7uMgzDNJfwc8TPMAzTRMJvt5q4nJNhGAZNJPwOixlJTu4yDMPMLfxE9EQ1ty11HFZO7jIMwwCApdIdROQA4ALQRkStAEi9ywdg9QKMraEoHj9H/AzDMBWFH8CHAPwRFJE/jLzwhwF8xdhhNR6H1YxwIr3Yw2AYhll0Kgq/EOKLAL5IRB8TQvyfBRyTIbDVwzAMo1BNcneYiLwAQESfIaL7iWifweNqOA4LWz0MwzBAdcL/F0KICBFdD+B2AP8G4J+NHVbjsXMdP8MwDIDqhF+GyXcC+GchxIMAbMYNyRgcVhOXczIMw6A64R8gon8B8G4APyYie5V/t6SwW8y8gIthGAbVCfi7ATwK4M1CiBCAAIBPGjkoI/A6LEhnBWZSheIvhMBIOLFIo2IYhll45hR+IUQcwGsAbieijwLoEEI8ZvjIGkzQrbhTk/FUwe1Pnh7FdZ97ksWfYZimoZqVu38I4DsAOtR/3yaijxk9sEbTqgr/VKxQ+M+Px5DJCfRPzSzGsBiGYRac2RZwST4I4GohRAwAiOjvAPwSwJy1/UR0AUAESoI4I4Q4QER7AXwNgANABsDvCSFeqGv0NSAj/oki4Ze/TxbdzjAMs1KpRvgJ+coeqD9ThceW4xYhxLju988D+CshxE+I6A7195treL66qBTxT0al8CeNHgLDMMySoBrh/1cAB4noAfX3dwD4xjxeU0Dp9wMAfgCD83iuqqkc8SfL3s4wDLNSmVP4hRD/QEQ/A3A9lEj//UKIl6t8fgHgMSISAP5FCHEPlP4/jxLRF6DkGF5f7g+J6G4AdwPAunXrqny5yvgcVphNVBLxS8Evvp1hGGalMqfwE9E1AF4RQryk/u4loquFEAereP7rhBCDRNQB4HEiOg3gVwH8sRDih0T0biizhzcW/6F6kbgHAA4cOCCqP6TymEyEVpe1JLKX3j5H/AzDNAvV1PH/M4Co7vcYqmzZIIQYVP8fBfAAgKsAvA/A/epDfqDetiC0umylEX+Uk7sMwzQX1Qg/CSG0iFsIkUN1MwW3rrmbG8CbAJyA4unfpD7sVgBnax10vbS6bQV1/MlMFtFkBgALP8MwzUM1yd0+IvoD5KP83wPQV8XfdQJ4gIjk63xXCPEIEUWhtHu2AEhA9fEXgqDbhrOj+cmLFHsiFn6GYZqHaoT/wwC+BOAzUJK1T6AKsRZC9AHYU+b25wDsr22YjaHVXWj1SJunu9Wp/cwwDLPSqaaqZxTAbyzAWAwn6LZhKp5CLidgMpGW0N3c4cXlyVEk0lk4rOZFHiXDMIyxLLsum/Oh1WVDTgDTM8oWjHLR1uYODwCu7GEYpjloKuEPegoXcUl7Z5Mq/FzLzzBMM9BUwt/qUts2xPO1+xYTobfNrf3OMAyz0qmmLPPjZW6eBnBYCHGk4SMykIBszSwbs0VTCLht+ZbN3K+HYZgmoJqI/wCUyp416r+7oTRV+zoR/YlxQ2s8xcI/EUsi4LZpt3NlD8MwzUA15ZxBAPuEEFEAIKK/BHAfgBsBHIbSXXNZUCr8KbR57Pk+PnEWfoZhVj7VRPzrAOgVMQ2gRwgxA2BZeSMOqxkumzlv9cQUq0fp42PjRVwMwzQF1UT83wXwPBE9qP7+NgDfU9swnDRsZAah79cjPX5AqfFnq4dhmGagmgVcf0NEP0a+LfOHhRCH1LvfY+TgjCDosWEilkIyk0UkmUGbWuLZ6rZyxM8wTFNQTVXPFwH8pxDiiwswHsNpdSmrd6XIB9x2AEDQbcep4fBiDo1hGGZBqMbjfwnAZ4joHBH9PREdMHpQRhJ0K16+tHWk1RNws8fPMExzMKfwCyH+TQhxB5S++WcA/B0RLVgr5UbTKoVfFXlp9QTcNoTiaWSyucUc3pLi6OUQHj85stjDYBimwdSycncTgG0AegGcNmQ0C0DAbUM8lcVQaEb7Xf//VDy9aGNbanz5qXP4zI+OL/YwGIZpMHMKPxHJCP+vAbwCYL8Q4m2Gj8wgpMDLvvxB1ePPCz/bPZLRcAKjkSTSPAtiGogQAh///hH88rWJxR5K01JNxH8ewLVCiDcLIb4phAgZPCZD0Qu/xUTwOZX8dpBX75YwGklCCGB4OrHYQ2EWkK88dQ53feXnhj3/TDqL+18awHPnxgx7DWZ2qinn/BoRtRLRVQAcutufMXRkBiGF/7XRKAJuG9QdwhDwFK7qbXZyOYHRiLI+b2g6gbUB1yKPiFkojlwO4ejlEELxFFrUxoaNJJpQtjudSfFMcrGoxur5fwA8A+BRAH+l/v+/jR2WccgOnQOhGe0iAAABFzdq0zMRSyGbU7ZaHpqeWeTRMAvJaFiZ4Z0aihjy/BF1n+uZdNaQ52fmphqr5w8BXAngohDiFgCvA7Bs52hBndi3eezaz61aHx9O7gLAaCRv7wyEWPibiWFN+I1Z1xJRI/4EC38JiXQW9z7bZ3h1YTXCnxBCJACAiOxCiNMAtho6KgPxO60wKe5OQcRvNZvgc1g44lcZDeffh6EQe/zNQjYnMKZafKcNWtCYt3pY+It5+swYPvvwKbx8OWTo61TTq6efiFoA/AjA40Q0BWDQyEEZiWzINhFLFQg/AAQ9dt6MRWVEjfp8DgtbPU3ERDQJ1eEzzOqJJpVZNVs9peS3hTVWh6pJ7r5T/fF/E9FTAPwAHjF0VAbT6rapLZkLhb/Vxf16JDKxu2dtCwY44m8apM3TG3Th1ZEIMtkcLObGbtQnrR4W/lLke2P0NrA1faJCiKeFEA8JIZa1OuYXbdmLbrez8KuMhBMIuG3oCbo44m8iRlSL7+atHUhlcjg/Hmv4a7DHX5lIQo34DV5P1FR77kpkBU+J1cP9ejRGwkl0eO3o8jsRiqcRT2UWe0gLQi4ncO+zfdoXsNmQFt/NW9sBACcNSPBGk+zxV0JeFEMGdxBoTuFXLZ5iqyfgUTp3CiHm/RovXpjEuVFjPNKFYCySQIfPgdUtytKNwSaxe44NTOOzD5/CE6dGF3soi8JIOAETAddsCMJqJkN8/iiXc1ZEi/iXktWzUqgU8QdcNqSzAuHE/KLbRDqLD3zrRXzqh8u3z81IOIlOrx2r/U4AzVPLf2kyDgBNHfG3e+1wWM3Y1OE1pKRTvrds9ZSyJD3+lUJP0AWn1YwOn6Pgdq1fzzzf9MdOjiCSyODwpSmMR5dfeWg2JzAWTaLT58DqFlX4myTivyyFP9kc1lYxI+EkVqnfiyu6jBJ+tnoqId8b9vgN4F37uvH0J2+Gx15Y1NTuVZK9Q0W9aSKJNN79tV/i5GB1X4IfHu6H22aGEMCTRZbBqaEwDl+cmsfojWdSXbXb4bOj0+cAUfMs4ro0oQh/dJ6zvuXKSDihBUTbu3wYjSQx0eDgRW/1NMJWXUnI2RBH/AZgNlFJtA8AV3T5AACvDE4X3H7owhReuDCJF87P3U1wJJzAs2fH8P7r1mNNixOP6frZZ3MCH/n2YXz6/qVtAckEX4fXAZvFhHaPvemsnmjTRvwJdPqUAEh+Hxrt88uoNieAFHd+LUCL+Fn4F452rx2r/Q4c6y8U/iPqKrrxKjp3PvDyAHIC+JX93bhteyeeOzemTWmfODWCCxNxjC1x+0e2a5AC0NXiLJkFrVQ04W/CiD+ZyWIqntZZPVL4G2v36N/bBDdqK0DmF8OJjKFtG1j4i9jV7cex/lDBbUfV3+fy64UQuO9wPw70tGJ9mxu3be9EIp3DM2eV1kbfeO48AKXn/1Le6Uu2a5CzotV+R1NYPelsTpvZNKPHX/y5B9w2dPrsjRd+3XvLlT2FRBJpOK1mAEBoxrgCAxb+InZ3t+DCRBzTah2tEAJHtYh/duE/1j+Nc6NR/Mr+bgDAVesD8DksePzkCE4MTOPg+UlsaHNDCOOTN/NBLuJpV5vYrW5xYiiUWPF+7GBoRmtX0IxVPdLi69TZoFd0+Rpeyx9OpNHqsgJA06wPqYZkJotkJoeeoNIC3Uifn4W/iN3dfgDA8QHF7rk8OaNtxziX1XPf4X7YLSbcubsLgNL47dZtHXjy9CjueaYPbpsZH7ppA4ClveHLSCSBoNsGm0U5Pbr8Dsyks1ofkZWKtHk8dktTevzygi8tPgDY0ulF31isYRd9IQSiyYxWSMERfx7p769T974w0udn4S9i1xpF+I8NhADkbZ7NHZ5ZI/5EOouHjg7i9h2r4HNYtdtv274Kk7EUHjo6iF87sBbr2zwA6hP+4ekE/uf3jxoeJY2Gk9oXE4BW0rnS7R4p/Nu7fGU9/l++NrGiL36yT88qXcTvd1qRyuaQzDTGmoynshAiX0HXrLX8L12awof+/VCB5Vss/EZuA8vCX0SLS+lPc1xN8B69HILdYsJ1m9owHk1WjHx+emoE0zNpvPvA2oLbb9raDpvZBCLgA9etR1BdLVxPff8jJ4bww5f68eIFY8tBRyOJgul+s9TyX5qMw2Y2YWOHuyTiT2ay+O1vHMQ//fTMgo/r4kQMN3z+SfRPxQ19ndFwAjaLCX5nPnCRJc+xBs2A5Pva4VXOr2bdheu5s+N49JWRgkIPaS9Kq8fIvUFY+Muwa41fq+w52h/CzjV+rPI7kEjnEKuw6OT7h/qxpsWJ128MFtzusVvwqwe68Z6r12Fd0IU2tTFcPcIvy+rOjhjbCkJf0gcoyV1g5a/evTwZR3fACZ/DqkVfkul4GpmcwFOnF76Vw6vDEVyenDF8/Yf83OV2pADg1oS/MZG5FLdmt3pkNK+f+WsRf9Bd8BgjMFT4iegCER0noiNEdEh3+8eI6FUieoWIPm/kGOphd7cfA6EZjIQTOD4wjT3dLdpuXeORUsEeDM3g2bNj+JX93TCZqOT+//edu/DZd+wCAPicFlhMVFff/1PqxhhnDBR+uRGHPuJv89hhNdOKb898aTKOdQEXPHYLkpkcUjp7Q1o8FybihnSsnA0ZJb82GjX0dYbDiQKbBwA8dnPBGOaLFDdZONCswi+bsOl9fHlRbPPY4LKZl31y9xYhxF4hxAEAIKJbANwFYLcQYgeALyzAGGpid3cLAOCHL/Ujkc5hz1q/ZtFMlNmh6/6X+iEE8GtqNc9sEBGCHlvNqyEz2RxODyuCf2bEOAGYiCkbcXToPH6TibDK71hyEf8Xf3oWn36gcYvhLk2owu8otTf03v5CR/1yHK+NNe6Ck8xk8e+/vIDHdQsMR8PJkoWNWsTfoLySZvWoM8pEk7ZtCKnRvF74ZQ2/z2FFq8tmaOXfYlg9HwHwOSFEEgCEEEuuDeLONX4QAd95/hIAYE93ixahjEUKPwwhBH5wuB/XbghirZqUmYs2j72qxWB6zo/HkMrk0Oqy4txo1LDSyuJabkmX37mkPP5EOot7n+vDj14e0DaFnw/T8TTCiYwW8QOFUa6M0KxmwlOvLuwpG1VtltfG5n/Bz+UE7n+pH7d+4Wn8xYOv4BM/OKolWEfCCXR6yws/R/yNRVYKTsRKrR6fw4qA27asI34B4DEiOkxEd6u3bQFwAxEdJKKniejKcn9IRHcT0SEiOjQ2trB7u3vsFmxs92AgNAO/04qeoCtv9RRF6i+cn8TFiTh+7cDc0b4k6LHXHPHLWuo7d3chmsxg0KCVtOVquQHF5784GWuIyDaCn706hkgig3gq2xDrRVb0dLe64FUjfr3PLyP+m7d24GDfZMOSndUgtyrsG5//+/+J+47i498/ila3FX/y5q2Ynknjv44OIpJII5bKYpW/cHOihid3pfA3ucefj/hLk7sehwWtbhsmDezJb7TwXyeE2AfgLQB+n4huhLLdYyuAawB8EsD3SZ9NUhFC3COEOCCEONDe3m7wMEvZrZZ17lnbotkzQKnwf/9QPzx2C96ys6vq525z22qO+E8OhWE1k/Y6Rvn8cstFvdUDALds68BIOImvPf2aIa9bKw8eGYBN3RKwuLdSPUjhVyJ+papFv4hLCv87X7cGqWwOv3ht7r5NjUImVlOZHAbnWVJ7sG8St23vxEO/fz0+ctNGbO7w4NvPX9TV8FewehoV8avPIwOppdSh89RQGOkFWlEfKrO3biSRgdtmhtlECLisyzfiF0IMqv+PAngAwFUA+gHcLxReAJAD0GbkOOpBLuTaq/5vNZvQ4rIWCH82J/CTE0O4c1cXnDZz1c8d9NgwEatcGlqOU0MRbOrwYrvaP8Woyh4Z8bcXCf/b96zG2/asxj88fkarLhFC4OvP9OE99z6/oCsww4k0njg9indf2Q2bxYRXquyaOhtS+NcGnFrEHy3j8d+6rQMeu2VB7R79zOPcPO2eyVgKPQEXTCYCEeG91/TgaP+05vV3FFk9Hpt8Lxpb1eN1WOCwmpZMHf9IOIE7v/Qsfnx8yPDXyuaEdj6NRwuTu151DVCLa5laPUTkJiKv/BnAmwCcAPAjALeqt28BYAMwbtQ46uXK9QEAwNUb8uWZQbetoPxqYGoG8VQW+3paanruoMeORDqHeA3RzqmhMLZ3+dDqtqHNYzcswTsSTqLNY4O1aINtIsLfvnMnuvwO/MH3Xkb/VBwf+vfD+Nsfn8LPz03g0VeGDRlPOR49MYxUJodf2deNbau8VUf8lyfj+MyPjpftk3RpMo6A2wavw6old4uFXxErM67f1IanTo82PM9yYmAav/edwyVRZ0y30nU+lT2JdBYz6SxadRsQvXPfGrhsZvzLM8pMbpW/UPhdalVPI60ep9UMi9kEp9W8ZKye/qk4cgIYK1O112jCM2nIU6c44pdBR8BtQySZKagsayRGRvydAJ4joqMAXgDwsBDiEQDfBLCBiE4A+A8A7xNLsAnMjtV+/OJTt+K6TfnJiJKUzZ8YfePKl1Cuxq2WSvmCSoxFkhiLJHFFlxcAsKXTY1jEPxZJoL0o6pP4HFZ86Tdfh+FwArd84Wd48vQo/uKt27GmxYn7Xxqo6vllI7v5JAsfOjqIdQEX9q5twY7VfpwYCFclwk+cGsG3n7+ECxOlC6H6p+Jact5rL+/xy4VNt2xrx9B0Aq82+DN46vQofnx8WLPbJNFkBmtbnWh1WedV2SMT1C2u/AItn8OKd7xujXZfscVnNZtgs5gauoBLipvTal4yVs/wtPKeF6/fMAJp85iouKonrb038uIcmjEm6jdM+IUQfUKIPeq/HUKIv1VvTwkh3iuE2CmE2CeEeNKoMcwXuWJV0uYtrMaRScX1be6anjefL6juQ5XdEaXNs6XTi7MGVfZMxFIlexHr2beuFZ+58wr0BN34zw9diw9evx7vfN0a/PzcOEbDcyecXxuL4RM/OIr/OjpY1/hGIwn8/Nw47tq7GkSEHat9mJ5JV9VOQibLyi2MkTX8ACpG/FIwb97aAQB46nRjiw5G1HbY4aK2ENFkBm614GA+lT1SZOTWo5L3Xt0DQLnguYs2JwIa27soksho76/DZlzEPxpO4N5n+6r+jsh2FQvRo0mef2sDroIiDyXiV84x+RlNGbR6l1fu1kC7x16wgOvCeAxeu2VWoSyHXL1bbWWPFH7ZH31zpwfxVNaQ3jlTsRRaXLMfz/uvW4+ffvwm7O9pBaDYBTkBPHhkbjGXawGG66xK+u+jQ8gJ4K69qwEopbcAcGJgbp9feqbFza8y2RwGpmawLqBc6J1WM0xU2Dc+FE9pEX+nz4H1bW4cV/s5NQqZYC0W/pgaJW9s96BvXhG/ctzFn+/21T5c1RuoWI7stptrivhfHY5UFNxIMqPNqJxWs2Ee/73PncdnHz5VdcWXDFoWYh8G+TlsbPcgnMho1p7e6ml1K+eaUY3aWPhroM2j+G7yZO0bj2F9uxtlipJmJb8YrPqIv8vv0KZ/WzoVy+esAT7/VDyNgM4KqIaN7R7s6fbj/pfntnvkhi5ys5daeeL0CLat8mJTh/IebFvlhdlEOFmFzy8jreKk2dB0Apmc0CJ+IiqJcvVWD6A07Wt0nkUm1ovthmgyA7fNgo0dboxHk1rL8FqRteNSVPR89b378C+/vb/s37ltlqqTuycGpnH7Pz2Dp8+Unw1FdQlMIz3+J04pyeqLk9X1N1rIiF/aahtUp0Cej/rkrrb/t0GLuFj4a0B681Kwz4/HarZ5AJ3VU2Ui6eRQWIv2AWCLKnqNLunMZHMIJ9JzRvzleNe+bpwaCs+5aYeM9GV0WytD0wlsaM+/5w6rGZvaPThRRWWP/BIVr4jsn1JmIWta8hGvt6hfz/RMpkD4t3R6cX48hmSmccIlhT+cKLV6PA4LNqi5pNfG67vgyOMvtnoA5dyuFPF77JaqI355DM+dLV+vEU1mtLUBTpsxHv+F8ZiWC7lQZcQvz8uF2IBHXoA3dqidelU9CScy8MnkrvoZccS/BAjq+vUk0orVUo/w2y1meB2WqiL+RDqL18Zimr8PAH6XFR3exlf2TKvVBq01RvwA8LY9q2ExER6YI+of0oS/voh/MpZC0F2YgNyx2ldVZY/sdlgc8cske4euMZ0S8ec34wnPpOF35gVzyyovsjnRsL49skcSUBjxCyEQU8VSCkW9lT3yuGu9sLvtlqpbNsiL1vMV9qcu8PitZsyka69a+e7BS7jry8/hb/77JH56cqQkSn9CbalhMREulknkl0Mm1KMLsAFPKJ6CifRdOFNIZrJIZXLwOfPlnIBxm7GUZnKYirTpFnE5J80QovbEbv657FVV9ZwbjSKbEwURPyATvLNH/JlsDnd86VlMxdPo9NmxyufAR27epHnzxeStgNoj/oDbhpu3tuPBIwP40zdvg7lMszoAGFY9/noi/nQ2h1A8rU2DJTvWKDbTaCRRUoeuJ79asvDLLT+HoO55vQ6LJsAz6SxS2VxRxK+I8KvDEWxbVfjZ1MNENKnt/qX3+GfSWeSEIr5rW52wmqnuyp6peBpum1nbYKdaPHYLLlfZEjo8o7xnJwfDJfYYoHjoHru+qqf2CPvBIwM4OxrFqeEIvvHceaz2O/DYx2/SnveJUyPY0umBxWTChYm53yshhBbxL1Ryt8Vl09pWTMRS2rkmPX6bxQSv3aJ9JxsNR/w1oC/DlEm2DTWWckqK1wRUQkayspRTsrnTg7MjUeRmWcI/HE7gzEgUvUEX2j12PHNmHD98qb/i46UwttZh9QBKO4mRcHJWC0pG/BOxZM2rJKVVUZxM37FaEd7ZFnIJIbRp82RRo72JaApmExUct8eR9/jlYhu9iG1o88BiooblWfQXQr3dIMfgsVtgMZvQG3TXXdkTis+duC9HLcld+V7lBHDowmTBfbmcQDSVtzPq9fjPjkbx9j2rcewv34Sv/NY+DE4n8E11P+twIo0Xzk/i1m2d6G1zVRXxhxMZbRwLk9xNo8Vp1QKYyWiyRPgBoMVtZY9/KSAX0YxHU9oUv7etusZsxQQ9tqoi/iOXp+F1WNAbLJxZbOn0YiadnTUSk971H7xhM/71/VdhfZt71koiKYz1Cv9mNfdwcZYoazicgN1ighC170kgL5SBIqtnuyr8J2cR/pl0VttFqrgHyng0iYDbVtBS22O3aCJQTvhtFhN629wNq+Uf1llf+ohfjkFGs0plT33CPxlPlcyWqsFtt1Tdjz88k4bNYoLNbMLB84XCH0tlIES+XLYej388msRkLIXNnV44rGbcubsLt+/oxD3P9GEylsIzZ8aQyQm88YoO9AbduDwZL7tgT4+s6Gnz2Bse8QshSiqcQnGlNLjFZQOptfzaimZ7/hwLuGzs8S8FHFYzPHYLxqNJnB+Pot1r17LwtdLmsVfl8R+5HMLetS0lff7397SCCPjwt1/C5QqVC1L4u1uVi5PSDrrya4ZmqfqohnWqZ1kpyppJZRGKp7USzFrtHvklCBZF/D6H0kjvxEBln19OmYnKe/zBIkH0Oixa5D1dZuET0NiFdDLn4bVbCpK7UnA14e9w4+JEvK6eMlPxdMkxVINH9firqYkPJ9IIuGzYu7YFz/cV+vz52YsyBofVjESNHr+cTUqrDQA+8aatiKcy+OpT5/DkqVG0uqx43bpW9AbdyOQEBufoKisvuhvblZ3XGrk+5h8fP4M7v/RcwW1T8RRaXTZtljlexuoBFMuVI/4lghKpp3BhPF63v688jx1T8dSs0Ug8lcGrw2HsXdtSct+WTi+++btXYmAqjrd9+bmyVRRyq77VLQ7tNWeLsqfmafUofcStFUvo5Bdsj7rfQa0J3nJevGTHap/WwbQcUuxX+51lhD9V0ptIH/GHykT8gPIZXJyMN6QWfTScABGwod1dkNyNqAlmty7iz+SE1luoFkKq4NSK226BEKiqxUh4JgOf04JrNgRwYmC6oNFdtEjcnFYzUtncnBG5HmmtyZJmANjc6cW79nXj/z5/EY+fGsEtWztgNpGWPJ3L55f+/sYOD3KisR1DHzs5gpND4YJzJBRPw+/Kl21ORnURv4Mj/iVJm7qIq288hvXB+oW/zWODEJg1eXO8fxo5gbLCDwC3bO3AQx+9Hh1eO37nmwdx5HKo4P7+qRl0+uywW8zaa84W8U/GU7CZTXDV0HCumJ6gu6LVI79ge9YqEX81K30LxqdF/PaS+9a0ODESTlSM1uTfbuzwlPRAGY8mtfyNxGO3YiadRSabK2v1AIr4CKEk4OeL0iPJjhaXrcDqKYn425VI98xw7TONqVj9Vo8ylrltkHAiDZ/Diqs3BFWff0p3nxrxq8Ivz7NEDf1ozoxE4HNYSlpL/NEbNwNCqRq69QplZXWvGpjNZj0C+Yoe+d42yucPxVPa5kly9i1vlxfggFsR93CliJ+Ff2nQ5rHhwkQM49Ek1rfPI+KvYu9dKeSVhB9QTu7v/o9rkBPAC0UldP1Tcc3mUcZuL1iAVkwoplgBtS5I09MTrJxQGw4rJ/+O1X6YTVSz1TMRVcrgWpyldsVcje+mtNWSymcmE9lCiLJWT34XrqwmxL4ywg80Zj2F3PbQ5yxcPyBLSuV4tnV5YbOY8HLRRX4ulDUamTqtnuq3Xwwn0vA5rdi3rhVWMxWUdcq/lyt3Harw1+Lznx2JYnOnt+Qc7W514X2v74HLZsaNW5Q27h1eOxxWU9neTHqGpxPwO61a0UCjavlf0OU4ZC4ulVH27ZYl00G30qlXO8f0Eb/bhlgq29C1IhIW/hpp89i1ypT5WD3yJJstAj9yOYS1AWfZCLd4TG0eG14bLYxs+qdm0N2a7zckxa1SbmGqzuSfnp6AC4OhmbJdBeX7tqbFiXaPvWarZ0KNWMvta6xVSFQ6Nhnxq1GdXMQVS2WRSOfQVhRBao3akmlMz6Rhovxtkt6gCzazqSEJXrnRuddR6PHLFbNuVXztFjN2r/GXVMzMhbSr6rJ6bNVvuD49k4bPYYHTZsae7hY835cfZ97qya/cBVC1VSaEwJnRSIG/r+dTb7kCP/vEzZp4EhF6y8xAixd1yYuu1o67KOL//e++hK/+7FzN3v8L5ychT1WZh5NN1/xFEX+kaDYE5D+rkAElnSz8NaK3BDbM0+MHyu/hK1ESu+Vr7ovZUNTAK5PNYXg6USD82srjCrMMpb64vsSupCfoRk6gbB+h4ekEWlxWOG1mdPrsGKmxBe6EWn1TjrYKG+VIJuNpEOUv1pPqBVe+F8VWj74nfyiuRLHFFxyL2YQN7e6GlHSORpT9bn0Oq1YLD+TtFX21x/6eVpwYCNeUW8j36akvuQtUGfHrVjhfsyGIEwPT2t8Vz16k8FfrqY9HUwjF01r1WDFmE5VsGdoTdBUssnv6zBhu/sLPCi6cI+EEOv0OLemsP850NoeHjw3h84+8ij/6zyM1vecHz0/iQG8AdosJl9RZh1ZAoYv4QzNKcCE3YZEEDOzXw8JfIzIyJMpXsdT1PHN06BwJJzA0nZjV5tGzqaNQ+EciSWRyosDqCc4xy5iKp+tO7EpmS6gNTSuRFaDs6VuPx1+8alciSzxni/j9TquWxJUR/7gm/OWtnkgiU3YhkmRLpxev1uG360lmspiMpbSoM5XNaQITTWRgIsBhzX9V9/e0IpXNzVrFVIxctGakx5/LCURUqwcArt4QQDYn8KIqspGi0lSnTTmmaq2es1pFT3nhL4dS0jmjbVn5kNpIUL+DmrLXsF0bl95qk0K9p9uPB48M4je//nxVPfsjiTReGZzGNesDWBdwaVbPVFHJdMCt5PouT8ZLKgRbDVy9y8JfI23qF6e71aklTevB57DCYqKK0ffLl0IAZvf39Wxs92AqntaEr1/bQ7Y04h+r8JqheKquVbt65MXwUhlfdXg6oW300ekrtXq+/+LlWWvUJ2IpBCp0QtVsrIoXtRQCLlvJl2ksIheFld9rNjqn8Ct7M8+nX70Ukk6fXRNNaffI3jZ6T3ufuvJa7oRWDfOp2NKEf45VtrFUBjmR96kP9ARgs5i0irNi4XfUGPGXK+Wci56gG6lsDkPTM0hnc/ip2rztkPreZbI5jEWSWOV3lN15Tc6UPnjDBnztvftwaiiMv3345Jyve/jiFHJC2chpXcCFS5PKDFgWc8jzKaCedxcmYvA5C63EdUEX7r5xQ8ksphGw8NeIjPhr3XylGJOJEHBXXsR15HIIVjNpq1LnQiYtZdRfXMMPzB7xCyHUiH9+Vk+7xw6XzVw2wTs0nUCXFH6vA1PxtJa4moql8Cc/PIbvHrxU8bknokntwluM1viugnUmbSxpdcgIWFptlayeSDKD0BwRP6CsJq0XeQFUrJ7CqFPf1EzS5rFjfZtbE69qaLTVE0mk8YNDlwt8b1mZIgXMaTPjqt4AnlE7dSpdRvN2Rq1Wz5nRaMGsrRp6dWtLfvnaBKZn0uhudeLli1PI5QTGoynkhNJqO3+xz3vqUzpr5s07u3DXnjV44tTonDtjHTw/CYuJ8Lp1LVgbcOHyZBxCCEyrHr8MsOT5fHlypiTi7/I78ek7rsCmjvlpTTlY+GtECsR8/H39c1WKUI9cnsL2Lp8WFc2FTFrKBl5S+GUNPwC4bBa4bOayF5twIoNsTszb6iEirAu4ShJqqUwO49EkVvmUGYjc1HtUrew50h8CUDnxnMooVSnFq3YlLpsFTqtZ8+6LmYwpPX6sZhN8DosWAY9H5GrgIqtH+r2JjNqgbXbhn6u88ifHh/ClJ86WvU/b6Nzr0KJlWeURUzdhKWZ/TyteujhVdcJxflZP6faL/3V0CJ+871hBxUy5ypQbt7Th7GgUg6EZRBLpguSl3Kc6UYPVs6XTU1PVWY/6Pb0wEcNPTgzDbTPj927ehEgygzOjEe2i2+lzaO+z/gJXPFN64/ZORJKZgoqdcrxwfhK7uv1w2SxYG3AhmsxgKp4uuJAA0GawqWyuoJTTaFj4a2SVz4EWl1Wbbs+HoEdZtVdMNidwvH+6apsHUCpl7BaTVlPePxUvqOGXKBebUuGfb58ePT1BV8kiLvkFkxG/7IQp+/IfVcsTK82A5BeweNWuHmUT+/LCX652Wr6e32ktaVyW34UrPavVszbggt0yd2XPtw9exD/99EzZfQjke7PK79Ci5YKIv4wg7O9pxUQsNWepoiQUT8FmMWlRdi24y2y4LhfjjZZpNaEve71pi1JT/+zZMXXbxfx9tUT8QgicUUs5a6HL54DNYkLfWAyPvTKMW7Z14LpNyj7ahy5MacexSn2c3WIqKOfUvhfqBfP6TW1wWE14/GTlPaZnUlkc6w/h6vXK68h9Hi5PxjGlrpWRx66/ENfbBaAeWPhrxGkz4/BnbsPbdnfN+7nkYrBizo5GEEtlsXddS9XPZTJRQWWPUspZmnyuJI5an5462zXo6Qm6cWkyXtBAblgnbkA+4pfRrlyzUGkGNNuqXYlSE13exprULV5q1Qn/RCxZdgc1l9UMormTu2YTYeca/5y2y7nRKHICePjYUMl9I+EkrGZCq8uqffmLPf5iDszi8wshcNeXn8N/vpi3zZQ2AfWt0TCZCC5bYaO2MfUCpi9OKLfQbUunB6t8DjxzZlxpyaw7llqEfyySxPRMGptrtD1MJmUG+uCRQUzEUnjLzi6sC7jQ5rHjpYtT+YjfrwQiXoeloJxTzpRkhO60mXHD5nY8fnKk4mzr5UtTSGcFrl4fAACsVXd2uzQZx3S8cK2MPtDiiH+JYzbRvBY5STp9DoxGEiUdNo9oid3aZhUb291ay97+ULwgsSsJuu1lqxLyG3HPP+JfF3Ahlclpe8gC+Rp+LeJXfVq52nauiH+2VbuSYIXZjGzQJo9NvxR+PJIq8fcBRTA8NguGpxPI5sSs3vgNm9twrD9UsfpieiatXeAeKrPX8EhYaSdNRDqrRxGfWAXh39jugc9hweGLpZbDWCSJo/3TeFLtSw/Mv2LLXbQZi7To9J+X5vHrIlciwg2b2/DcuXFMz6QLxK2WBVxnyrRqqJbeoLJzmd1iws1b20FE2N/TgkOq8JtNpG2HWrzzWrmZ0m1XdGJwOlGxRchBtX5/f6/y/V2rBmCX1Ihf/zlYzSbtQsnC3yR0+R1IZ0VJlHp+PAabxYSeCjsiVWJjuweXp+KIpzIYCiXKCn+7t3xUPN8+PXpkJ9EL43kbQvbhlxF/q8sGq1lZvat8IdIIqpF4uVbT+c6cc0T8ZWYM0leVddH65lfl2jVIvA6Lth6hUsQPADdsbocQwM9fK7/rlLTfDvS04uVLoZKmenLxlnxNAFrvlmiivMdvMhH297SWjfjPqbO+U0N5+2kqVl+fHkmxIMrKMH0Qkbd6Csd745Z2TM+k8cpguGzEX01tvKzo2VxDRY9EJnhv3tquvZcHegK4NBnHiYEwOrx2bY2GpyjiLzdTuvWKDhABj58cKft6L5yfxBVdPu0C6Fb35e6fUs5zf1EQIWexPrZ6mgMpgsUbjw+q1S/lVqjOxsYOD4QAnu+bKKnhlwTd9rLiKiPgctvy1Yqs5b80mU/wDk0n4LFbNCvDZCJ0eJVafmnz3Ly1A5mcKNl6EMgnfWfb2D7gUS4cxVPw4tppeYGR7RoqPaenSuHf0+2Hz2HRqleKkQn3j9+2BUBp1D8Szpe5utSql7msHkDx+c+MREv24JWzvkuTcU2sp+Kpedl4xT35y0f86gKtovFev6kNREruSh/VWs0mWExUldVzdjSCFpdV27ykFmSC9y078/aszNH9/Ny4ZjsCSj4jUpDcLZ0ptXns2L+utazwp7M5HLkcwpW9gYLb1wZcmtVTXDkngxmO+JsEaXsMTReuch0KzWC1vzRanwtZ0vn0q4oAlbV6PDZkc0Jbwi8JxdW2BA04+br8jpJt7/Q1/JIOnx0jkQRevhSC02rG6zcqybByi9omokmYTTRrVNTmtiOVzZX0WsnnL/IefzKjNF8LJzIVI36P3YJBVfiL+/TosZhNuH5zG545M17W9z07GoHdYsLVG4LY39OK/yoS/tFwUts5TLF7lN2/hBBzCL8iLi9dKoz69Vszvjqs2BFKD/h5WD22fE9+pQyyjPDPZLQNY/S0um3YrXZk9dgL30dlF67ZSyMnYyn8+PgwDvS01mWx3nZFJ37zqnV4045O7bada3ywWUzI5IS2qBBQzv9YkdVTzuZ74/ZOvDIY1s4PycnBMGbS2ZJd7ta2uspaPQALf9OhRfxFC5kGQzPoanGU+5NZ2dDmAZGyLB1A2Yi/UtsGuR1crbOMcljMJnS3OguEX1/DL+n0OjASTuLI5RB2rfFrkVc5n14mZ2cbX35Ho8ILR7GNJWc1sva+Ut7A47AinVWEvMU5u2jesLkdw+FE2U6d50aj2NDugdlEePue1Tg9HNGsi1gyg0gyUxB1eh1WhGfSSKRz2raL5djd7QcRcFQthZX0jce0z/nUUAS5nNAWsNWL3uqZjKeQUWeMY7r3WunMWX6sN21uU56n6H6HzYyZ9OwLw/7+0dOIJjP45O3b6hr7Kr8D/9+7dsFly7+27HkEQLPZgFJLS18UoOe27cpFRC4Ik8gk/4HeQuFfF3BhMJTAVDxVavWoM06vna2epqDNbYfFRFriE1BWEo5EkljTUnvE77SZsabFqZX4rS5z8ZAnWfHqXellNoqeoBsXdVbP8HSiQNwA5Qs3FJrBycEw9q5ryS8wK5ODGI+mZq3oAXQL1IoWcUmrR1/VA+R940pWj74pW/GXtRjZEfLpMnbP2dGoVo1yx64umCjfOiBfypkXH59TifgjRb1tinHbLdjY7ilp3fDaaBSv3xiEz2HBqaEwIgllRe18+jDpN1yXvr7LZi6oSpueSVecGd20VXl/ihvdKRF/Zavn6OUQ/uPFy/jd1/di66raE7uzIZOvnbqApNjjrzRT2tjuwYZ2Nx45UVjWeejCJNa0ONFVNGNfF3AhmxNIZ0vXysjzcrZZZaNh4V9ETCZCp89R4PGPRpLI5kTJiVMtciFXuRp+APkNnouj4tj8+/Toke2ZhRDKsvhosiTi7/A5EEspG5nv6W6ZtYncZCw5aw0/kG91XXxsskGbtkxe9bplc7XizpwSvcUym8cPKOsoNra78WzRhjjxVAYDoRlt9WW7147rNrXhPw9dxrH+UMHiLYnXbkU4kdb14q9ce79ztQ8nBvLVJTOprPZ627p8ODUUbkjiXl/VI/vXX9Hlw1g0qdlb4VmEf+/aVnz0lk24fceqgttn23c3lxP4Xw+eQJvHrvTbbzD71ynCr7d6PHarZhUKoViilQKiO3d14fm+Ce1CKITAoYtTuLK3tBqvO5D/Ppd6/IWJ/YWAhX+R6fI7Cjx++XM9Vg+QF/5yNg+g6wpaweppFOsCLkQSGfyybwKPnxxBNidKPH79DGDvuha1eqKCxz9LgzZJpRmDbNAmWwVIAZQRf6WEoYy0zSaCu4rNaW7Y3I6D5ycKqlT6xmIQAgX155+8fSvMRHjnV3+BLz5xBgAK+rH4nBaEZzKa0LptlQVh5xo/hsMJTXz6xpWL2cZ2D7Z3+fDqcER7P+aT3PXYzZoFIhdtbe/yIZXJ51TCiUzFHIzZRPjE7VtLGhsqVk95j//7hy7jaP80Pn3HNkMWN924pR3vv64XN2/t0G7zOixIZXJIZrJzrmZ/6+7VyAngJyeUtRmXJ2cwFklif1FiF8gv4gIAf5FtuKfbj9V+B1bXMcuvFxb+RWaVvzDiHwjle9bXw8YOJcFbLrELKJuYmE1UIq5KL/7Gfbk2qkL3W18/iI985yUASg5Cj/RW2712rPY7YDGb0OqylW1VPRmde68AeX+5i5re3w5oVo/0+CtU9agRf4uzuoVPN21pRyKd07pRAkpiF0BBv5Xd3S149I9vxF17Vmv96vU+s9dhRSSRLtujvZhdqk8t7R5Z0bOxw40ruryIpbI4ruYA5hvxJ9LKNon6iB+AZvcoEX9tUavTaqrYsuFbv7iAPWtb8I69a+oe92w4rGb85dt2FJxX8jOPJbO6/kbl37etq7zY3OHBfx9VhF9+7uUi/i6/ExYt8Cj8nh3oDeAXf/aGOWeVjWTh5hZMWVa3OLVVgESEIbVKoNgWqZZ8xF9e+GVzOL245hu0NS7iv3FzO/7lt/fDRASvw4I2j72k2ZSM+PeubdGEtVwtfjKTRSSZmbWUE1C+yB67pTTiL+o66nPIi18SLpu5IOmnR069q/1CXr0hAJvZhGfOjOGGzYqnfW40CouJ0FO0TaffacU//PpevGVXF14bixZEtD6HFeFEPuKvVNUDANvVJn4nBqZxy7YO9I1FQaSspUiqkbRsQTzfOn5A2bhmLJKE127RVqSORZLY0O7Rtl2sBafVXHaGF09lcGYkgo/eurkhiyWrRd+VtXj9Rznetmc1/vGnZzA8ncChi1PwOizYUma/ALOJsEYteJhvB9xGwBH/IrPK50Ayk9NWzg5NJ+B1WOqe2m5b5YXLZsbO1f6Kjwm6bVo7YkBZ2ZrSrWxtBGYT4fYdq3Db9k5csyFYtsPgKr8DVjMVREjBMvsCa2sM5rB6Kv99oU9rMpH2e6VSTiAv/NUm3Vw2C67bFMSDRwa1rqNnR6LoCbpKegFJbtveiQ/ftLHgNp9TqSyRLRBmE36vw4oNbW4c10X8a1tdcFjN2LrKCxMp6zqA+Uf8gFKFNBZJot1n17pkjkeVdSHRZKbmBKXTVt7jPzUURk7kZzQLhbYPQzKtFQXM9r146+4uCAE8fHwIhy5MYn9Pa8XKM2n3lNs6dKFh4V9k8rX8isUzUGcNv6TFZcOLf/5GvHnnqoqPafPYCyL+vLAu7Anpc1jx8B/cgPe9vle7Leixl7RWlkI+V3IXKGzAJgmVqZ3WFnPN8pyy5ryWKfgHrl+P0UgSD6pVO+fGohV3jKqEvOjLlhezCT+g+Pya1TMaxQZ1PYfDasb6NjfCiQzMJppX8lAv/KORBNo9du2iOR5NIpLMQAhULOeshKNCVc+xfuV4dncvrPB7CyL+uZPiG9RcyvdeuISzo1Gth1I51krhb2CAVS8s/ItMvpZfsXiGpuur4dfjLtq4o5i2oqi4kX16amVLp7eg+qitjNUjrZu5yjmVx9gLFhUVN2iTaP3QZ4n4ZfRXSxnk9ZvasL3Lh68/04dEOouLE/Ga+6lL8RxS8z2V6vglO9f4MDitJHj7xqOa3QcA21QfvqXM1pG1oN9wXW4T2eqywUSK8IfLNGirBqfVXLZlw/GBabR77SUlwEaT78qaKWmhXIm37unKt+Uok9iVvH3Parzv2p6Ks7+FZPFH0OTIsk0Z8Q+GEoZn94OeQnFsZJ+e+RL02DE9ky7Y6EIma+fadB5QL2qxQhsrmcmV+Koy2Tur8Ntr8/gBZeXt3TduwNnRKL71iwvI5kTN/WVkxD8YmgGRUi8/GztVO+TxkyNIpHMFwr9dCv8812joN1wfiyTR4bXDbCIE1KZ/02VaMleDq4LVc7x/WltgtZDoN50JxVMw0dw9dN66azUAwGIi7FFXKJfjmg1B/NVdOxs21vnAwr/ItKtfoKFQAom0svfq6joTu9XS5rEjnsoiri7I0VoaNHABV71I60VejIDqGrRJAm4bpnS9iCpFbfmIv/Jz1prcldy5uwur/Q7800+VUk29EFeDrIwZnE7AY5t99gYAO9R8zoNHBtTXyyeSr+hSbKb5XtTlrGM0kkA8ldW6q7Z5lF3kZJ+eepK7M+lsQauLWDKD18ai2LXANg9QuNdytavZ1wVdONDTin09rdrmMksdFv5FxmwidHjtGJpOaFG/8RF/4RaM0upZCtUGslZfPyOZiKVgNVNV/nHQYy9o9FbcoE0i8xmNjvgBpfnYB65fj0Q6B6I6hF8Vz6HpmVlLOSV+pxU9QRcOqrtCbdRZS9tWKRH/fD9b+V6cH1fKRTt0pbhj0ZTWRrrWck6HzQwhgKRuhndykRK7QGHEPxVLVz1T+vrvHMDX3rvfyKE1FEOFn4guENFxIjpCRIeK7vsEEQkiajNyDMuBVX4HhsMzWsOnelftVouMcqW4yuh6KVQb5Memj/iTCLhtVZX1aZuuq4I/GSs/W2itwurp8jvwP2/bgjt21b7pzm9ctQ5ehwVrW101R4FS+EPx9Jz+vkTaPX6ntSAX0uV3oM1jK1idWg9yHH2q8Ld7HOr/ymZC84n4gcLWzDKxuxjC77SaYaJ8crfamVKr21bXtpaLxULU8d8ihChYx05EawHcBqDyztpNRJffgdPDEU34y/XYaSRtRW0bpmIpeB2lXRUXg3IriyerWLWb//v8bGZju+6iVvQFlo+bzeohInzsDfW1CvDYLfjcu3ZrZZ21oI+a56rokexa48fDx4awsd1dcIEkIvzH3dfM2+rRIv6xwoi/zWsvSO7WXM6p24WrRb3txMA0On32gtXMCwURaY3apuJprDH4u7hYLNY3/R8B/AmA6naKXuGs8jkxPJ3AYKhwe0KjCHoK7ZSpeHrJRCvFNhQAjMdSVZVyAvp+PcmC5yk+vqvWB3Hrtg5tAZQR3Lm7C+/a113z3+nFvlrhl+s2NpSxlTZ1eKtKjM+Gw2qCiZRNywEUePzJTA4DaiK6uAnbXDjL7MJ1rD+EXWta5jXe+aCsnM6oLZmXxvei0Rgt/ALAY0R0mIjuBgAiejuAASHEUYNfe9nQ5Xcgnsri1ZEw2jzlm6s1Er0dIssdl8oJ7rVbYDObtFr+bE7g/Fi04krkYor79TzyyjDWBpwlyd01LU5883evXNANrqvFYjZpvYHcszRo07NrjR82iwk7DLqQERHcdgviqSxsuu0C5SKu18Zi8NotNZeMOor23Y0mM+gbjy2KzSNRIv50wzvWLiWMtnquE0IMElEHgMeJ6DSAPwfwprn+UL1Q3A0A69atM3aUi4ys23/pYmhBppYOqxleuwXfe+ES/v2XFzEcTuBN2zvn/sMFgIgKVt++OhxBOJHBVesr10frkZbGRDSFVwan8cL5Sfz5HVcs6LL/RuB1WBFLZUs2LqmE32XFEx+/ydC6d49daRfd7rVr76e0DfvGonW1FS72+F8ZmIYQC79wS4/HYcF4NIVEurQMeKVgaMQvhBhU/x8F8ACAmwCsB3CUiC4A6AbwEhGVLDMVQtwjhDgghDjQ3t5u5DAXnS7dhixGJ3Yl+3tbkckKXLk+gL++awc++46lUV8MyLYLSsT/wnml3cBV64NV/a3NYoLPYcFkLIl/+8UFOK1mvPvAWsPGahTS55+tJXMxawOVW0M0Apngbde1sZbCPxCaqWvP2LzVo1T1yNYTOxc54pf7Ii+FtS1GYFjET0RuACYhRET9+U0A/loI0aF7zAUAB4qTv83GKp3Yz3fVbrV86/1XLcjr1EPQbdesmhfUjS1q6Vba5rHj7GgUhy9O4Vf2d8+5icpSRFpQ1ZRzLhRS+DvKCL8QtZdyAoXJXUAR/i6/o+DistB4HBatAylbPbXTCeABdUpoAfBdIcQjBr7esqXDaweR8uWptx3zSiLoseHcaBRCCLxwfhI3bq5txhf02LSOlO+7tteAERqPXLNQbTnnQiBnHx26FtIBt9K2ISdqL+UE8h5/PJVBMpPFwb7JRbV5gMIE9VLJfTUaw84qIUQfgD1zPKbXqNdfTljNJrR77BiNJBfM6lnKtKktJfrGYxiPpqr29yWyguf1G4MN365voZB+ebVVPQuBbNsga/gBaG0bxqPJuvrJS6snkc5q+abPX7O7MQOuE/17vlKtnsUv3GYA5H1+o2v4lwNBt1Ii+NTpUQDAlTUKvyxd1Hf9XG7IdhFLSfjlWPQRP5BfCzGf5O7wdBL/58lzuHFLu7Z/8WKht9fms2vZUoaFf4mwShN+jvilcP/kxDDaPDZsaHPP8ReFXL+pDTdvbccbr1galUr1IG2TpWT1lPP4gXyyt67krir833iuD+FEGn/2lm3zHOX80V9sW5wrM+JfOmdVk7OmRanImK2FQLMga/EPX5zCHbtW1VyKeceurrraLCwlZHK31gVRRpIX/sJZqTxn60nu2tUqpHAig3cf6Na2c1xM9LOtpdBC2QiWzlnV5Nx94wa88YoObUPwZqZN157hqln6m69kpIgupYi/XHIXmF/EbzIRHFZFXD9+29Z5jrAxyPd8vq2slzJL56xqclb5HYa3algu6NszVFu/v9LoCbhhMdGSOidu2daB4bCy+5ae+Xj8AHDthiCu29S2ZI5VWj0rNbELsPAzSxBZleNzWJZtVc58uW5TEAc//YZ599hpJDtW+/HZd+wquV1aPfVU9QDAvy6xNSXeOnZeW26sTAOLWdbIlhJX9gaa1vpSWlcsHdGfjSt7A7hqfQBbatxpbKki22QslcaFRsARP7Mk+ew7d9a8gQmzOKwNuPD9D1272MNoGLKck60ehllg7tq7ZrGHwDQpHk7uMgzDNBc+hwWfvH0r3rKzpHfkioGFn2EYRgcR4fdv2bTYwzAUTu4yDMM0GSz8DMMwTQYLP8MwTJPBws8wDNNksPAzDMM0GSz8DMMwTQYLP8MwTJPBws8wDNNkkBBisccwJ0Q0BuBinX/eBmC8gcNZLjTjcTfjMQPNedzNeMxA7cfdI4Qo2ctyWQj/fCCiQ0KIA4s9joWmGY+7GY8ZaM7jbsZjBhp33Gz1MAzDNBks/AzDME1GMwj/PYs9gEWiGY+7GY8ZaM7jbsZjBhp03Cve42cYhmEKaYaIn2EYhtHBws8wDNNkrBjhJ6I3E9GrRHSOiD5V5n4ioi+p9x8jon2LMc5GUsUxv0c91mNE9Asi2rMY42w0cx237nFXElGWiH51IcdnBNUcMxHdTERHiOgVInp6ocdoBFWc434i+i8iOqoe9/sXY5yNhIi+SUSjRHSiwv3z1zIhxLL/B8AM4DUAGwDYABwFsL3oMXcA+AkAAnANgIOLPe4FOObXA2hVf37Lcj/mao9b97gnAfwYwK8u9rgX4LNuAXASwDr1947FHvcCHfenAfyd+nM7gEkAtsUe+zyP+0YA+wCcqHD/vLVspUT8VwE4J4ToE0KkAPwHgLuKHnMXgP8rFJ4H0EJEXQs90AYy5zELIX4hhJhSf30eQPcCj9EIqvmsAeBjAH4IYHQhB2cQ1RzzbwG4XwhxCQCEEM1y3AKAl4gIgAeK8GcWdpiNRQjxDJTjqMS8tWylCP8aAJd1v/ert9X6mOVErcfzQShRwnJnzuMmojUA3gngaws4LiOp5rPeAqCViH5GRIeJ6HcWbHTGUc1xfxnAFQAGARwH8IdCiNzCDG/RmLeWrZTN1qnMbcV1qtU8ZjlR9fEQ0S1QhP96Q0e0MFRz3P8E4E+FEFklEFz2VHPMFgD7AbwBgBPAL4noeSHEGaMHZyDVHPftAI4AuBXARgCPE9GzQoiwwWNbTOatZStF+PsBrNX93g0lAqj1McuJqo6HiHYDuBfAW4QQEws0NiOp5rgPAPgPVfTbANxBRBkhxI8WZISNp9rze1wIEQMQI6JnAOwBsJyFv5rjfj+AzwnF/D5HROcBbAPwwsIMcVGYt5atFKvnRQCbiWg9EdkA/AaAh4oe8xCA31Ez4tcAmBZCDC30QBvInMdMROsA3A/gt5d55KdnzuMWQqwXQvQKIXoB3Afg95ax6APVnd8PAriBiCxE5AJwNYBTCzzORlPNcV+CMssBEXUC2Aqgb0FHufDMW8tWRMQvhMgQ0UcBPAqlEuCbQohXiOjD6v1fg1LdcQeAcwDiUCKFZUuVx/y/AAQBfFWNfjNimXc0rPK4VxTVHLMQ4hQRPQLgGIAcgHuFEGXLAZcLVX7WfwPgW0R0HIoF8qdCiGXdrpmIvgfgZgBtRNQP4C8BWIHGaRm3bGAYhmkyVorVwzAMw1QJCz/DMEyTwcLPMAzTZLDwMwzDNBks/AzDMEuMuRq1lXn8u4nopNqo7rtzPZ6Fn2kKiOgX6v+9RPRbDX7uT5d7LYaZB98C8OZqHkhEmwH8GYDrhBA7APzRXH/Dws80BUKI16s/9kJpaFY1RGSe4yEFwq97LYapi3KN2ohoIxE9ovZiepaItql3/Q8AX5ENGatp0MfCzzQFRBRVf/wclBWuR4joj4nITER/T0Qvqr3NP6Q+/mYiekqdNh9Xb/uR+qV7hYjuVm/7HACn+nzf0b+WurLy74noBBEdJ6Jf1z33z4joPiI6TUTfUbtLgog+p07ZjxHRFxbyPWKWPPcA+JgQYj+ATwD4qnr7FgBbiOjnRPQ8Ec05U1gRK3cZpgY+BeATQoi3AoAq4NNCiCuJyA7g50T0mPrYqwDsFEKcV3//gBBikoicAF4koh8KIT5FRB8VQuwt81rvArAXSs+cNvVvnlHvex2AHVB6rPwcwHVEdBJKV9FtQghBRC2NPXRmuUJEHij7a/xA13jQrv5vAbAZymrfbgDPEtFOIUSo0vOx8DPNzpsA7Kb8Ll1+KF+iFIAXdKIPAH9ARO9Uf16rPm62xnfXA/ieECILYISUXbGuBBBWn7sfAIjoCBQL6nkACQD3EtHDAP57/ofHrBBMAEIVAox+AM8LIdIAzhPRq1DOzRdnezKGaWYIyvR5r/pvvRBCRvwx7UFENwN4I4BrhRB7ALwMwFHFc1ciqfs5C8AihMhAmWX8EMA7ADxSw3EwKxi1zfR5Ivo1QLMR5VaqPwJwi3p7GxTrZ9ZGdSz8TLMRAeDV/f4ogI8QkRUAiGgLEbnL/J0fwJQQIq4m1a7R3ZeWf1/EMwB+Xc0jtEPZUq9iu2B1Ou8XQvwYSmXG3uoPi1lJqI3afglgKxH1E9EHAbwHwAeJ6CiAV5DfjexRABOqVfgUgE/O1YKdrR6m2TgGIKN+eb4F4ItQbJaX1ATrGJRou5hHAHyYiI4BeBWKLSO5B8AxInpJCPEe3e0PALgWyl6xAsCfCCGGddUYxXgBPEhEDiizhT+u6wiZZY8Q4jcr3FWSuFX3Ivi4+q8quDsnwzBMk8FWD8MwTJPBws8wDNNksPAzDMM0GSz8DMMwTQYLP8MwTJPBws8wDNNksPAzDMM0Gf8//W1A2j2HuS4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "iterations = np.arange(len(cost_list)) * 10000\n",
    "plt.plot(iterations,cost_list)\n",
    "plt.xlabel(\"iterations\")\n",
    "plt.ylabel(\"avg cost\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, that was kind of cool, but I wish that I didn't need to calculate the gradients myself before I can apply gradient descent or stochastic gradient descent. I wonder if there is some tool that can do that for whichever functions that I am interested in... <b>Pytorch</b> comes to help!\n",
    "\n",
    "So let's use the same example from above and use Pytorch to help us calculate gradients automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Local minimum occurs where:\n",
      "theta_0 = 6.781114101409912\n",
      "theta_1 = 2.968665599822998\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "torch.manual_seed(42)\n",
    "\n",
    "lr = 0.00005 # learning rate or step size\n",
    "\n",
    "# let's get our data points x and y into the expected format\n",
    "x_tensor = torch.from_numpy(x).float()\n",
    "y_tensor = torch.from_numpy(y).float()\n",
    "\n",
    "# theta_0, theta_1\n",
    "theta = torch.tensor([1., 1.], requires_grad=True, dtype=torch.float)\n",
    "\n",
    "iter_num = 0\n",
    "cost_list = []\n",
    "\n",
    "# stopping condition based on iteration count\n",
    "for i in range(m * 5):\n",
    "    iter_num += 1\n",
    "    # random choice of matching x and y\n",
    "    idx = np.random.randint(0, m, size=batch)\n",
    "    x_i = x_tensor[idx]\n",
    "    y_i = y_tensor[idx]\n",
    "    \n",
    "    yhat = theta[0] + theta[1] * x_i\n",
    "    error = y_i - yhat\n",
    "    loss = torch.mul(0.5, (error ** 2)).mean()\n",
    "    sum_cost += float(loss.data)\n",
    "\n",
    "    loss.backward() # this is the magic\n",
    "\n",
    "    theta.data -= lr * theta.grad\n",
    "    theta.grad.detach()\n",
    "    theta.grad.zero_()\n",
    "    \n",
    "    # saved for plot\n",
    "    if (i + 1) % 10000 == 0:\n",
    "        cost_list.append(sum_cost / 10000.0)\n",
    "        sum_cost = 0\n",
    "            \n",
    "print(\"Local minimum occurs where:\")\n",
    "print(\"theta_0 =\", float(theta[0].data))\n",
    "print(\"theta_1 =\", float(theta[1].data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEGCAYAAACdJRn3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAwoUlEQVR4nO3deXzU1bnH8c+TjZCNEJJASIAQ1iCyhEVRQRGtlaqItLXV1qW2Lq3V2sVra3vbahdte9taa7ku1WvrvoDWDUVcwAWBsK8CgUAgkAQCCYGQ7dw/ZgIRIcwkM5lJ8n2/Xrwy+c3M7/ccJ+bJ+Z1znmPOOURERPwREeoARESk/VHyEBERvyl5iIiI35Q8RETEb0oeIiLit6hQB+CL1NRUl52dHeowRETalfz8/DLnXFowzt0ukkd2djZLliwJdRgiIu2KmRUG69y6bSUiIn5T8hAREb8peYiIiN+UPERExG9KHiIi4jclDxER8ZuSh4iI+K1DJ49315cw873NoQ5DRKTD6dDJ44NNZdw371MaGrRniYhIIHXo5JGTFk91bQM79x8KdSgiIh1Kx04eqQkAFJRWhTgSEZGOpUMnjwFp8QAUlB4IcSQiIh1Lh04eaYldSOgSRUGZeh4iIoHUoZOHmTEgLV63rUREAiyoycPMks3sBTNbb2brzGyCmf3R+/1KM5ttZsnBjCEnLYHNum0lIhJQwe553AfMcc4NBUYC64C5wHDn3AjgU+CnwQwgJzWe4v3VHKypC+ZlREQ6laAlDzNLAiYB/wRwztU45/Y5595yzjX+Jl8IZAUrBvD0PEAzrkREAimYPY8coBR4zMyWmdkjZhZ/zGu+BbxxvDeb2fVmtsTMlpSWlrY8iMYZVxo0FxEJmGAmjyggD5jpnBsNVAF3ND5pZncCdcCTx3uzc+4h59xY59zYtLSWb8HbPzUeM03XFREJpGAmjyKgyDn3iff7F/AkE8zsauAi4ErnXFBrh8RGR5KZ3FW3rUREAihoycM5twvYbmZDvIemAGvN7IvAfwGXOOcOBuv6TWnGlYhIYEUF+fzfB540sxigALgWWAx0AeaaGcBC59yNwQwiJzWeJVv34pzDe00REWmFoCYP59xyYOwxhwcG85rHMyAtnoM19eyqqCajW9e2vryISIfToVeYNxqg6boiIgHVKZLH0bUeGvcQEQmETpE8eiZ1IT4mks3qeYiIBESnSB5mRv+0eM24EhEJkE6RPMCzMZTGPEREAqPTJI8BaQns3H+I6tr6UIciItLudZrkkZMWj3OwRTWuRERarVMlD0DjHiIiAdBpkkf/1Mb9zNXzEBFprU6TPOJioujdLVZrPUREAqDTJA+AAekJ2tdDRCQAOlXyyEmNp6C0iiBXgRcR6fA6V/JIS+DA4TpKKg+HOhQRkXatkyUPzbgSEQmETpY8VF1XRCQQOlXyyEiKJTY6QslDRKSVOlXyiIgwT42rMt22EhFpjU6VPMAz7qExDxGR1umEySOBonIVSBQRaY1OlzwGeAskFu45GOpQRETarU6XPHJStSWtiEhrdb7k4V3roTIlIiIt1+mSR3yXKHolxWrQXESkFTpd8oDGGVfqeYiItFSnTR4FpQdUIFFEpIU6Z/JITaCyuo6yAzWhDkVEpF3qlMljQLpmXImItEanTB45qZpxJSLSGp0yeWQmd6VLVASbS9TzEBFpiU6ZPCIijP6p8ep5iIi0UKdMHnB0xpWIiPiv0yaPAWkJbC8/RE1dQ6hDERFpdzpt8shJi6e+wbFtr25diYj4K6jJw8ySzewFM1tvZuvMbIKZpZjZXDPb6P3aPZgxnEhjgcRNJUoeIiL+CnbP4z5gjnNuKDASWAfcAcxzzg0C5nm/b3NHCyRq3ENExF9BSx5mlgRMAv4J4Jyrcc7tA6YBj3tf9jhwabBiaE5ibDRpiV20n7mISAsEs+eRA5QCj5nZMjN7xMzigZ7OuWIA79f0473ZzK43syVmtqS0tDQoAQ7QjCsRkRYJZvKIAvKAmc650UAVftyics495Jwb65wbm5aWFpQAc9IS2FxapQKJIiJ+CmbyKAKKnHOfeL9/AU8y2W1mGQDeryVBjKFZOanx7D9Uy94qFUgUEfFH0JKHc24XsN3MhngPTQHWAv8BrvYeuxp4OVgxnMyANG+BRK00FxHxS1SQz/994EkziwEKgGvxJKznzOw6YBvwlSDHcEJHZlyVHmBcdkqowhARaXeCmjycc8uBscd5akowr+urrO5xxERGaMaViIifOu0Kc4DICCM7NU5b0oqI+KlTJw/wrDTXdF0REf8oeaTFs23vQWrrVSBRRMRXSh5pCdQ1OLbtPRjqUERE2g0ljyMzrjTuISLiq06fPAZ4q+tq3ENExHedPnl0i4smNSFGPQ8RET90+uQBnhlXm9XzEBHxmZIH3v3MVaJERMRnSh54ksfeqhr2HVSBRBERXyh5cLRAolaai4j4RskDz1oP0IwrERFfKXkAfbp3JTrS1PMQEfGRkgcQFRlB35Q49TxERHyk5OGVk5agGVciIj5S8vAakJZA4Z4q6lQgUUTkpJQ8vHLS4qmtdxSVHwp1KCIiYU/Jw2uAt0CiVpqLiJyckodXzpECiRr3EBE5GSUPr+7xMXSPi6agTD0PEZGTUfJoYkBagtZ6iIj4QMmjiZy0eN22EhHxgZJHEzlpCZQdOMz+Q7WhDkVEJKydNHmY2TxfjnUEOamNW9Jq3ENEpDknTB5mFmtmKUCqmXU3sxTvv2ygd5tF2IaOFkjUrSsRkeZENfPcDcAP8CSKfMC8xyuAB4IbVmj06xFHVIRpxpWIyEmcMHk45+4D7jOz7zvn7m/DmEImOjKCfj3i2LBLyUNEpDm+DJjvMrNEADP7uZnNMrO8IMcVMrkZSazfVRHqMEREwpovyeMXzrlKMzsLuAB4HJgZ3LBCJzcjiaLyQ1RUa8aViMiJ+JI86r1fvwTMdM69DMQEL6TQGpaRBMD64soQRyIiEr58SR47zOxB4KvA62bWxcf3tUtDMxIBWFesW1ciIifiSxL4KvAm8EXn3D4gBfhJMIMKpV5JsSTHRSt5iIg046TJwzl3ENgMXGBmNwPpzrm3gh5ZiJgZub2SlDxERJrhywrzW4EngXTvvyfM7Pu+nNzMtprZKjNbbmZLvMdGmdnCxmNmNr41DQiG3IwkNuyupL7BhToUEZGw1NwiwUbXAac556oAzOxe4GPA17Ufk51zZU2+/wPwa+fcG2Y21fv9Ob6HHHy5GYlU1zawpayKgekJoQ5HRCTs+DLmYRydcYX3sZ3gtb5wQJL3cTdgZyvOFRS5jTOutN5DROS4fOl5PAZ8Ymazvd9fCvzTx/M74C0zc8CDzrmH8JQ8edPM/oQneZ1xvDea2fXA9QB9+/b18XKBMahnAlERxrriCi4a0SHLeImItMpJk4dz7s9m9h5wFp4ex7XOuWU+nv9M59xOM0sH5prZeuDLwG3OuRfN7Kt4EtF5x7nuQ8BDAGPHjm3TwYcuUZEMSEtgndZ6iIgc10mTh5mdDqxxzi31fp9oZqc55z452Xudczu9X0u8PZfxwNXArd6XPA880tLgg2loRiKLtuwNdRgiImHJlzGPmUDTSoFV+FCexMzim9TEige+AKzGM8Zxtvdl5wIb/Qm4reRmJFG8v5p9B2tCHYqISNjxZczDnHNHbhs55xrMzJf39QRmm1njdZ5yzs0xswN4qvVGAdV4xzXCTeOg+driCs4YkBriaEREwosvSaDAzG7haG/ju0DByd7knCsARh7n+AfAGH+CDIXcI2VKKpU8RESO4cttqxvxzIjaARQBpxGmvYVASk+MJTUhRivNRUSOw5fZViXA19oglrCjvT1ERI6vw1bHDYTcjCQ+3X2AuvqGUIciIhJWlDyakZuRSE1dAwVlVaEORUQkrCh5NGNoL8+MK417iIh8li+LBH94nMP7gXzn3PKARxRGBqQlEB1prC2uYNqozFCHIyISNnzpeYzFM+Mq0/vvejxVcB82s9uDF1roxURFMDA9UWVKRESO4Uvy6AHkOed+5Jz7EZ5kkgZMAq4JYmxhITcjUbetRESO4Uvy6As0rdFRC/Rzzh0CDgclqjAyLCOJ0srDlB3o8E0VEfGZLyvMnwIWmtnL3u8vBp721qtaG7TIwsSRvT2KKzlrUJcQRyMiEh582cP8buA7wD48A+U3Oufucs5VOeeuDHJ8IdeYPHTrSkTkKF9mW90HPOucu68N4gk7KfEx9EzqouQhItKEL2MeS4Gfm9kmM/ujmY0NdlDhZmivJNYqeYiIHOHLbavHnXNT8Wzk9Clwr5mF5R4cwZKbkcTm0gPU1KlMiYgI+LfCfCAwFMgG1gclmjCVm5FIbb1jU8mBk79YRKQTOGnyMLPGnsZdwBpgjHPu4qBHFkaGadBcROQzfJmquwWY4JwrC3Yw4ap/ajwxURFKHiIiXr7s5/G/ZtbdzMYDsU2Ozw9qZGEkKjKCIT0TWb9LZUpERMC3qbrfBm4FsoDlwOnAx8C5QY0szORmJDJvXQnOObz7souIdFq+DJjfCowDCp1zk4HRQGlQowpDuRlJ7KmqobRSZUpERHxJHtXOuWoAM+vinFsPDAluWOGncW8PrfcQEfEteRSZWTLwEjDXW+NqZzCDCkdHZ1xp3ENExJcB8+neh78ys3eBbsCcoEYVhrrFRdO7W6xmXImI4NtU3SOcc+8HK5D2IDcjSclDRATtYe6X3IwkCsqqqK6tD3UoIiIhpeThh9yMJOobVKZERETJww+5GYmAZlyJiCh5+KFfj3hio1WmREREycMPkRHGkF4aNBcR8Wu2lcCwjEReX7VLZUpaaOe+Qzz6wRZSEmIY1SeZEVnJJHTRj6FIe6P/a/2Um5HE04u2U7y/mt7JXUMdTrvR0OB44pNC7n1jPYfrGqhrcACYweD0REb1SWZU32RG9UlmcM9EIiOUmEXCmZKHn3Kb7O2h5OGbjbsruWPWKvILy5k4KJXfTT+VhC5RLC/ax/Jt+1i+fR9z1uzi2SXbAYiLieTUzG6M6pvM6D7JjOrTnV7dYk9yFRFpS0oefhrayzPjal1xBVNyewb9etW19Xy8eQ/vrC9hwoAeTD01I+jXDJSaugZmvreZB97dRFyXSP781ZFMH5155Hbf5CHpTB6SDoBzjq17DrJ8e/mRhPLoB1uorff0UEZkdePZ6yfQNSYyZO0RkaOCmjzMbCtQCdQDdc65sd7j3wduBuqA15xztwczjkBKjI2mT0pX1gVxb4/9B2t5Z8Nu5q7dzfsbSqmqqccMnl28nT7d4zg1q1vQrh0o+YXl/HTWSj7dfYBpo3rzi4uGkZrQ5YSvNzP6p8bTPzWe6aOzAE/iXFtcwbvrS7j/nU28tXYX00ZltlUTRKQZbdHzmNx0F0IzmwxMA0Y45w6bWXobxBBQuUGYcbVj3yHmrtnFW2t388mWvdQ3ONITuzBtdCZfGNaT3Iwkpj/wId99Kp9Xb55It7jogF4/UA4cruNPb27g8Y+3kpEUy2PXjGPy0JZ9xLHRkeT17c6orGRezC9i1tIdSh4iYSIUt61uAu5xzh0GcM6VhCCGVhmakcTb63ZzqKa+xbdRnHOsLa5g7trdvLVm95GFhwPTE7hhUg7nD+vJyKxkIpoMHP/9yjwuf/BjfvT8Ch6+akzYzfZ6d30Jd85eRXFFNVdPyObHFwwJyEyqiAhjel4mM9/bTElFNelJGv8QCbVgJw8HvGVmDnjQOfcQMBiYaGa/BaqBHzvnFh/7RjO7HrgeoG/fvkEO0z/DMhJpcLBhdyWj+iT7/f65a3fzq/+sYce+Q5jBmL7d+dnUoZw/rBf9U+NP+L68vt352dRcfv3KWh6aX8ANZw9oRSsCp+zAYe56ZS3/WbGTQekJvHDjGYzp1z2g17gsL4sH3t3MS8t3cP2k8Gi3SGcW7ORxpnNup/fW1FwzW++9Znc829mOA54zsxznnGv6Rm+ieQhg7NixjjDSdMaVv8ljw65Kbnl6Gf16xHHvjFM5d2hP0hJPPBZwrGvOyGbJ1nL+8OYGRvftzvj+KX5dP5Ccc8xauoO7X1tL1eE6bjtvMDeek0OXqMAPag9IS2BUn2RezN/BdybmhF2vK5wcqqlnx76DDExPDHUo0oEFdYW5c26n92sJMBsYDxQBs5zHIqABSA1mHIHWp3sc8TGRfo97VFbXctMT+STERvGvb43n8nF9/Uoc4BlYvmfGqfRNiePmp5aGbFvcLWVVXPnIJ/zo+RXkpMbz+i0TufW8QUFJHI1m5GWyYXdlUGqL1dY38N8vr+bT3e1zs6/q2nrmrN7FzU8tJe/uuZz35/m8t6Hd3RGWdiRoycPM4s0ssfEx8AVgNZ4dCc/1Hh8MxABlJzhNWIqIMIb6ubeHc46fPL+Swr0HeeCKvFbdt0+MjeYfV+ax/1Attz6zjPqGtuuY1dQ1cP+8jVzw1/ms2rGf31w6nBduPINBPYP/V+7FI3sTHWnMWroj4Od+deVO/vVxIXe/ujbg5w6Ww3X1vL12Nz94Zhljf/M2Nz6Rz0eb93BZXiY5qfH84uXVHKrpfNsHvL6qmG17DoY6jA4vmLetegKzvbcXooCnnHNzzCwGeNTMVgM1wNXH3rJqD3IzEnl52U6fy5Q8vKCAOWt28fMv5QbkVlNuRhJ3Xzqc219YyX1vf8oPvxD8beUXb93LT2etYlPJAb40IoNfXjSsTQevk+NimDK0Jy8v38EdFw4lOjIwf/s453h4/hYiI4wFG8tYuq2cvL6BHbMJlJq6Bj7cXMarK4p5a+0uKqvrSI6L5qIRGVw0ojen56QQFRnBx5v38PWHF3L/Oxu5/YtDQx12m/loUxnffXIpY/t154Wbzgh1OK1SV9/Awwu2cO2Z2cRGh9/6pqAlD+dcATDyOMdrgG8E67ptJTcjiScWbqOo/BB9UuKafe3Cgj3cO2cDU0/txXVn9Q9YDF8d24fFW/byt3c2kdevO+cMCc6s530Ha7jnjfU8s3g7mcldWzX9trUuy8tkzppdLNhYyrlDA7NI8+PNe1hbXMEvLx7G/e9s4v55G3ns2vEBOXcg1NU38NHmPby2spg5a3ax/1AtibFRXHBKLy4akcGZA1M/l0gnDOjBl8dk8dD8Ai4dncngNugZhtrhunp+/tJqoiONJYXlLN66l3HZoRsTbI2GBsd/vbiKF5cW0T81ji8OD7/Fwaqq20JNB82bs7uimpuf8gyQ/+HLIwM+0HvXtOEM7ZXIbc8uZ+e+QwE9t3OOl5fv4Lw/v8/z+UXcMCmHuT+cFLLEAXDOkHRS4mN4MYC3rh5eUEBqQgxfH9+Xb0/sz7sbSllZtC9g52+Nw3X1XHjfAq56dBGvrSpmytB0/nn1WJb8/Dz+9JWRnDMk/YQ9sJ9NzSUxNoo7Z6+ioQ1vbYbKg+8XUFBWxd+vyCMlPoZ/vLsp1CG1iHOO37+xjheXFnHbeYPDMnGAkkeLDemZiBmsKz7xAGttfQPfe3IpB2vqePAbY4JSPbZrTCT/uDKP2nrH955aSk1dQ0DOW7iniqseXcStzywns3scr9x8Fj+dmktcTGgr2sRERXDJyN7MXbub/QdrW32+jbsreXdDKVdN8NwauGpCNslx0fxt3sYARNt6a3ZWsLHkAD+5YAhLfn4ef758FFNye/o0MSElPoafTs1l8dZynvPWDeuotpZV8fd3N3HRiAwuOKUX156RzbsbStvl9gkPzi/g4QVbuHpCP26ZMjDU4ZyQkkcLxXeJol9KXLM/nL9/fT1LCsu5Z8aIoA4o56Ql8Icvj2DZtn38/o11rTpXbX0D/3hvE1/4y3yWbdvHry85hVk3ncGw3kkBirb1LsvLpKaugddWFbf6XI8s2EJsdATfOL0fAAldorjuzP68va6E1Tv2t/r8rbW0sByAr4zJatF976+MyWJ8/xR+/8Z6yg6EZmZesDnn+MXLq+kSGcEvLhoGwFUTsomPieR/398c4uj889zi7dzzxnouHtmbX158SlhPSVfyaIXcjCTW7Tp+8nh15U4e/XAL15yRzSUjewc9lqmnZnDNGdk89uFWXm/BL9XdFdU8v2Q7F/3tA/4wZwPnDk3n7R+ezdVnZIddefRTM7sxKD2BWUuLWnWe0srDzF62gxl5WaTExxw5fvWZ2STGRnH/O6HvfeQXltMnpWuLJyaYGb+bPpyDNXX87rXW/WERrl5dWcyCjWX86AuD6en979QtLporT+/HKyt2tpuZV2+u2cUds1YyaXAa//OVkZ+pLhGOlDxaITcjicI9BzlwuO4zxzeVVHL7CysZ08+zIryt/GxqLqP6JHP7CyvZUlbV7Gura+uZ/2kpv31tLRf8ZT6n/W4eP3lhJVU1dTxy1VhmfmNM2JZBNzMuy8tiSWE5W0/Szub8++Ot1DY0fG4SQ1JsNN86sz9vrtkd0tsezjmWFJYzppUzvwamJ3Lj2QOYtWwHH21qV7PiT6qiupa7Xl3LqZnd+OaE7M88d91Z/YmKiOChBeHf+1hYsIfvP72MEVnJ/O838oiJCv9fzeEfYRhrHDTf0KT3ceBwHTf8O5+4mEgeuKJtfwhioiJ44Mo8oiKNm57Ip7r26Bx/5xwbdlXyyIICvvnPTxj567e46tFFPP5RIamJMdxx4VBev2Ui838ymfOGBb/UfGtdOro3ZjBrWcsGzg/V1PPvhYVMGdqTnLSEzz3/rTP7k9Alir+/E7pB16LyQ5RWHg5IqZfvTR5Ivx5x3PnS6s/8XLR3//PmBsoOHOa304d/rofcMymWGWMyeW5JUcgW0/pi9Y79fOfxJfRNieOxa8aFfFzRV+0jyjCVm+EZx1hbXMmYfik45/ivFz1/9T/x7dNC8pd7ZnJX/nL5KK59bDE/m72Kc4akM//TUhZsLGV3hed/oIHpCVxxWl8mDU7jtP4p7eaHtamMbl05a2Aqs5YW8YMpg/zu4r+4tIjyg7V8Z+Lxp053i4vmmjOyeeC9TXy6uzIkU12XbvOMd+QFIHnERkfym0uH881/LmLme5u57fzBrT5nqK0s2se/FxZy1en9GJGVfNzXXD9pAM8u3s6jH27hv8JwvcvWsiqueWwRid6qE92b3D4Nd+p5tEJmcleSYqNY77218eiHW3ltZTG3f3EoZwwIXcWVyUPSuXnyQGYt3cEtTy9j7trdjO2Xwr0zTuXDO87l7R+ezS8vPoXJQ9LbZeJodFleJkXlh1i8da9f72tocDz6wRZGZHVrdsHmdWf1p2t0ZMh6H0sLy4mLiWRIgBLXxEFpXDKyNzPf28zm0gMBOWeo1Dc47py9mh4JXfjRBSdeINs/NZ4LT83giY8Lqahu/ey8QCqpqOabj35CfYPjX9ed1u52JlXyaAWzo2VKFm/dy+9fX8cXhvXkhkk5oQ6N284fzF8uH8ns757B0l+czwNX5nH5uL5ktrMf0OZccEov4mIi/S5XMm99CQVlVXz7JAUWu8fHcNWEbF5ZuZNNJW3/yzZ/Wzmj+iQTFaCV9AA/vyiXLtER3Dl7Fe2wsMMRTywsZNWO/fz3RcNIim1+b5ubzh5A5eE6nly4LWDX/+XLq8m7ey4/m72KhQV7/F5Hs/9QLVc9uog9B2r4v2vHMzD987dOw52SRyvl9kpkXXEl33tyKX1S4vjTVwO/ELAlIiOM6aOzGN23e9jNlgqUuJgoLhyewWuriv26j//wggIyk7sydXivk772OxP7ExsV2eYLzqoO17GuuDLgpe3TE2O548KhLCzYG5QaYW1hd0U1f3xzAxMHpXLRiJMvoBue2Y2Jg1L55wdbAjLe80J+EY9/XEif7l2ZvXQHX3toIWfe+w6/e30dq3fsP2lSPlRTz7cfX8zm0gM89M2xjGzBtg7hQMmjlXIzkjhUW09FdS0zv5F30r+CJLBmjMnkwOE63lyzy6fXryzax6Ite7n2zGyf/qLvkdCFb07ox0vLd5x0BlsgrSjaR32DC8h4x7G+Pq4veX2T+e3r6yivqgn4+YPt7lfXUlPfwN3Thvv8h9p3zxlI2YHDvJDfuunda3dWcOfsVZwxoAcv3nQG+b84j/u+NopTeifx6AdbuOj+D5jy5/e57+2Nx/15qa1v4OanlrKksJy/Xj6aswa1q4Lin6Hk0Upjs1OIiYzgnstGMLRX+Cyk6yxO79+D3t1iff4r+uEFW0jsEsXl4/r4fI3vTMwhOjKCB9qw99G4ODCvT+CTR0SE8dvpp7L/UC33vLE+4OcPpvc/LeXVlcXcPHkg2c1snHas03NSGNUnmQfnb6auvmVVGCqqa/nuk/meCgRfH01UZARxMVFMG5XJI1ePY/Gd5/G76aeSntiFv877lMl/eo9L/v4BjywoYHdFNQ0NjjteXMW89SXcNW04X/Kh1xTOlDxaaWB6Aqt/fQGXjtbe2qHQuEXtgo2llFRUN/vaovKDvL6qmK+N70OiHz3EtMQuXHlaP2Yv29FmC87yC8sZlJ4QtL3qczOS+PZZ/Xl2yXYWbfFvwkGoVNfW898vryYnNZ4bzvZvXNHMuOmcAWzfe6hFlQmcc/z4uRUUlR/igSvySE34/D483eNjuOK0vjxz/QQ+uuNc7pyaS4Nz/Oa1dZz++3lc8Nf5R+pVfdNb0aA9U/IIgPawoKcjuywviwYHLy1vvvfx2IdbMeDaM/2vbHzD2TlERhj/eC/4vY+GBsey7fuCXhb+1vMGkZnclZ/NXhWwmmjB9I93N1G45yC/uXR4izYdOz+3JwPTE5j53ma/Jws8vKCAt9bu5o4LhzLWh0q9Gd268p1JObz6/YnM+9HZ3HLuIMzghkk5YV2vyh/6rSftXtMtak/0S6GiupZnF2/nSyMyWjQlsmdSLF8f14cX8osoKg9u76OgrIp9B2sDPlh+rLiYKO6adgqbSg7w8IKCk75+38Ealm/fx8vLd/DXtz/lJ8+v4D8rdrbJrK1NJQeY+f5mLh3VmzMGtmycICLCuPHsAazfVcl7G0p9ft8nrdxSYUBaAredP5i3bjubn07NDYsJNYHQfif5izQxIy+TX7y8hrXFFZzSu9vnnn9m0TYOHK7jOxNbPo36xnMG8PSi7cx8bzO/nX5qa8Jt1pHxjiAnD4ApuT25cHgv/jZvIxeNyCAxNpqte6oo3FPFlrKDFO6pYuueg2wtq2L/oaPrJMw8ZVyezy/iiYWF/OriU4JWPNM5xy9eWk1sdCR3fmlYq841bVRv/vzWBma+t9mnrQVKKqq5+ell9EuJ494ZIzrML/5AUPKQDuHikb2569W1zFq643PJo7a+gcc+3MrpOSkMz/x8YvFVRreufGVsFs8t2c73Jg8M2qKu/MJykuOiyfFjQLg1fnnxKSzYWMaU/3mfuibrFcygd7eu9E+N56IRGWT3iCc7NZ7sHnH0SYkjOjKC55Zs5w9z1nPR/Qv4xun9+OH5g0mOC+wq6ZeW7+Djgj385tLhpCV+fqzBH9GREXxnUg6/fmUtS7bubfYWVF19Azc/vYwD1XU8cd1pfo2TdQZKHtIhNLdF7euriineX81vLh3e6uvcdI6n3MWD72/m19Naf77jyfdug9tWVVV7dYvlL5ePYv6npfTrEXckSfRJ6XrSsYWvj+/L1OEZ/HnuBv69sJBXVuzkJxcM5fJxfQKyvmj/wVp+8+o6RvVJ5orxfVt9PoDLx/Xhb/M2MvO9zfzzmhMnjz++uYFFW/byl8tHMqRXx9+J0V8a85AO47K8TMoO1LBg49H72c45Hl5QQE5aPJMDsE1vVvc4vjwmi6cXb2f3SWZ3tcS+gzVsKjkQ9PGOY50/rCd3Xzqcb0/M4bxhnoFlXwelu8VF8+tpw3ntlokM6pnIz2avYtoDH5Dvvf3mr/oGR37hXu6ds55L//Eh5Qdr+O304QFLpnExUVx7Zn/mrS9h/Qm2VJizehcPzi/gG6f3ZfrorIBct6NR8pAO43hb1C4s2MvqHRV8+6ycgP3y+e45A6lvcEHZaGjZ9n0AQZ9pFQy5GUk8e/3p/O3roymrrGHGzI/44XPLKak8eZKtrK7ltZXF/PC55Yz77dvMmPkxD88voFdSLPd/Pe+441itcdWEfsTFRPLg+5+fKLClrIqfPL+CkVndjmwuJZ+n21bSYTRuUfvUom3sP1hLt7hoHllQQI/4GC7LC9w6nL494pg+OpOnPtnGTecMID0xcNWTlxaWExlhjOwT2F+WbcXMuGRkb6YMTeeBdzfxyIItvLVmN7dOGcTVZ2R/Zlp74Z4q5q0rYd763SzaspfaekdyXDSTh6Rz7tB0Jg1Oo1vX4IwzJMfFcMX4vjz20VZ+eP5g+qTEAZ7SITc9kU9kpPHAlXktmhLcWSh5SIdyWV4m//fRVl5bVcz4/inMW1/CrVMGtWgL1+Z4qhYX8fD8glbPAGoqv7Cc3IzEdl3tGDzbNN/+xaF8dWwf7np1Lb99fR3PLN7Gd88ZyKcllcxbV3Kk2OTA9AS+dVZ/zsvtyegAF4Jszrcn5vD4x1t5eEEBd00bjnOOn7+0mg27K3nsmnFkdY9rkzjaq/b9EypyjKZb1K7asZ+YqAi+OSHwq3mzU+O5dFQm/15YyPcmDwzIDKO6+gaWb9/HV8Z0nHvs2anxPHrNON5Zv5u7XlnLj55fQVSEcVpOCleM78uU3HT69WibWWXH6tUtlstGZ/Hs4u3cMmUQc9fu5sWlRdwyZRDnBGB8rKNT8pAOpXGL2nvnrGdl0X5mjMk8bimJQPjWWf2ZtWwHr6zY+bktUFti/a5KDtbUt8n6jrZ27tCenDkwleXb9pHbOylsCojecHYOz+Vv5+ezV/POhhImDkrl1imDQh1Wu6ABc+lwGreoralv4Lqzgre3yim9kxjaK5EXAlTavHHnwLaeadVWukRFclpOj7BJHAA5aQlcOLwXc9bsokd8DPd9bXSH3cIg0JQ8pMPJ6NaVS0b2ZvrozKBusmNmzMjLYsX2fQHZLCq/sJyeSV061IZd7cGtUwYzPDOJf1yZR0o72gY21JQ8pEO672uj+cvlo4J+nWmjexMZYby4tHX7RICn55HXt7tKYLSxIb0SefX7ExndDqdHh5KSh0grpCfGMmlQKrOX7qDez61ImyqpqGb73kMd9paVdDxKHiKtNGNMFrsqqvloc1mLz9E43tERB8ulY1LyEGml83J7khQb1aotTvMLy4mJiuCUIFWmFQk0JQ+RVoqNjuTikb15c80uKqtrT/6G48gvLGdEZjetaJZ2Q8lDJABmjMmiuraB11uwxenhunpW76jQeIe0K0oeIgEwuk8yOanxvJjv/5qP1TsqqKlv0GwfaVeCmjzMbKuZrTKz5Wa25JjnfmxmzsxatqekSBgxM2aMyWLR1r1s2+PfNrVHdw5MDkJkIsHRFj2Pyc65Uc65sY0HzKwPcD6wrQ2uL9Impo/OxAy/13zkF5bTNyUuoNV5RYItVLet/gLcDrR8YrxImOmd3JUzB6Qya1kRDT6u+XDOkb+tXOMd0u4EO3k44C0zyzez6wHM7BJgh3NuRXNvNLPrzWyJmS0pLS1t7qUiYWPGmEy27z3Eoq17fXp9UfkhSisPa32HtDvBTh5nOufygAuB75nZJOBO4L9P9kbn3EPOubHOubFpaWlBDlMkMC44pRfxMZG86OOaj8atWsdosFzamaAmD+fcTu/XEmA2cDbQH1hhZluBLGCpmfUKZhwibSUuJoqpp2bw+qpiDtbUnfT1S7eVEx8TyZBeiW0QnUjgBC15mFm8mSU2Pga+ACx2zqU757Kdc9lAEZDnnNsVrDhE2tqMMVlU1dTz5pqT/1jnF5Yzqm+yyoBLuxPMnkdP4AMzWwEsAl5zzs0J4vVEwsL47BSyunc96ZqPqsN1rCuu0C0raZeCtpOgc64AGHmS12QH6/oioRIR4dnN8P53NrJz3yF6n2B/jhXb99HgVAxR2ietMBcJghl5mTgHs5eduPfROFiuleXSHil5iARBvx7xjM9O4cWlRTh3/DUf+dvKGdwzgW5dw2dbVhFfKXmIBMmMMZkUlFaxbPu+zz3X0OBYtm2fFgdKu6XkIRIkU0/NIDY64rhrPgrKDrD/UK1uWUm7peQhEiSJsdFccEovXlmxk+ra+s88d2RxoHoe0k4peYgE0Yy8LCqq65i3ruQzx/MLy0mOiyYnNT5EkYm0jpKHSBCdOTCVXkmxn6u0m19Yzpi+3THT4kBpn5Q8RIIoMsKYnpfJ+5+WUlp5GIB9B2vYXFql9R3Sril5iATZjLws6hscLy/3rPlYtm0foPEOad+UPESCbGB6AiP7JPNCvmfNR35hOZERxsis5FCHJtJiSh4ibeDLeZms31XJmp0V5BeWMywjia4xkaEOS6TFlDxE2sDFI3sTExnBc0u2s3y7FgdK+xe0wogiclRyXAxTctN5etE2auudBsul3VPPQ6SNfHlMFrX1njpX6nlIe6eeh0gbmTQ4jdSEGKIiIujdLTbU4Yi0ipKHSBuJjozg7mnDOVzXoMWB0u4peYi0oQtPzQh1CCIBoTEPERHxm5KHiIj4TclDRET8puQhIiJ+U/IQERG/KXmIiIjflDxERMRvSh4iIuI3c86FOoaTMrNSoLCFb08FygIYTnvTmduvtndenbn9TdvezzmXFoyLtIvk0RpmtsQ5NzbUcYRKZ26/2t452w6du/1t1XbdthIREb8peYiIiN86Q/J4KNQBhFhnbr/a3nl15va3Sds7/JiHiIgEXmfoeYiISIApeYiIiN86dPIwsy+a2QYz22Rmd4Q6npYys61mtsrMlpvZEu+xFDOba2YbvV+7N3n9T71t3mBmFzQ5PsZ7nk1m9jfzbmdnZl3M7Fnv8U/MLLvNG9mEmT1qZiVmtrrJsTZpr5ld7b3GRjO7uo2afMQJ2v4rM9vh/fyXm9nUJs91pLb3MbN3zWydma0xs1u9xzvLZ3+i9ofn5++c65D/gEhgM5ADxAArgGGhjquFbdkKpB5z7A/AHd7HdwD3eh8P87a1C9Df+98g0vvcImACYMAbwIXe498F/tf7+GvAsyFu7yQgD1jdlu0FUoAC79fu3sfdw6DtvwJ+fJzXdrS2ZwB53seJwKfeNnaWz/5E7Q/Lz78j9zzGA5uccwXOuRrgGWBaiGMKpGnA497HjwOXNjn+jHPusHNuC7AJGG9mGUCSc+5j5/lp+dcx72k81wvAlMa/VELBOTcf2HvM4bZo7wXAXOfcXudcOTAX+GKg29ecE7T9RDpa24udc0u9jyuBdUAmneezP1H7TySk7e/IySMT2N7k+yKa/yDCmQPeMrN8M7vee6ync64YPD90QLr3+Inanel9fOzxz7zHOVcH7Ad6BKEdrdEW7Q3nn5mbzWyl97ZW422bDtt27+2U0cAndMLP/pj2Qxh+/h05eRzvL+f2Oi/5TOdcHnAh8D0zm9TMa0/U7ub+e7Tn/1aBbG+4/neYCQwARgHFwP94j3fItptZAvAi8APnXEVzLz3OsY7Y/rD8/Dty8igC+jT5PgvYGaJYWsU5t9P7tQSYjeeW3G5v9xTv1xLvy0/U7iLv42OPf+Y9ZhYFdMP3WydtpS3aG5Y/M8653c65eudcA/Awns8fOmDbzSwazy/OJ51zs7yHO81nf7z2h+3n35YDQm35D4jCM+jTn6MD5qeEOq4WtCMeSGzy+CM89yL/yGcHEf/gfXwKnx1EK+DoINpi4HSODqJN9R7/Hp8dRHsuDNqdzWcHjYPeXjyDhVvwDBh29z5OCYO2ZzR5fBue+9wdru3eWP8F/PWY453is2+m/WH5+Yf0F0QbfBhT8cxY2AzcGep4WtiGHO8PyApgTWM78NynnAds9H5NafKeO71t3oB3loX3+Fhgtfe5v3O0wkAs8DyeAbdFQE6I2/w0nu55LZ6/iK5rq/YC3/Ie3wRcGyZt/zewClgJ/OeYXyYdqe1n4blVshJY7v03tRN99idqf1h+/ipPIiIifuvIYx4iIhIkSh4iIuI3JQ8REfGbkoeIiPhNyUNERPym5CGdgpl95P2abWZXBPjcPzvetUQ6Mk3VlU7FzM7BU6H0Ij/eE+mcq2/m+QPOuYQAhCfSbqjnIZ2CmR3wPrwHmOjdF+E2M4s0sz+a2WJv4bkbvK8/x7u3wlN4FmhhZi95i1OuaSxQaWb3AF2953uy6bXM449mttq7t8LlTc79npm9YGbrzezJJvst3GNma72x/Kkt/xuJ+CMq1AGItLE7aNLz8CaB/c65cWbWBfjQzN7yvnY8MNx5yl0DfMs5t9fMugKLzexF59wdZnazc27Uca51GZ5idiOBVO975nufG42nvMRO4EPgTDNbC0wHhjrnnJklB7bpIoGjnod0dl8ArjKz5XjKX/cABnmfW9QkcQDcYmYrgIV4isgNonlnAU87T1G73cD7wLgm5y5ynmJ3y/HUs6oAqoFHzOwy4GAr2yYSNEoe0tkZ8H3n3Cjvv/7OucaeR9WRF3nGSs4DJjjnRgLL8NQJOtm5T+Rwk8f1QJTz7K8wHk9V1UuBOX60Q6RNKXlIZ1OJZ4vPRm8CN3lLYWNmg80s/jjv6waUO+cOmtlQPBVLG9U2vv8Y84HLveMqaXi2mF10osC8+zh0c869DvwAzy0vkbCkMQ/pbFYCdd7bT/8H3IfnltFS76B1KUe37GxqDnCjma3EU8F0YZPnHgJWmtlS59yVTY7PxrOP9Ao81VJvd87t8iaf40kEXjazWDy9ltta1EKRNqCpuiIi4jfdthIREb8peYiIiN+UPERExG9KHiIi4jclDxER8ZuSh4iI+E3JQ0RE/Pb/sSqRlbRT37AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "iterations = np.arange(len(cost_list)) * 10000\n",
    "plt.plot(iterations,cost_list)\n",
    "plt.xlabel(\"iterations\")\n",
    "plt.ylabel(\"avg cost\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loss above really fluctuates (noisy) due to random choices of x and y at each iteration (epoch)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to learn about Pytorch, feel free to check out its documentation and more tutorials [here](https://pytorch.org/). We will be discussing more Pytorch in depth in later weeks of the quarter when we talk about neural networks, so stay tuned and excited!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}