iui-group-l-name-zensiert/0-pilot-project/MNIST-template.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7a0c752a",
   "metadata": {},
   "source": [
    "### Load MNIST dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e07d82fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python ≥3.5 is required\n",
    "import sys\n",
    "assert sys.version_info >= (3, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1f97dcb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# scikit-learn ≥0.20 is required\n",
    "import sklearn\n",
    "assert sklearn.__version__ >= \"0.20\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "01f83832",
   "metadata": {},
   "outputs": [],
   "source": [
    "# common imports\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "affa0e2b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sklearn.utils.Bunch"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import function to scikit-learn datasets\n",
    "from sklearn.datasets import fetch_openml\n",
    "\n",
    "# load specified dataset (MNIST)\n",
    "mnist = fetch_openml('mnist_784', version=1, as_frame=False)\n",
    "\n",
    "# print type of dataset\n",
    "type(mnist)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d51fd43",
   "metadata": {},
   "source": [
    "Bunch objects are sometimes used as an output for functions and methods. They extend dictionaries by enabling values to be accessed by key, bunch[\"value_key\"], or by an attribute, bunch.value_key.\\\n",
    "=> dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "78be57ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['name', 'age'])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reminder of how dicts work\n",
    "example = {'name': 'somename', 'age': 15}\n",
    "example.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d0450c41",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['data', 'target', 'frame', 'categories', 'feature_names', 'target_names', 'DESCR', 'details', 'url'])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let us check out the keys of the mnist dataset\n",
    "mnist.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e61e2adb",
   "metadata": {},
   "source": [
    "Datasets loaded by Scikit-Learn generally have a similar dictionary structure, including the following:\\\n",
    "* __DESCR__ a key describing the dataset\n",
    "* __data__ a key containing an array with one row per instance and one column per feature\n",
    "* __target__ a key containing an array with labels, one for each row of the data key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fe285433",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"**Author**: Yann LeCun, Corinna Cortes, Christopher J.C. Burges  \\n**Source**: [MNIST Website](http://yann.lecun.com/exdb/mnist/) - Date unknown  \\n**Please cite**:  \\n\\nThe MNIST database of handwritten digits with 784 features, raw data available at: http://yann.lecun.com/exdb/mnist/. It can be split in a training set of the first 60,000 examples, and a test set of 10,000 examples  \\n\\nIt is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image. It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting. The original black and white (bilevel) images from NIST were size normalized to fit in a 20x20 pixel box while preserving their aspect ratio. The resulting images contain grey levels as a result of the anti-aliasing technique used by the normalization algorithm. the images were centered in a 28x28 image by computing the center of mass of the pixels, and translating the image so as to position this point at the center of the 28x28 field.  \\n\\nWith some classification methods (particularly template-based methods, such as SVM and K-nearest neighbors), the error rate improves when the digits are centered by bounding box rather than center of mass. If you do this kind of pre-processing, you should report it in your publications. The MNIST database was constructed from NIST's NIST originally designated SD-3 as their training set and SD-1 as their test set. However, SD-3 is much cleaner and easier to recognize than SD-1. The reason for this can be found on the fact that SD-3 was collected among Census Bureau employees, while SD-1 was collected among high-school students. Drawing sensible conclusions from learning experiments requires that the result be independent of the choice of training set and test among the complete set of samples. Therefore it was necessary to build a new database by mixing NIST's datasets.  \\n\\nThe MNIST training set is composed of 30,000 patterns from SD-3 and 30,000 patterns from SD-1. Our test set was composed of 5,000 patterns from SD-3 and 5,000 patterns from SD-1. The 60,000 pattern training set contained examples from approximately 250 writers. We made sure that the sets of writers of the training set and test set were disjoint. SD-1 contains 58,527 digit images written by 500 different writers. In contrast to SD-3, where blocks of data from each writer appeared in sequence, the data in SD-1 is scrambled. Writer identities for SD-1 is available and we used this information to unscramble the writers. We then split SD-1 in two: characters written by the first 250 writers went into our new training set. The remaining 250 writers were placed in our test set. Thus we had two sets with nearly 30,000 examples each. The new training set was completed with enough examples from SD-3, starting at pattern # 0, to make a full set of 60,000 training patterns. Similarly, the new test set was completed with SD-3 examples starting at pattern # 35,000 to make a full set with 60,000 test patterns. Only a subset of 10,000 test images (5,000 from SD-1 and 5,000 from SD-3) is available on this site. The full 60,000 sample training set is available.\\n\\nDownloaded from openml.org.\""
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnist[\"DESCR\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a70a746",
   "metadata": {},
   "source": [
    "### Prepare the MNIST dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9b7a120",
   "metadata": {},
   "source": [
    "$f(X) = y$\n",
    "\n",
    "$X$ is the data that we have and\\\n",
    "$y$ is what we want to predict\n",
    "\n",
    "In this example, we have images of handwritten digits $X$ and want to predict the digit $y$. In ML, we show the algorithm examples of X and y so that it learns the function $f(X) = y$. If it is successful, we can present $X$ to the algorithm that we did not train with and still get the $y$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4e02cf2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = mnist[\"data\"], mnist[\"target\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "001d736f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b344be1d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(70000, 784)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cef23e9f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(70000,)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe3b1259",
   "metadata": {},
   "source": [
    "### Plot data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "953d9415",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import plotting libraries\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b68f6cee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'numpy.ndarray'>\n",
      "(784,)\n"
     ]
    }
   ],
   "source": [
    "# numpy type\n",
    "print(type(X))\n",
    "\n",
    "example_digit = X[0]\n",
    "print(example_digit.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8779b1a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(28, 28)\n"
     ]
    }
   ],
   "source": [
    "# change shape\n",
    "example_digit = example_digit.reshape(28, 28)\n",
    "print(example_digit.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "dcc605cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAOUElEQVR4nO3dX4xUdZrG8ecF8R+DCkuHtAyRGTQmHY1AStgEg+hk8U+iwI2BGERjxAuQmQTiolzAhRdGd2YyihnTqAE2IxPCSITErIMEY4iJoVC2BZVFTeNA+FOE6Dh6gTLvXvRh0mLXr5qqU3XKfr+fpNPV56nT502Fh1Ndp7t+5u4CMPQNK3oAAK1B2YEgKDsQBGUHgqDsQBAXtfJgY8eO9YkTJ7bykEAovb29OnXqlA2UNVR2M7tT0h8kDZf0krs/nbr/xIkTVS6XGzkkgIRSqVQ1q/tpvJkNl/SCpLskdUlaYGZd9X4/AM3VyM/s0yR96u6fu/sZSX+WNCefsQDkrZGyj5f0t35fH8m2/YCZLTazspmVK5VKA4cD0Iimvxrv7t3uXnL3UkdHR7MPB6CKRsp+VNKEfl//PNsGoA01UvY9kq4zs1+Y2cWS5kvals9YAPJW96U3d//ezJZKelN9l95ecfcDuU0GIFcNXWd39zckvZHTLACaiF+XBYKg7EAQlB0IgrIDQVB2IAjKDgRB2YEgKDsQBGUHgqDsQBCUHQiCsgNBUHYgCMoOBEHZgSAoOxAEZQeCoOxAEJQdCIKyA0FQdiAIyg4EQdmBICg7EARlB4Kg7EAQlB0IgrIDQVB2IIiGVnFF+zt79mwy/+qrr5p6/LVr11bNvv322+S+Bw8eTOYvvPBCMl+xYkXVbNOmTcl9L7300mS+cuXKZL569epkXoSGym5mvZK+lnRW0vfuXspjKAD5y+PMfpu7n8rh+wBoIn5mB4JotOwu6a9mttfMFg90BzNbbGZlMytXKpUGDwegXo2W/RZ3nyrpLklLzGzm+Xdw9253L7l7qaOjo8HDAahXQ2V396PZ55OStkqalsdQAPJXd9nNbKSZjTp3W9JsSfvzGgxAvhp5NX6cpK1mdu77vOru/5PLVEPMF198kczPnDmTzN99991kvnv37qrZl19+mdx3y5YtybxIEyZMSOaPPfZYMt+6dWvVbNSoUcl9b7rppmR+6623JvN2VHfZ3f1zSelHBEDb4NIbEARlB4Kg7EAQlB0IgrIDQfAnrjn44IMPkvntt9+ezJv9Z6btavjw4cn8qaeeSuYjR45M5vfff3/V7Oqrr07uO3r06GR+/fXXJ/N2xJkdCIKyA0FQdiAIyg4EQdmBICg7EARlB4LgOnsOrrnmmmQ+duzYZN7O19mnT5+ezGtdj961a1fV7OKLL07uu3DhwmSOC8OZHQiCsgNBUHYgCMoOBEHZgSAoOxAEZQeC4Dp7DsaMGZPMn3322WS+ffv2ZD5lypRkvmzZsmSeMnny5GT+1ltvJfNaf1O+f3/1pQSee+655L7IF2d2IAjKDgRB2YEgKDsQBGUHgqDsQBCUHQiC6+wtMHfu3GRe633lay0v3NPTUzV76aWXkvuuWLEimde6jl7LDTfcUDXr7u5u6HvjwtQ8s5vZK2Z20sz299s2xsx2mNmh7HP6HQwAFG4wT+PXS7rzvG0rJe109+sk7cy+BtDGapbd3d+RdPq8zXMkbchub5A0N9+xAOSt3hfoxrn7sez2cUnjqt3RzBabWdnMypVKpc7DAWhUw6/Gu7tL8kTe7e4ldy91dHQ0ejgAdaq37CfMrFOSss8n8xsJQDPUW/ZtkhZltxdJej2fcQA0S83r7Ga2SdIsSWPN7Iik1ZKelrTZzB6WdFjSfc0ccqi74oorGtr/yiuvrHvfWtfh58+fn8yHDeP3sn4qapbd3RdUiX6V8ywAmoj/loEgKDsQBGUHgqDsQBCUHQiCP3EdAtasWVM127t3b3Lft99+O5nXeivp2bNnJ3O0D87sQBCUHQiCsgNBUHYgCMoOBEHZgSAoOxAE19mHgNTbPa9bty6579SpU5P5I488ksxvu+22ZF4qlapmS5YsSe5rZskcF4YzOxAEZQeCoOxAEJQdCIKyA0FQdiAIyg4EwXX2IW7SpEnJfP369cn8oYceSuYbN26sO//mm2+S+z7wwAPJvLOzM5njhzizA0FQdiAIyg4EQdmBICg7EARlB4Kg7EAQXGcPbt68ecn82muvTebLly9P5qn3nX/iiSeS+x4+fDiZr1q1KpmPHz8+mUdT88xuZq+Y2Ukz299v2xozO2pm+7KPu5s7JoBGDeZp/HpJdw6w/ffuPjn7eCPfsQDkrWbZ3f0dSadbMAuAJmrkBbqlZtaTPc0fXe1OZrbYzMpmVq5UKg0cDkAj6i37HyVNkjRZ0jFJv612R3fvdveSu5c6OjrqPByARtVVdnc/4e5n3f2fktZJmpbvWADyVlfZzaz/3xbOk7S/2n0BtIea19nNbJOkWZLGmtkRSaslzTKzyZJcUq+kR5s3Iop04403JvPNmzcn8+3bt1fNHnzwweS+L774YjI/dOhQMt+xY0cyj6Zm2d19wQCbX27CLACaiF+XBYKg7EAQlB0IgrIDQVB2IAhz95YdrFQqeblcbtnx0N4uueSSZP7dd98l8xEjRiTzN998s2o2a9as5L4/VaVSSeVyecC1rjmzA0FQdiAIyg4EQdmBICg7EARlB4Kg7EAQvJU0knp6epL5li1bkvmePXuqZrWuo9fS1dWVzGfOnNnQ9x9qOLMDQVB2IAjKDgRB2YEgKDsQBGUHgqDsQBBcZx/iDh48mMyff/75ZP7aa68l8+PHj1/wTIN10UXpf56dnZ3JfNgwzmX98WgAQVB2IAjKDgRB2YEgKDsQBGUHgqDsQBBcZ/8JqHUt+9VXX62arV27Nrlvb29vPSPl4uabb07mq1atSub33ntvnuMMeTXP7GY2wcx2mdlHZnbAzH6dbR9jZjvM7FD2eXTzxwVQr8E8jf9e0nJ375L075KWmFmXpJWSdrr7dZJ2Zl8DaFM1y+7ux9z9/ez215I+ljRe0hxJG7K7bZA0t0kzAsjBBb1AZ2YTJU2R9J6kce5+LIuOSxpXZZ/FZlY2s3KlUmlkVgANGHTZzexnkv4i6Tfu/vf+mfetDjngCpHu3u3uJXcvdXR0NDQsgPoNquxmNkJ9Rf+Tu5/7M6gTZtaZ5Z2STjZnRAB5qHnpzcxM0suSPnb33/WLtklaJOnp7PPrTZlwCDhx4kQyP3DgQDJfunRpMv/kk08ueKa8TJ8+PZk//vjjVbM5c+Yk9+VPVPM1mOvsMyQtlPShme3Ltj2pvpJvNrOHJR2WdF9TJgSQi5pld/fdkgZc3F3Sr/IdB0Cz8DwJCIKyA0FQdiAIyg4EQdmBIPgT10E6ffp01ezRRx9N7rtv375k/tlnn9UzUi5mzJiRzJcvX57M77jjjmR+2WWXXfBMaA7O7EAQlB0IgrIDQVB2IAjKDgRB2YEgKDsQRJjr7O+9914yf+aZZ5L5nj17qmZHjhypa6a8XH755VWzZcuWJfet9XbNI0eOrGsmtB/O7EAQlB0IgrIDQVB2IAjKDgRB2YEgKDsQRJjr7Fu3bm0ob0RXV1cyv+eee5L58OHDk/mKFSuqZldddVVyX8TBmR0IgrIDQVB2IAjKDgRB2YEgKDsQBGUHgjB3T9/BbIKkjZLGSXJJ3e7+BzNbI+kRSZXsrk+6+xup71UqlbxcLjc8NICBlUollcvlAVddHswv1Xwvabm7v29moyTtNbMdWfZ7d/+vvAYF0DyDWZ/9mKRj2e2vzexjSeObPRiAfF3Qz+xmNlHSFEnn3uNpqZn1mNkrZja6yj6LzaxsZuVKpTLQXQC0wKDLbmY/k/QXSb9x979L+qOkSZImq+/M/9uB9nP3bncvuXupo6Oj8YkB1GVQZTezEeor+p/c/TVJcvcT7n7W3f8paZ2kac0bE0CjapbdzEzSy5I+dvff9dve2e9u8yTtz388AHkZzKvxMyQtlPShme3Ltj0paYGZTVbf5bheSel1iwEUajCvxu+WNNB1u+Q
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot example digit\n",
    "plt.imshow(example_digit, cmap=mpl.cm.binary)\n",
    "#plt.axis(\"off\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6d41d752",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n",
      "<class 'str'>\n"
     ]
    }
   ],
   "source": [
    "# plot label of example image\n",
    "print(y[0])\n",
    "print(type(y[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "230cfd35",
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert string labels to int\n",
    "y = y.astype(np.uint8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "25a3a2e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to quickly plot an image\n",
    "def plot_digit(data):\n",
    "    image = data.reshape(28, 28)\n",
    "    plt.imshow(image, cmap = mpl.cm.binary, interpolation=\"nearest\")\n",
    "    plt.axis(\"off\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "f1552762",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAGrUlEQVR4nO3dX2jPexzH8e90kqIt+VNTcuWeceVmw40kLtBcrJSUKBRyIRcLF3KhFBcuTflTEjXXuKKVNbnb7RQXUlsiUjvXp/Z7/zqbP69tj8elV1/7NufZt86n3/fXMT093QB5lvztGwBmJk4IJU4IJU4IJU4I9U+b3f/Khd+vY6Y/9OSEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUOKEUO2+ApAF5s2bN+V+8+bNltudO3fKaw8fPlzuJ0+eLPeenp5yX2w8OSGUOCGUOCGUOCGUOCGUOCGUOCFUx/T0dLWXI3nGxsbKffv27eU+NTX1C+/mv7q6usr98+fPv+1nh+uY6Q89OSGUOCGUOCGUOCGUOCGUOCGUOCGUz3POMyMjI+W+f//+cp+cnCz3jo4Zj9yapmmazs7O8tqlS5eW+6dPn8r91atXLbctW7bM6WfPR56cEEqcEEqcEEqcEEqcEEqcEMpHxv6Cr1+/ttxGR0fLawcGBsp9YmKi3Nv8e5dHKe2OM86fP1/u/f395V7d25UrV8prL1y4UO7hfGQM5hNxQihxQihxQihxQihxQihxQigfGfsLjh071nK7d+/eH7yT/6fd1wd++fKl3Ht7e8v9xYsXLbd3796V1y5EnpwQSpwQSpwQSpwQSpwQSpwQSpwQyjnnb9DuPHB4eLjl1u7zlu309fWV+549e8r93LlzLbd169aV127evLncV65cWe7Pnz9vuc319zIfeXJCKHFCKHFCKHFCKHFCKHFCKHFCKO+tnYWxsbFy3759e7lPTU3N+mfv3r273O/fv1/u1Wcmm6b+3OTRo0fLa9esWVPu7SxZ0vpZsXz58vLaly9flntPT8+s7ukP8d5amE/ECaHECaHECaHECaHECaHECaGcc85gfHy83AcHB8v9wYMH5V6dB3Z3d5fXXrx4sdwPHDhQ7smqc87qe0Obpv13fya/D7hxzgnzizghlDghlDghlDghlDgh1KJ8Neb379/LvXo9ZNM0zbNnz8q9s7Oz3IeGhlpuW7duLa/99u1buS9WExMTf/sWfjlPTgglTgglTgglTgglTgglTgglTgi1KM85R0dHy73dOWY7T58+Lffe3t45/f0sDp6cEEqcEEqcEEqcEEqcEEqcEEqcEGpRnnOeOXOm3Nu8LrTp6+srd+eYs9Pu9/67rk3lyQmhxAmhxAmhxAmhxAmhxAmhxAmhFuw55/DwcMttbGysvLbd183t3bt3NrdEG9Xvvd2/yaZNm37x3fx9npwQSpwQSpwQSpwQSpwQSpwQSpwQasGec1bfY/njx4/y2rVr15Z7f3//rO5poWv3vaeDg4Oz/rt37txZ7levXp31353KkxNCiRNCiRNCiRNCiRNCiRNCLdijlLlYtmxZuXd3d/+hO8nS7qjkypUr5X7t2rVyX79+fcvt7Nmz5bUrVqwo9/nIkxNCiRNCiRNCiRNCiRNCiRNCiRNCOeecwWJ+9WX12tB255QPHz4s93379pX748ePy32x8eSEUOKEUOKEUOKEUOKEUOKEUOKEUAv2nHN6enpWW9M0zZMnT8r9xo0bs7mlCNevXy/3y5cvt9wmJyfLawcGBsp9aGio3PkvT04IJU4IJU4IJU4IJU4IJU4IJU4ItWDPOTs6Oma1NU3TfPz4sdxPnTpV7keOHCn3VatWtdxev35dXnv37t1yf/v2bblPTEyU+4YNG1puu3btKq89ceJEufP/eHJCKHFCKHFCKHFCKHFCKHFCqAV7lDIXP3/+LPdbt26V+6NHj8q9q6ur5TY+Pl5eO1fbtm0r9x07drTcLl269Ktvh4InJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4TqaPOayPodksHev3/fcjt48GB57cjIyJx+drtXb7b7yFpl9erV5X7o0KFyn8+v9VzAZvwPwpMTQokTQokTQokTQokTQokTQokTQi3Yc87Khw8fyv327dvlXn1NXtPM7Zzz9OnT5bXHjx8v940bN5Y7kZxzwnwiTgglTgglTgglTgglTgglTgi1KM85IYxzTphPxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmhxAmh/mmzz/jVZMDv58kJocQJocQJocQJocQJocQJof4Ftv8iCGE1mZwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# quickly plot a single digit\n",
    "plot_digit(X[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "74b3a063",
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to quickly plot several digits\n",
    "def plot_digits(instances, **options):\n",
    "    size = 28\n",
    "    images = [instance.reshape(size,size) for instance in instances]\n",
    "    image = np.concatenate(images, axis=1)\n",
    "    plt.imshow(image, cmap = mpl.cm.binary, **options)\n",
    "    plt.axis(\"off\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "949b3914",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgQAAABACAYAAABlVG/FAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAzS0lEQVR4nO2d6XOb13X/v9gXYt8IkABJkAT3nYoWyhYt2fJuxU5aN46dzmT8sm2SV/0nMm3euW4nXTxTN5mkit1kPJEsK5Jsi1pISlzAfQUBEgCx7zue3wv97g0pUbIckwAoP58ZTyZcxAvgee5z7jnf8z0chmHAwsLCwsLC8u2GW+4FsLCwsLCwsJQfNiBgYWFhYWFhYQMCFhYWFhYWFjYgYGFhYWFhYQEbELCwsLCwsLCADQhYWFhYWFhYAPC/4vtsTyILCwsLC8uTBWevL7IZAhYWFhYWFhY2IGBhYWFhYWFhAwIWFhYWFhYWfLWGgIWFhYVlB8ViEbFYDD6fD3a7HV6vFx6PBydOnIDVaoXZbIZYLAaHs2eZloWlYmEDAhbKzrkW7GZWPhiGQbFYBIfDAZf75CfxCoXCA1/jcDj0v0qjUCggHA5jbW0NFy9exNzcHKamppDJZMAwDHQ6HUQiUUWufScMw9B7/lEzbbhcbsW/Fpb9gfMVw43YLoNvCalUCuPj44jH4wgGg+jo6EBfX1+5l/WtolAoIBQKYXp6Gu+99x6effZZfP/734dCoYBIJCr38vadQCCA7e1t/OIXv4Df76cPJQ6Hg66uLjQ3N+Ps2bMwmUxlXumfYRgGgUAAH3zwARYWFvDZZ58hFoshGo3CarXCYrHg5z//OVpbWyGRSMq93IfidDqxtbWF27dvY2trC6urq8hmsw88+Hk8Hn70ox+hra0NVqsVAoGgTCsG3G43IpEIuFwuhEIhLBYLeDxe2dZzyNkzwmMzBCwAgGw2i+XlZQQCAWxtbUGhULABQYlhGAapVAobGxv4v//7P6hUKpw+fRoikeiJDAii0SjcbjcuXLgAp9NJv87hcBCPx1EsFnHy5MkyrnA3DMMgmUwiGAxiamoKi4uLWF9fB3BvzYuLi9ja2oLf70ddXV3FlA2KxSLNOpH/9Xg8WFpawo0bN7CysoKpqSmk02kAu7ODfD4fvb29kEgk0Gq1qKqqglgsLsm6GYZBoVBAPp9HLpeD2+3G9vY2uFwupFIp1Go1xGIxhEJhRbzPTwJsQMACAAiFQnjvvfewvb2NbDYLtVqNl156iU0Xlgkul4v5+Xn8+te/xhtvvIGenp5yL2lfKRaLGBsbw+TkJFKpFP26WCyGQqFAW1sbjh49CoVCUcZV/pl8Po9UKoX/+I//wMzMDD799FPEYjEwDAOFQgGlUoloNAoAuH79OqLRKF555ZWyB3LFYhGbm5tIJpMIBAKIRqPY3t7G1atXcfPmTQQCAaRSKWSzWQAPlgoLhQI++OADfPLJJxgeHkZbWxveeecd8PkH/+iIRCKYm5vD9PQ0bt68iaWlJXg8HvB4PCiVSrz44otoa2vDyy+/DLFYXPb3+kmg7AEBiVpJFJhMJmnqMJFI0LoccO9iFYvFUKlUkEgkJYtUvw3k83lsbW3B6/UCAPx+P8LhMGQyGXujlYlYLAaXy4VEIlHupew75JTqdDqRz+fpg0ipVKKhoQFmsxlGo7Eirj2GYRAMBrG9vY25uTnMz88jHA4DAPR6PYxGI2pra7GyskLLB+FwGMVisazrLhQKyGQyWFtbQzgchsfjoQHB4uIiNjY2kM1mkc/nAfw5GLg/KHC73YhGozCbzZDJZCgUCuDxeAd2UCgWi3S9s7OzsNvtmJ6exvr6Ovx+P7hcLhQKBWpra1EsFtHW1gaNRgONRoNkMkmDGwIpMYhEIgiFQgiFwrJocxiGQS6XQyaTQTwe3/U98mzj8/kQiUTgcrllKYeULSAgghby0Pf7/djc3MSNGzdoAHDlyhWsrKzQVJZUKkV7ezvefPNN9PX1sSntA6BYLKJYLGJtbQ2fffYZjh8/jvr6+nIv61tJNBrFxsbGExcQFAoF5HI5jI+P4/PPP0cymaTfGxgYwE9+8hO0tLTAYrGU5CT6KIrFIgqFAv7whz/giy++wGeffYZAIACRSIT6+no8++yzOHnyJIaHh/HP//zPuH79OoRCYdmDAeDe9eP1evHzn/8cKysr8Hg89ACWzWYfeHA+jFQqhUKhALfbDZPJhFQqBQ6HA6FQeCDrjsVi+N///V/Y7Xb89re/RSqVQiqVooFLsVhENBrFH//4R4yNjWFsbAxHjhzB0NAQbty4gfX19V3BikKhgNVqhdVqhc1mg8VigVKpPJC1P4psNguXy4WFhQVcvnx51/dEIhF6enqg1+vR0tICuVwOlUpV8jWW7G4j9aBEIkGj6FQqBafTiXg8jmg0Cr/fD7vdTgOCzc1NBINBMAwDLpcLLpeLYrEImUx2YBdjKdip7n1YVF5ucrkc0un0ngrwwwSpmaZSKUSjUSQSCSSTSfj9fmSzWej1eojFYqjValRVVZXlJnwYJHP2FcLfQwMJBIigbWtrC5FIZNfDUyKRoKamBkqlsqwCNuDeg4dkBhYXF7G4uIhoNAqGYWA2m9HU1IT+/n40NjZCq9VCp9NBp9PBYrHAaDSWXfCWz+eRyWTg8/ng8/kQiUTA5/MhkUhgNBohl8v3/D0SBG1vbyMUCtGvBQIB+Hw+BAIBMAxzIHuw3++Hx+PB9PQ0lpaWEA6HIRKJoNFooFQqIZFI6P5J9qbNzU3IZDIIBAI4HA74/X66d2WzWUilUoTDYWxvb8PtdqOtrQ1GoxENDQ2QSCQHuvfGYjGkUim4XC6Ew2Gsrq7C4XBgbm5u130tEAiQSqWgVqvhdrtRX1+Pnp4eSCSSkj7rShYQFAoFJJNJrKysYHZ2FjMzM3A6nbh8+TJ96AO721/EYjEEAgGUSiWEQiGkUikaGhrQ09MDnU5XqqXvG+ThRKJ0Agl2vg0tZqWGBAMul4sKwRwOBz777DMEg0E8//zzsFgsOHLkCFpaWnDkyJFyL/mJJZPJIBgM4g9/+AMuXLiAiYkJmnonyGQyWCwWSKXS8izy/1MsFpHL5TA/P4+rV6/i0qVLmJiYAMMwUKvVePrppzE4OIi33nqLps9NJhPa29tx+vRp1NTUlD2gyefzSKfTCAQCdI+VSqUwm834zne+g97e3j1/j6S0L126hFu3bgG4d0BYXFyEUCjE3NwcGhsboVar93W9hUIBU1NTmJ2dxfnz52kwYjQa0draiv7+ftTV1SGXy9Hy8tLSEn7zm9/A6XTi2rVrqK2thVwupwfNSCSCQqEALpcLPp8PoVCIkydPoq2tDf/wD/9w4Fkoh8MBp9OJDz74AA6HA7Ozs8hms8hkMg/8LMm6qFQqnD17Fj/72c9gNpthMBgObH33U7KAIBKJYGpqClNTU7h16xY8Hg9CoRDi8ThyuRyAey0ufD4fYrEYYrEYfX19qKmpgcVioWrSuro6aDSaim3p2Rm9JhIJrKys0As4FoshkUjA6/VSIZVcLkdbWxtqa2ths9nKvPrDRS6Xg9/vRyQSgcfj2fU9EnzF43EsLi7S7gmfz4dgMIhwOIx0Oo2FhQUEAgFks1kUi0V0dnZCIBCUPVUNAOl0GsFgEF6vF263GzqdruwPmW9CIpHAxsYGHA4H1tfXd4kJSbBfW1sLPp9f9uA4Ho9jZWUFd+7cwZdffont7W1wOBxayhgeHkZjY+OuWnpbWxvUajW0Wm1FdBhUVVXBYDDgzJkz6OjoAMMwqK6uRktLC+rr61FTU7Pn7yWTSYRCIdy5c2fX18neRu6t/SSbzSKVSmFsbAxTU1NIJpOQSqWw2Wzo6enB8ePHYTaboVaraQYjm82Cz+fTa6e9vR1WqxVKpRITExMIhUIIh8PgcrmQy+X0utvY2EAqlcLvf/97NDY2YmhoCFKpdN+fKQzDYHR0FBMTE1RfQjojlEolfD4fQqEQzWbkcjlks1nE43Fks1kIBIKSZ5lKtusFg0FcvXoVIyMjD9RPCFwuF2KxGBqNBnq9Hq+88gq
      "text/plain": [
       "<Figure size 648x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# quickly plot several digits\n",
    "plt.figure(figsize=(9,9))\n",
    "plot_digits(X[100:110])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec8a9d34",
   "metadata": {},
   "source": [
    "### Prepare data for machine learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "febbd286",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "70000"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how many images do we have\n",
    "len(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "fff839b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we use the first 60000 for training and test with the other 10000 images\n",
    "X_train, X_test, y_train, y_test = X[:100], X[100:], y[:100], y[100:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bdbeb4e",
   "metadata": {},
   "source": [
    "### Train classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "4c32ae9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import support vector machine\n",
    "import sklearn.svm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "fe06ae55",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=10, kernel='poly')"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# specify the parameter of the SVM\n",
    "classifier = sklearn.svm.SVC(C=10, gamma=\"scale\", kernel=\"poly\") #gamma=0.1 degree=3\n",
    "\n",
    "# train the SVM\n",
    "classifier.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "e6209258",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAGJUlEQVR4nO3dO2gUexjGYVds7BIlaVRsbLQTLUQRC0G0shBBCxtFwRtY2WjlpbeJlSBio2UQrMSAKIJgFwUxsdLGIuAFiRfYUx9O9pvDTja+yT5P6cvsjpEfA/7ZTafb7a4C8qz+2zcALEycEEqcEEqcEEqcEGpNw+6/cmHwOgv9oScnhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhBInhFrzt29gGM3NzfXc1q1bt4R3srTm5+fL/dy5cz23Bw8elNe+ePGi3Ldv317uiTw5IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZRzzgG4c+dOuU9MTPTcHj16VF67cePGvu4pwczMTLnfu3ev79eenZ0td+ecwKIRJ4QSJ4QSJ4QSJ4QSJ4RylNKHd+/elfvFixfL/devXz23qamp8toTJ06Ue7I2RyXDyJMTQokTQokTQokTQokTQokTQokTQjnn7MPv37/LvTrHbHL37t1yTz7n/PTpU7lPTk72/dpbt24t9127dvX92qk8OSGUOCGUOCGUOCGUOCGUOCGUOCGUc04WTdM5ZtNXY3Y6nZ7btWvXymuX81eG9uLJCaHECaHECaHECaHECaHECaHECaGccw5At9vt+9rR0dFFvJOldfv27XJv83PZu3dv39cuV56cEEqcEEqcEEqcEEqcEEqcEEqcEMo5Zx8+fPhQ7tXnEpscP36872sH7fPnz+X+7du3cm/6ubT5ua1EnpwQSpwQSpwQSpwQSpwQSpwQylHKAl69elXux44dW6I7yXL58uVy//jxY6vXv379es9t/fr1rV57OfLkhFDihFDihFDihFDihFDihFDihFBDec45PT1d7leuXCn3+fn5cm/z0adbt26V+9u3b8v91KlT5b5hw4Zy//LlS89tamqqvLbJpk2byv306dM9t9Wrh+85Mnx/Y1gmxAmhxAmhxAmhxAmhxAmhxAmhVuw557Nnz3puhw8fLq/9+vXrYt/O//by5ctW+5MnT8q9+rmsWlV/ZrPt5zV3795d7mNjY61ef6Xx5IRQ4oRQ4oRQ4oRQ4oRQ4oRQ4oRQK/ac8+jRoz236jOL/0e32211/fj4eM/t58+f5bVN9970WdULFy6U+/3793tubf/ee/bsaXX9sPHkhFDihFDihFDihFDihFDihFDihFCdhrOrdgdbAzQzM1PuO3bs6Ll9//691Xvv37+/3Ju+93bz5s09tx8/fpTXXrp0qdybPs/Z5jt1m2zZsqXcnz59Wu5N36m7gi34j+LJCaHECaHECaHECaHECaHECaGW7UfGmv7b/uDBgz239+/fl9devXq13A8dOlTua9euLfc2Jicny/3169flvm/fvsW8nX85cOBAuQ/xUUlfPDkhlDghlDghlDghlDghlDghlDgh1LI952zy8OHDv30LA9F0hjoyMrI0N8LAeXJCKHFCKHFCKHFCKHFCKHFCKHFCqBV7zjms3rx5M7DXbvo85tmzZwf23sPIkxNCiRNCiRNCiRNCiRNCiRNCiRNCOedcZubm5sr95s2b5d7wKx9LJ0+eLPdt27b1/dr8lycnhBInhBInhBInhBInhBInhHKUsszcuHGj3Kenp8u90+mU+86dO3tu58+fL69lcXlyQihxQihxQihxQihxQihxQihxQijnnGGeP39e7hMTEwN9/yNHjvTcxsbGBvre/JsnJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4Ryzhnm8ePH5f7nz59Wrz8+Pl7uZ86cafX6LB5PTgglTgglTgglTgglTgglTgglTgjlnDPM7Oxsq+tHR0fLvelXBI6MjLR6fxaPJyeEEieEEieEEieEEieEEieEEieE6nS73WovR2BRLPhLUz05IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IVTTrwBc8Cv7gMHz5IRQ4oRQ4oRQ4oRQ4oRQ4oRQ/wAj3eVzPh6F+gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# take a test digit\n",
    "test_digit = X[12121]\n",
    "plot_digit(test_digit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "62773b1b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n"
     ]
    }
   ],
   "source": [
    "# see label for test digit\n",
    "print(y[12121])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "0ce21474",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4]\n"
     ]
    }
   ],
   "source": [
    "# see prediction for test digit\n",
    "print(classifier.predict([X[12121]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "78a8e8a7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.27831745,  5.82217802,  0.72414032,  2.85240395,  9.30320665,\n",
       "         3.83670072,  4.8744213 ,  7.20637122,  1.74812151,  8.27550294]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# see propability for all classes\n",
    "classifier.decision_function([X[12121]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "45d93a99",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint8)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# see the classes to understand which received which score\n",
    "classifier.classes_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc739051",
   "metadata": {},
   "source": [
    "### Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "990a5b7c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Train 100.0\n"
     ]
    }
   ],
   "source": [
    "# trainings accuracy\n",
    "wrong_images = X_train[(classifier.predict(X_train)-y_train) != 0]\n",
    "percentage = ((1-len(wrong_images)/len(X_train)) * 100)\n",
    "print(\"Accuracy Train \" + str(percentage))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "f125a37d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Test 58.92989985693848\n"
     ]
    }
   ],
   "source": [
    "# test accuracy\n",
    "wrong_images = X_test[(classifier.predict(X_test)-y_test) != 0]\n",
    "percentage = ((1-len(wrong_images)/len(X_test)) * 100)\n",
    "print(\"Accuracy Test \" + str(percentage))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdcb6e6e",
   "metadata": {},
   "source": [
    "Accuracy is strongly influenced by the distribution of the classes in the test data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be858cd5",
   "metadata": {},
   "source": [
    "#### Cross Validation\n",
    "[Find more information on cross validation here.](https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "7adb1ea7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.67647059 0.63636364 0.84848485]\n"
     ]
    }
   ],
   "source": [
    "# cross validation score\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "print(cross_val_score(classifier, X_train, y_train, cv=3, scoring=\"accuracy\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "11d22c5e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 0 1 1 9 9 1 3 1 4 3 1 3 6 1 7 1 9 1 9 4 0 9 1 1 2 1 3 7 1 1 1 1 9 0 1 6\n",
      " 0 7 6 1 8 1 9 1 9 1 1 1 3 1 0 7 1 4 8 0 9 4 1 4 1 6 0 6 5 6 1 1 0 1 7 1 6\n",
      " 3 0 1 1 1 7 6 0 2 6 7 8 1 9 0 4 6 7 4 6 8 0 7 8 3 1]\n"
     ]
    }
   ],
   "source": [
    "# prediction of classifier\n",
    "from sklearn.model_selection import cross_val_predict\n",
    "\n",
    "y_train_pred = cross_val_predict(classifier, X_train, y_train, cv=3)\n",
    "print(y_train_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b54e83a5",
   "metadata": {},
   "source": [
    "#### Precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "ef7a9e7e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8456190476190476"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score\n",
    "\n",
    "precision_score(y_train, y_train_pred, average='weighted')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da723740",
   "metadata": {},
   "source": [
    "#### Recall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "cb77bf58",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.72"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import recall_score\n",
    "\n",
    "recall_score(y_train, y_train_pred, average='weighted')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28867d1b",
   "metadata": {},
   "source": [
    "#### F1 Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "0674e0de",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7283140672193305"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "\n",
    "f1_score(y_train, y_train_pred, average='weighted')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da59da11",
   "metadata": {},
   "source": [
    "#### Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "adbdeece",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[12  1  0  0  0  0  0  0  0  0]\n",
      " [ 0 14  0  0  0  0  0  0  0  0]\n",
      " [ 0  2  2  0  0  0  0  1  0  1]\n",
      " [ 0  4  0  7  0  0  0  0  0  0]\n",
      " [ 0  4  0  0  6  0  1  0  0  0]\n",
      " [ 0  4  0  0  0  1  0  0  0  0]\n",
      " [ 0  2  0  0  0  0  9  0  0  0]\n",
      " [ 0  2  0  0  0  0  0  8  0  0]\n",
      " [ 0  2  0  0  0  0  0  0  5  1]\n",
      " [ 0  1  0  0  1  0  1  0  0  8]]\n"
     ]
    }
   ],
   "source": [
    "# confusing matrix\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "print(confusion_matrix(y_train, y_train_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "fb50c5a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.92307692 0.07692308 0.         0.         0.         0.\n",
      "  0.         0.         0.         0.        ]\n",
      " [0.         1.         0.         0.         0.         0.\n",
      "  0.         0.         0.         0.        ]\n",
      " [0.         0.33333333 0.33333333 0.         0.         0.\n",
      "  0.         0.16666667 0.         0.16666667]\n",
      " [0.         0.36363636 0.         0.63636364 0.         0.\n",
      "  0.         0.         0.         0.        ]\n",
      " [0.         0.36363636 0.         0.         0.54545455 0.\n",
      "  0.09090909 0.         0.         0.        ]\n",
      " [0.         0.8        0.         0.         0.         0.2\n",
      "  0.         0.         0.         0.        ]\n",
      " [0.         0.18181818 0.         0.         0.         0.\n",
      "  0.81818182 0.         0.         0.        ]\n",
      " [0.         0.2        0.         0.         0.         0.\n",
      "  0.         0.8        0.         0.        ]\n",
      " [0.         0.25       0.         0.         0.         0.\n",
      "  0.         0.         0.625      0.125     ]\n",
      " [0.         0.09090909 0.         0.         0.09090909 0.\n",
      "  0.09090909 0.         0.         0.72727273]]\n"
     ]
    }
   ],
   "source": [
    "cm = confusion_matrix(y_train, y_train_pred, normalize='true')\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "2f0d536a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "dddf5fe8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjEAAAGtCAYAAADnIyVRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAABQ/UlEQVR4nO3deXhU5d3/8fc3G8giazJsMaLgguIKoqCyFxQFFHF70IJFHp+fVrQiUq3Qggtai/uWKqhFsdhSHwQLPiCUCoqiyA6KiIQt7CiKhszcvz8yTgkkmURn5sxhPq/rmss5c+455zO3CfPNfZ/FnHOIiIiI+E2a1wFEREREfgoVMSIiIuJLKmJERETEl1TEiIiIiC+piBERERFfUhEjIiIivqQiRkREROLOzMab2TYzW17OejOzJ8xsrZktNbOzom1TRYyIiIgkwktAzwrWXwS0DD+GAM9G26CKGBEREYk759w8YFcFTfoAr7gSHwB1zaxxRdvMiGXAWAoEAr67lPD69eu9jlAlRx11lNcRRESkNEvozsxi+V3735SMoPwo3zmXX4X3NwUKDlreGH5tS3lvSNoiRkRERPwjXLBUpWj52VTEiIiIpCizhA78RLMJyD1ouVn4tXLpmBgREZEUZWYxe8TAVOD68FlK5wJ7nXPlTiWBRmJEREQkAcxsEtAJaGhmG4FRQCaAc+454G3gYmAt8B0wKNo2VcSIiIikqEROJznnromy3gE3V2WbKmJERERSVFqav48q8Xd6ERERSVkaiREREUlRSXZ2UpWpiBEREUlRfi9iNJ0kIiIivqSRGBERkRTl95EYFTEiIiIpyu9FjKaTRERExJc0EiMiIpKiNBKTRDp37sz8+fP54IMP+PWvf33Y+mbNmvG3v/2NOXPmMGXKFBo3bgzAKaecwvTp0/nXv/7FnDlz6NOnT1xzzp8/nz59+nDppZcyfvz4w9YXFRUxfPhwLr30UgYMGMCmTSX3v5o+fTpXXnll5HHmmWeyevVqvv3221Kvd+rUiYcffjiun6E88+bNo0ePHnTv3p38/ITezPQnU+b481teUOZE8Fte8GfmiqSlpcXs4QnnXFI+cnJyXFUejRo1cl9++aVr27ata9q0qVu+fLk7//zzS7X53//9X3fLLbe4nJwcd/nll7vJkye7nJwcd+6557p27dq5nJwc17p1a7d161bXokWLKu0/JyfHfffdd1Ef33zzjevSpYv77LPP3N69e90ll1zili1bVqrNhAkT3N133+2+++47N2XKFHfLLbcctp0lS5a4Ll26lLmPPn36uH//+99Rs8RacXGx69q1q9uwYYP74Ycf3KWXXuo+//zzmO8nlpQ5/vyW1zllTgS/5XUuYZkT+l1bq1YtF6tHorM75+I3EmNmJ5nZXWb2RPhxl5mdHK/9nXXWWXz55Zd89dVXHDhwgDfffJOePXuWanPCCSfw3nvvAfDee+9F1q9bt44vv/wSgMLCQnbs2EGDBg3iknP58uXk5ubSrFkzMjMz6dGjB3Pnzi3VZu7cuVx66aUAdOvWjQ8//BDnXKk2//znP+nRo8dh2//qq6/YtWsXZ511VlzyV2Tp0qXk5eWRm5tLVlYWvXr1Yvbs2QnPURXKHH9+ywvKnAh+ywv+zBxNkt3FusriUsSY2V3A64ABH4YfBkwysxHx2GejRo3YvHlzZHnz5s00atSoVJuVK1fSq1cvAC6++GJq165NvXr1SrU588wzyczMZP369fGIybZt20rlCgQCbNu2rdw2GRkZ1KpViz179pRq884773DRRRcdtv0ZM2bQo0cPT36gCgsLD/tshYWFCc9RFcocf37LC8qcCH7LC/7MHI2KmLL9CmjrnBvrnJsYfowFzgmvK5OZDTGzRWa2aP/+/TEP9fvf/57zzjuPWbNm0b59ezZv3kwwGIysz8nJ4amnnuK22247bOQjmSxbtozq1avTokWLw9bNnDnzsBEoERGRI1G8zk4KAU2Arw55vXF4XZmcc/lAPkAgEKhSFbF161aaNGkSWW7SpAlbt24t1aawsJAbbrgBgBo1atCrVy++/vprAGrVqsWrr77Kgw8+yMcff1yVXVdJTk5OqVyFhYXk5OSU2SYQCFBcXMy+ffuoW7duZP2MGTPKLFTWrFlDcXExrVq1ilv+igQCgcM+WyAQ8CRLZSlz/PktLyhzIvgtL/gzczQ6O6lstwGzzeyfZpYffswAZgND47HDxYsXc9xxx3HMMceQmZlJ3759mTlzZqk29evXj/wPGzp0KJMmTQIgMzOTl156iTfeeINp06bFI17EKaecwoYNG9i0aRMHDhxg5syZdOzYsVSbjh078tZbbwEwa9Ys2rZtG8kdCoV45513yixiyituEqV169asX7+egoICioqKmD59Ol26dPEsT2Uoc/z5LS8ocyL4LS/4M3M0fp9OistIjHNuhpmdQMn0UdPwy5uAj5xzwfLf+dMFg0F++9vf8vrrr5Oens6kSZNYs2YNw4cPZ8mSJcycOZP27dtzzz334Jzjgw8+YMSIksNzevfuzbnnnku9evW46qqrALj11ltZsWJFzHNmZGQwYsQI/ud//odQKESfPn1o0aIFzzzzDK1ataJTp05cdtll3HPPPVx66aUcffTRPPTQQ5H3f/zxxzRq1IhmzZodtu133nmHp556KuaZKysjI4ORI0cyePBggsEg/fr1o2XLlp7lqQxljj+/5QVlTgS/5QV/Zj7SWbIe+1HV6aRkEK+DgePlqKOO8jqCiIiUltAhjfr168fsu3bXrl0JH47RFXtFRERSlGcXqYsRf6cXERGRlKWRGBERkRTl97OTVMSIiIikKL8XMZpOEhEREV/SSIyIiEiK8vtIjIoYERGRFKUiRkRERHzJ70WMjokRERERX9JIjIiISIry+8XuVMSIiIikKE0niYiIiHhAIzEiIiIpyu8jMSpiREREUpSKmDgpLCz0OkKV+e2HwbmY3YFdREQk4ZK2iBEREZH48tsf34dSESMiIpKi/F7E6OwkERER8SWNxIiIiKQoXexOREREfEnTSSIiIiIe0EiMiIhIivL7SIyKGBERkRTl9yJG00kiIiLiSxqJERERSVF+H4lRESMiIpKi/H6Ktb/Ti4iISMrSSIyIiEiK0nSSiIiI+JLfi5iUmk6aN28ePXr0oHv37uTn53sdJ6oXX3yRwsJCli1b5nWUSvNbH4MyJ4Lf8oIyJ4Lf8oI/Mx/RnHPJ+oip4uJi17VrV7dhwwb3ww8/uEsvvdR9/vnnMd0HENPHBRdc4M4880y3bNmymG+75H99bCWij2NNmePPb3mdU+ZE8Fte5xKWOaHftaeccoqL1SPR2Z1zqTMSs3TpUvLy8sjNzSUrK4tevXoxe/Zsr2NV6N///je7du3yOkal+bGPlTn+/JYXlDkR/JYX/Jk5GjOL2cMLKVPEFBYW0qhRo8hyIBCgsLDQw0RHHj/2sTLHn9/ygjIngt/ygj8zH+kSXsSY2aAK1g0xs0VmtkhzjSIiIvGVlpYWs4cXvDg76Q/AhLJWOOfygR+rFxfLnQYCAbZu3RpZLiwsJBAIxHIXKc+PfazM8ee3vKDMieC3vODPzNHo7KQymNnSch7LAE/+j7du3Zr169dTUFBAUVER06dPp0uXLl5EOWL5sY+VOf78lheUORH8lhf8mflIF6+RmADQA9h9yOsGLIjTPiuUkZHByJEjGTx4MMFgkH79+tGyZUsvolTaa6+9RqdOnWjYsCEFBQWMGjWK8ePHex2rXH7sY2WOP7/lBWVOBL/lBX9mjsbvIzHmXExnbUo2avYiMME5914Z615zzl1bic3EPlic+e2HIR7/70VE5GdJ6BfJ2WefHbMvgo8//jjhX4JxGYlxzv2qgnWVKWBEREREKqTbDoiIiKQov80gHEpFjIiISIry6tToWPF3ehEREUlZGokRERFJUZpOEhEREV/SdJKIiIiIBzQSIyIikqI0nSQiIiK+5PciRtNJIiIi4ksaiREREUlROrBXREREfMnMYvaoxL56mtkaM1trZiPKWH+Mmc0xs8VmttTMLo62TRUxIiI
      "text/plain": [
       "<Figure size 720x504 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "set_digits = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }\n",
    "\n",
    "df_cm = pd.DataFrame(cm, index=set_digits, columns=set_digits)\n",
    "plt.figure(figsize = (10,7))\n",
    "sn_plot = sn.heatmap(df_cm, annot=True, cmap=\"Greys\")\n",
    "plt.ylabel(\"True Label\")\n",
    "plt.xlabel(\"Predicted Label\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "44537aae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      0.92      0.96        13\n",
      "           1       0.39      1.00      0.56        14\n",
      "           2       1.00      0.33      0.50         6\n",
      "           3       1.00      0.64      0.78        11\n",
      "           4       0.86      0.55      0.67        11\n",
      "           5       1.00      0.20      0.33         5\n",
      "           6       0.82      0.82      0.82        11\n",
      "           7       0.89      0.80      0.84        10\n",
      "           8       1.00      0.62      0.77         8\n",
      "           9       0.80      0.73      0.76        11\n",
      "\n",
      "    accuracy                           0.72       100\n",
      "   macro avg       0.88      0.66      0.70       100\n",
      "weighted avg       0.85      0.72      0.73       100\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print(classification_report(y_train, y_train_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57d96f56",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}