introduction_to_nlp_in_tensorflow (1).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "introduction_to_nlp_in_tensorflow.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vtAgo5zYCClj"
      },
      "source": [
        "*italicized text*#  Natural Language Processing with TensorFlow\n",
        "\n",
        "![](https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/images/08-example-nlp-problems.png)\n",
        "*A handful of example natural language processing (NLP) and natural language understanding (NLU) problems. These are also often referred to as sequence problems (going from one sequence to another).*\n",
        "\n",
        "The main goal of [natural language processing (NLP)](https://becominghuman.ai/a-simple-introduction-to-natural-language-processing-ea66a1747b32) is to derive information from natural language.\n",
        "\n",
        "Natural language is a broad term but you can consider it to cover any of the following:\n",
        "* Text (such as that contained in an email, blog post, book, Tweet)\n",
        "* Speech (a conversation you have with a doctor, voice commands you give to a smart speaker)\n",
        "\n",
        "Under the umbrellas of text and speech there are many different things you might want to do.\n",
        "\n",
        "If you're building an email application, you might want to scan incoming emails to see if they're spam or not spam (classification).\n",
        "\n",
        "If you're trying to analyse customer feedback complaints, you might want to discover which section of your business they're for.\n",
        "\n",
        "> 🔑 **Note:** Both of these types of data are often referred to as *sequences* (a sentence is a sequence of words). So a common term you'll come across in NLP problems is called *seq2seq*, in other words, finding information in one sequence to produce another sequence (e.g. converting a speech command to a sequence of text-based steps).\n",
        "\n",
        "To get hands-on with NLP in TensorFlow, we're going to practice the steps we've used previously but this time with text data:\n",
        "\n",
        "```\n",
        "Text -> turn into numbers -> build a model -> train the model to find patterns -> use patterns (make predictions)\n",
        "```\n",
        "\n",
        "> 📖 **Resource:** For a great overview of NLP and the different problems within it, read the article [*A Simple Introduction to Natural Language Processing*](https://becominghuman.ai/a-simple-introduction-to-natural-language-processing-ea66a1747b32).\n",
        "\n",
        "\n",
        "\n",
        "Let's get specific hey?\n",
        "\n",
        "* Downloading a text dataset\n",
        "* Visualizing text data\n",
        "* Converting text into numbers using tokenization\n",
        "* Turning our tokenized text into an embedding\n",
        "* Modelling a text dataset\n",
        "  * Starting with a baseline (TF-IDF)\n",
        "  * Building several deep learning text models\n",
        "    * Dense, LSTM, GRU, Conv1D, Transfer learning\n",
        "* Comparing the performance of each our models\n",
        "* Combining our models into an ensemble\n",
        "* Saving and loading a trained model\n",
        "* Find the most wrong predictions\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "Don't worry if you make mistakes, we all do. The way to get better and make less mistakes is to write more code.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Zh2N1hZtvpN"
      },
      "source": [
        "## Check for GPU\n",
        "\n",
        "In order for our deep learning models to run as fast as possible, we'll need access to a GPU.\n",
        "\n",
        "In Google Colab, you can set this up by going to Runtime -> Change runtime type -> Hardware accelerator -> GPU.\n",
        "\n",
        "After selecting GPU, you may have to restart the runtime."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DEYTFigmc3CI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "57c44ca3-b01f-412c-a670-73b70e861937"
      },
      "source": [
        "# Check for GPU\n",
        "!nvidia-smi -L"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "GPU 0: Tesla K80 (UUID: GPU-7c8181f1-42c3-e0c6-0862-932bb75fde7b)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gS3YnNNI8oFk"
      },
      "source": [
        "## Get helper functions\n",
        "\n",
        "In past modules, we've created a bunch of helper functions to do small tasks required for our notebooks.\n",
        "\n",
        "Rather than rewrite all of these, we can import a script and load them in from there.\n",
        "\n",
        "The script containing our helper functions can be [found on GitHub](https://github.com/mrdbourke/tensorflow-deep-learning/blob/main/extras/helper_functions.py)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aFOHPqgE8pv-",
        "outputId": "523e55ff-21ee-41ed-e3a4-7e19274f9ea8"
      },
      "source": [
        "# Download helper functions script\n",
        "!wget https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/extras/helper_functions.py"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2021-09-23 05:25:54--  https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/extras/helper_functions.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.110.133, 185.199.111.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 10246 (10K) [text/plain]\n",
            "Saving to: ‘helper_functions.py.4’\n",
            "\n",
            "\rhelper_functions.py   0%[                    ]       0  --.-KB/s               \rhelper_functions.py 100%[===================>]  10.01K  --.-KB/s    in 0s      \n",
            "\n",
            "2021-09-23 05:25:54 (43.6 MB/s) - ‘helper_functions.py.4’ saved [10246/10246]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ICFbSkoM85tq"
      },
      "source": [
        "# Import series of helper functions for the notebook\n",
        "from helper_functions import unzip_data, create_tensorboard_callback, plot_loss_curves, compare_historys"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cCZrclc2COWW"
      },
      "source": [
        "## Download a text dataset\n",
        "\n",
        "Let's start by download a text dataset. We'll be using the [Real or Not?](https://www.kaggle.com/c/nlp-getting-started/data) datset from Kaggle which contains text-based Tweets about natural disasters. \n",
        "\n",
        "The Real Tweets are actually about diasters, for example:\n",
        "\n",
        "```\n",
        "Jetstar and Virgin forced to cancel Bali flights again because of ash from Mount Raung volcano\n",
        "```\n",
        "\n",
        "The Not Real Tweets are Tweets not about diasters (they can be on anything), for example:\n",
        "\n",
        "```\n",
        "'Education is the most powerful weapon which you can use to change the world.' Nelson #Mandela #quote\n",
        "```\n",
        "\n",
        "For convenience, the dataset has been [downloaded from Kaggle](https://www.kaggle.com/c/nlp-getting-started/data) (doing this requires a Kaggle account) and uploaded as a downloadable zip file. \n",
        "\n",
        "> 🔑 **Note:** The original downloaded data has not been altered to how you would download it from Kaggle."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C0FEcci5IH8S",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "75a40c19-6eae-4323-d84e-c6af405fc0ba"
      },
      "source": [
        "# Download data (same as from Kaggle)\n",
        "!wget \"https://storage.googleapis.com/ztm_tf_course/nlp_getting_started.zip\"\n",
        "\n",
        "# Unzip data\n",
        "unzip_data(\"nlp_getting_started.zip\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2021-09-23 05:25:57--  https://storage.googleapis.com/ztm_tf_course/nlp_getting_started.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 66.102.1.128, 172.253.120.128, 74.125.206.128, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|66.102.1.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 607343 (593K) [application/zip]\n",
            "Saving to: ‘nlp_getting_started.zip.4’\n",
            "\n",
            "\r          nlp_getti   0%[                    ]       0  --.-KB/s               \rnlp_getting_started 100%[===================>] 593.11K  --.-KB/s    in 0.006s  \n",
            "\n",
            "2021-09-23 05:25:57 (101 MB/s) - ‘nlp_getting_started.zip.4’ saved [607343/607343]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wBIR6tTI9QcR"
      },
      "source": [
        "Unzipping `nlp_getting_started.zip` gives the following 3 `.csv` files:\n",
        "* `sample_submission.csv` - an example of the file you'd submit to the Kaggle competition of your model's predictions.\n",
        "* `train.csv` - training samples of real and not real diaster Tweets.\n",
        "* `test.csv` - testing samples of real and not real diaster Tweets."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7HpxZKYdD6V-"
      },
      "source": [
        "## Visualizing a text dataset\n",
        "\n",
        "Once you've acquired a new dataset to work with, what should you do first?\n",
        "\n",
        "Explore it? Inspect it? Verify it? Become one with it?\n",
        "\n",
        "All correct.\n",
        "\n",
        "Remember the motto: visualize, visualize, visualize.\n",
        "\n",
        "Right now, our text data samples are in the form of `.csv` files. For an easy way to make them visual, let's turn them into pandas DataFrame's.\n",
        "\n",
        "> 📖 **Reading:** You might come across text datasets in many different formats. Aside from CSV files (what we're working with), you'll probably encounter `.txt` files and `.json` files too. For working with these type of files, I'd recommend reading the two following articles by RealPython:\n",
        "* [How to Read and Write Files in Python](https://realpython.com/read-write-files-python/)\n",
        "* [Working with JSON Data in Python](https://realpython.com/python-json/)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qRvkeYEJIKsw",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "052a1d3e-1d52-47e9-ea39-df6dd4d149ca"
      },
      "source": [
        "# Turn .csv files into pandas DataFrame's\n",
        "import pandas as pd\n",
        "train_df = pd.read_csv(\"train.csv\")\n",
        "test_df = pd.read_csv(\"test.csv\")\n",
        "train_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Our Deeds are the Reason of this #earthquake M...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Forest fire near La Ronge Sask. Canada</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>All residents asked to 'shelter in place' are ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>6</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>13,000 people receive #wildfires evacuation or...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>7</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Just got sent this photo from Ruby #Alaska as ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id keyword  ...                                               text target\n",
              "0   1     NaN  ...  Our Deeds are the Reason of this #earthquake M...      1\n",
              "1   4     NaN  ...             Forest fire near La Ronge Sask. Canada      1\n",
              "2   5     NaN  ...  All residents asked to 'shelter in place' are ...      1\n",
              "3   6     NaN  ...  13,000 people receive #wildfires evacuation or...      1\n",
              "4   7     NaN  ...  Just got sent this photo from Ruby #Alaska as ...      1\n",
              "\n",
              "[5 rows x 5 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1xGqlnQaLmaT"
      },
      "source": [
        "The training data we downloaded is probably shuffled already. But just to be sure, let's shuffle it again."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ACCE7h6OMVjR",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "51f2ae2e-df2a-4e15-a618-7087ecba1914"
      },
      "source": [
        "# Shuffle training dataframe\n",
        "train_df_shuffled = train_df.sample(frac=1, random_state=42) # shuffle with random_state=42 for reproducibility\n",
        "train_df_shuffled.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2644</th>\n",
              "      <td>3796</td>\n",
              "      <td>destruction</td>\n",
              "      <td>NaN</td>\n",
              "      <td>So you have a new weapon that can cause un-ima...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2227</th>\n",
              "      <td>3185</td>\n",
              "      <td>deluge</td>\n",
              "      <td>NaN</td>\n",
              "      <td>The f$&amp;amp;@ing things I do for #GISHWHES Just...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5448</th>\n",
              "      <td>7769</td>\n",
              "      <td>police</td>\n",
              "      <td>UK</td>\n",
              "      <td>DT @georgegalloway: RT @Galloway4Mayor: ÛÏThe...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>132</th>\n",
              "      <td>191</td>\n",
              "      <td>aftershock</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Aftershock back to school kick off was great. ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6845</th>\n",
              "      <td>9810</td>\n",
              "      <td>trauma</td>\n",
              "      <td>Montgomery County, MD</td>\n",
              "      <td>in response to trauma Children of Addicts deve...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        id  ... target\n",
              "2644  3796  ...      1\n",
              "2227  3185  ...      0\n",
              "5448  7769  ...      1\n",
              "132    191  ...      0\n",
              "6845  9810  ...      0\n",
              "\n",
              "[5 rows x 5 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lw4mKW1yL0kI"
      },
      "source": [
        "Notice how the training data has a `\"target\"` column.\n",
        "\n",
        "We're going to be writing code to find patterns (e.g. different combinations of words) in the `\"text\"` column of the training dataset to predict the value of the `\"target\"` column.\n",
        "\n",
        "The test dataset doesn't have a `\"target\"` column.\n",
        "\n",
        "```\n",
        "Inputs (text column) -> Machine Learning Algorithm -> Outputs (target column)\n",
        "```\n",
        "\n",
        "![](https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/images/08-text-classification-inputs-and-outputs.png)\n",
        "*Example text classification inputs and outputs for the problem of classifying whether a Tweet is about a diaster or not.*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tDh5t7thI5BM",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "9f4321e2-a45e-4eef-93d1-71972c992e37"
      },
      "source": [
        "# The test data doesn't have a target (that's what we'd try to predict)\n",
        "test_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Just happened a terrible car crash</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Heard about #earthquake is different cities, s...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>there is a forest fire at spot pond, geese are...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Apocalypse lighting. #Spokane #wildfires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>11</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Typhoon Soudelor kills 28 in China and Taiwan</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id keyword location                                               text\n",
              "0   0     NaN      NaN                 Just happened a terrible car crash\n",
              "1   2     NaN      NaN  Heard about #earthquake is different cities, s...\n",
              "2   3     NaN      NaN  there is a forest fire at spot pond, geese are...\n",
              "3   9     NaN      NaN           Apocalypse lighting. #Spokane #wildfires\n",
              "4  11     NaN      NaN      Typhoon Soudelor kills 28 in China and Taiwan"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O4JhBRn5Mn-V"
      },
      "source": [
        "Let's check how many examples of each target we have."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k4P5DnLhIciD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "50fa6a7a-a7cf-4dc5-b5a9-01ccb8dfbe61"
      },
      "source": [
        "# How many examples of each class?\n",
        "train_df.target.value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    4342\n",
              "1    3271\n",
              "Name: target, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WjEDQ297Ihy4"
      },
      "source": [
        "Since we have two target values, we're dealing with a **binary classification** problem.\n",
        "\n",
        "It's fairly balanced too, about 60% negative class (`target = 0`) and 40% positive class (`target = 1`).\n",
        "\n",
        "Where, \n",
        "\n",
        "* `1` = a real disaster Tweet\n",
        "* `0` = not a real disaster Tweet\n",
        "\n",
        "And what about the total number of samples we have?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jQxg7EKKIy5L",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ca4aefe4-3405-4a94-8e8a-163386b25709"
      },
      "source": [
        "# How many samples total?\n",
        "print(f\"Total training samples: {len(train_df)}\")\n",
        "print(f\"Total test samples: {len(test_df)}\")\n",
        "print(f\"Total samples: {len(train_df) + len(test_df)}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total training samples: 7613\n",
            "Total test samples: 3263\n",
            "Total samples: 10876\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q1upY8-xNPWV"
      },
      "source": [
        "Alright, seems like we've got a decent amount of training and test data. If anything, we've got an abundance of testing examples, usually a split of 90/10 (90% training, 10% testing) or 80/20 is suffice.\n",
        "\n",
        "Okay, time to visualize, let's write some code to visualize random text samples.\n",
        "\n",
        "> 🤔 **Question:** Why visualize random samples? You could visualize samples in order but this could lead to only seeing a certain subset of data. Better to visualize a substantial quantity (100+) of random samples to get an idea of the different kinds of data you're working with. In machine learning, never underestimate the power of randomness."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vH3EXknTI3bQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4e068f05-06fd-4611-b0c9-519f4e67d8b9"
      },
      "source": [
        "# Let's visualize some random training examples\n",
        "import random\n",
        "random_index = random.randint(0, len(train_df)-5) # create random indexes not higher than the total number of samples\n",
        "for row in train_df_shuffled[[\"text\", \"target\"]][random_index:random_index+5].itertuples():\n",
        "  _, text, target = row\n",
        "  print(f\"Target: {target}\", \"(real disaster)\" if target > 0 else \"(not real disaster)\")\n",
        "  print(f\"Text:\\n{text}\\n\")\n",
        "  print(\"---\\n\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Target: 0 (not real disaster)\n",
            "Text:\n",
            "https://t.co/eCMUjkKqX1 @ArianaGrande @ScreamQueens \n",
            "Katherine's Death\n",
            "\n",
            "---\n",
            "\n",
            "Target: 0 (not real disaster)\n",
            "Text:\n",
            "@TinyJecht Are you another Stand-user? If you are I will have to detonate you with my Killer Queen.\n",
            "\n",
            "---\n",
            "\n",
            "Target: 1 (real disaster)\n",
            "Text:\n",
            "70 Years After Atomic Bombs Japan Still Struggles With War Past http://t.co/5wfXbAQMBK The anniversary of the devastation wrought by theÛ_\n",
            "\n",
            "---\n",
            "\n",
            "Target: 0 (not real disaster)\n",
            "Text:\n",
            "My lifelong all-time favorite song is 'Landslide'.  This song has gotten me through a lot of though times &amp;... http://t.co/RfB3JXbiEJ\n",
            "\n",
            "---\n",
            "\n",
            "Target: 0 (not real disaster)\n",
            "Text:\n",
            "I hear the mumbling i hear the cackling i got em scared shook panicking\n",
            "\n",
            "---\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1FhRRewGPNS_"
      },
      "source": [
        "### Split data into training and validation sets\n",
        "\n",
        "Since the test set has no labels and we need a way to evalaute our trained models, we'll split off some of the training data and create a validation set.\n",
        "\n",
        "When our model trains (tries patterns in the Tweet samples), it'll only see data from the training set and we can see how it performs on unseen data using the validation set.\n",
        "\n",
        "We'll convert our splits from pandas Series datatypes to lists of strings (for the text) and lists of ints (for the labels) for ease of use later.\n",
        "\n",
        "To split our training dataset and create a validation dataset, we'll use Scikit-Learn's [`train_test_split()`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) method and dedicate 10% of the training samples to the validation set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7OJf31TQ-X8s"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "# Use train_test_split to split training data into training and validation sets\n",
        "train_sentences, val_sentences, train_labels, val_labels = train_test_split(train_df_shuffled[\"text\"].to_numpy(),\n",
        "                                                                            train_df_shuffled[\"target\"].to_numpy(),\n",
        "                                                                            test_size=0.1, # dedicate 10% of samples to validation set\n",
        "                                                                            random_state=42) # random state for reproducibility"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NWGOTjanBaTQ",
        "outputId": "dd1a42bc-d291-42ad-f370-6b4008b3ec45"
      },
      "source": [
        "# Check the lengths\n",
        "len(train_sentences), len(train_labels), len(val_sentences), len(val_labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(6851, 6851, 762, 762)"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VqhvQK9wBTbw",
        "outputId": "31c510a5-5b33-4c3c-b093-de3982bf145b"
      },
      "source": [
        "# View the first 10 training sentences and their labels\n",
        "train_sentences[:10], train_labels[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['@mogacola @zamtriossu i screamed after hitting tweet',\n",
              "        'Imagine getting flattened by Kurt Zouma',\n",
              "        '@Gurmeetramrahim #MSGDoing111WelfareWorks Green S welfare force ke appx 65000 members har time disaster victim ki help ke liye tyar hai....',\n",
              "        \"@shakjn @C7 @Magnums im shaking in fear he's gonna hack the planet\",\n",
              "        'Somehow find you and I collide http://t.co/Ee8RpOahPk',\n",
              "        '@EvaHanderek @MarleyKnysh great times until the bus driver held us hostage in the mall parking lot lmfao',\n",
              "        'destroy the free fandom honestly',\n",
              "        'Weapons stolen from National Guard Armory in New Albany still missing #Gunsense http://t.co/lKNU8902JE',\n",
              "        '@wfaaweather Pete when will the heat wave pass? Is it really going to be mid month? Frisco Boy Scouts have a canoe trip in Okla.',\n",
              "        'Patient-reported outcomes in long-term survivors of metastatic colorectal cancer - British Journal of Surgery http://t.co/5Yl4DC1Tqt'],\n",
              "       dtype=object), array([0, 0, 1, 0, 0, 1, 1, 0, 1, 1]))"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EN-houoSD-hP"
      },
      "source": [
        "## Converting text into numbers\n",
        "\n",
        "Wonderful! We've got a training set and a validation set containing Tweets and labels.\n",
        "\n",
        "Our labels are in numerical form (`0` and `1`) but our Tweets are in string form.\n",
        "\n",
        "> 🤔 **Question:** What do you think we have to do before we can use a machine learning algorithm with our text data? \n",
        "\n",
        "If you answered something along the lines of \"turn it into numbers\", you're correct. A machine learning algorithm requires its inputs to be in numerical form.\n",
        "\n",
        "In NLP, there are two main concepts for turning text into numbers:\n",
        "* **Tokenization** - A straight mapping from word or character or sub-word to a numerical value. There are three main levels of tokenization:\n",
        "  1. Using **word-level tokenization** with the sentence \"I love TensorFlow\" might result in \"I\" being `0`, \"love\" being `1` and \"TensorFlow\" being `2`. In this case, every word in a sequence considered a single **token**.\n",
        "  2. **Character-level tokenization**, such as converting the letters A-Z to values `1-26`. In this case, every character in a sequence considered a single **token**.\n",
        "  3. **Sub-word tokenization** is in between word-level and character-level tokenization. It involves breaking invidual words into smaller parts and then converting those smaller parts into numbers. For example, \"my favourite food is pineapple pizza\" might become \"my, fav, avour, rite, fo, oo, od, is, pin, ine, app, le, piz, za\". After doing this, these sub-words would then be mapped to a numerical value. In this case, every word could be considered multiple **tokens**.\n",
        "* **Embeddings** - An embedding is a representation of natural language which can be learned. Representation comes in the form of a **feature vector**. For example, the word \"dance\" could be represented by the 5-dimensional vector `[-0.8547, 0.4559, -0.3332, 0.9877, 0.1112]`. It's important to note here, the size of the feature vector is tuneable. There are two ways to use embeddings: \n",
        "  1. **Create your own embedding** - Once your text has been turned into numbers (required for an embedding), you can put them through an embedding layer (such as [`tf.keras.layers.Embedding`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding)) and an embedding representation will be learned during model training.\n",
        "  2. **Reuse a pre-learned embedding** - Many pre-trained embeddings exist online. These pre-trained embeddings have often been learned on large corpuses of text (such as all of Wikipedia) and thus have a good underlying representation of natural language. You can use a pre-trained embedding to initialize your model and fine-tune it to your own specific task.\n",
        "\n",
        "![](https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/images/08-tokenization-vs-embedding.png)\n",
        "*Example of **tokenization** (straight mapping from word to number) and **embedding** (richer representation of relationships between tokens).*\n",
        "\n",
        "> 🤔 **Question:** What level of tokenzation should I use? What embedding should should I choose?\n",
        "\n",
        "It depends on your problem. You could try character-level tokenization/embeddings and word-level tokenization/embeddings and see which perform best. You might even want to try stacking them (e.g. combining the outputs of your embedding layers using [`tf.keras.layers.concatenate`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/concatenate)). \n",
        "\n",
        "If you're looking for pre-trained word embeddings, [Word2vec embeddings](http://jalammar.github.io/illustrated-word2vec/), [GloVe embeddings](https://nlp.stanford.edu/projects/glove/) and many of the options available on [TensorFlow Hub](https://tfhub.dev/s?module-type=text-embedding) are great places to start.\n",
        "\n",
        "> 🔑 **Note:** Much like searching for a pre-trained computer vision model, you can search for pre-trained word embeddings to use for your problem. Try searching for something like \"use pre-trained word embeddings in TensorFlow\"."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8UnRcM1PELHn"
      },
      "source": [
        "### Text vectorization (tokenization)\n",
        "\n",
        "Enough talking about tokenization and embeddings, let's create some.\n",
        "\n",
        "We'll practice tokenzation (mapping our words to numbers) first.\n",
        "\n",
        "To tokenize our words, we'll use the helpful preprocessing layer [`tf.keras.layers.experimental.preprocessing.TextVectorization`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/TextVectorization).\n",
        "\n",
        "The `TextVectorization` layer takes the following parameters:\n",
        "* `max_tokens` - The maximum number of words in your vocabulary (e.g. 20000 or the number of unique words in your text), includes a value for OOV (out of vocabulary) tokens. \n",
        "* `standardize` - Method for standardizing text. Default is `\"lower_and_strip_punctuation\"` which lowers text and removes all punctuation marks.\n",
        "* `split` - How to split text, default is `\"whitespace\"` which splits on spaces.\n",
        "* `ngrams` - How many words to contain per token split, for example, `ngrams=2` splits tokens into continuous sequences of 2.\n",
        "* `output_mode` -  How to output tokens, can be `\"int\"` (integer mapping), `\"binary\"` (one-hot encoding), `\"count\"` or `\"tf-idf\"`. See documentation for more.\n",
        "* `output_sequence_length` - Length of tokenized sequence to output. For example, if `output_sequence_length=150`, all tokenized sequences will be 150 tokens long.\n",
        "* `pad_to_max_tokens` - Defaults to `False`, if `True`, the output feature axis will be padded to `max_tokens` even if the number of unique tokens in the vocabulary is less than `max_tokens`. Only valid in certain modes, see docs for more.\n",
        "\n",
        "Let's see it in action."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PVcZk-LcNunF"
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras.layers.experimental.preprocessing import TextVectorization\n",
        "# Note: in TensorFlow 2.6+, you no longer need \"layers.experimental.preprocessing\"\n",
        "# you can use: \"tf.keras.layers.TextVectorization\", see https://github.com/tensorflow/tensorflow/releases/tag/v2.6.0 for more\n",
        "\n",
        "# Use the default TextVectorization variables\n",
        "text_vectorizer = TextVectorization(max_tokens=None, # how many words in the vocabulary (all of the different words in your text)\n",
        "                                    standardize=\"lower_and_strip_punctuation\", # how to process text\n",
        "                                    split=\"whitespace\", # how to split tokens\n",
        "                                    ngrams=None, # create groups of n-words?\n",
        "                                    output_mode=\"int\", # how to map tokens to numbers\n",
        "                                    output_sequence_length=None) # how long should the output sequence of tokens be?\n",
        "                                    # pad_to_max_tokens=True) # Not valid if using max_tokens=None"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u0Ej5mzKGkK8"
      },
      "source": [
        "We've initialized a `TextVectorization` object with the default settings but let's customize it a little bit for our own use case.\n",
        "\n",
        "In particular, let's set values for `max_tokens` and `output_sequence_length`.\n",
        "\n",
        "For `max_tokens` (the number of words in the vocabulary), multiples of 10,000 (`10,000`, `20,000`, `30,000`) or the exact number of unique words in your text (e.g. `32,179`) are common values.\n",
        "\n",
        "For our use case, we'll use `10,000`.\n",
        "\n",
        "And for the `output_sequence_length` we'll use the average number of tokens per Tweet in the training set. But first, we'll need to find it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SQ3ZCINnR56H",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "093c7d74-8c3a-4fd1-8624-baa5cc95e183"
      },
      "source": [
        "# Find average number of tokens (words) in training Tweets\n",
        "round(sum([len(i.split()) for i in train_sentences])/len(train_sentences))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "15"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AFGTRcw8Hv7R"
      },
      "source": [
        "Now let's create another `TextVectorization` object using our custom parameters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eYPcGwdbafmW"
      },
      "source": [
        "# Setup text vectorization with custom variables\n",
        "max_vocab_length = 10000 # max number of words to have in our vocabulary\n",
        "max_length = 15 # max length our sequences will be (e.g. how many words from a Tweet does our model see?)\n",
        "\n",
        "text_vectorizer = TextVectorization(max_tokens=max_vocab_length,\n",
        "                                    output_mode=\"int\",\n",
        "                                    output_sequence_length=max_length)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BSWycfB3H3wV"
      },
      "source": [
        "Beautiful!\n",
        "\n",
        "To map our `TextVectorization` instance `text_vectorizer` to our data, we can call the `adapt()` method on it whilst passing it our training text."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0083KHXPO4m2"
      },
      "source": [
        "# Fit the text vectorizer to the training text\n",
        "text_vectorizer.adapt(train_sentences)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Syh0VB9wIHUq"
      },
      "source": [
        "Training data mapped! Let's try our `text_vectorizer` on a custom sentence (one similar to what you might see in the training data)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uizmdJKvO2OW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "908d8b3e-0110-47ba-b7d4-ab2e402af79c"
      },
      "source": [
        "# Create sample sentence and tokenize it\n",
        "sample_sentence = \"There's a flood in my street!\"\n",
        "text_vectorizer([sample_sentence])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(1, 15), dtype=int64, numpy=\n",
              "array([[264,   3, 232,   4,  13, 698,   0,   0,   0,   0,   0,   0,   0,\n",
              "          0,   0]])>"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M0RmAeplIW57"
      },
      "source": [
        "Wonderful, it seems we've got a way to turn our text into numbers (in this case, word-level tokenization). Notice the 0's at the end of the returned tensor, this is because we set `output_sequence_length=15`, meaning no matter the size of the sequence we pass to `text_vectorizer`, it always returns a sequence with a length of 15.\n",
        "\n",
        "How about we try our `text_vectorizer` on a few random sentences?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SZFka4BtRR6_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "227a570e-d987-49a1-bcf1-3f561d907d00"
      },
      "source": [
        "# Choose a random sentence from the training dataset and tokenize it\n",
        "random_sentence = random.choice(train_sentences)\n",
        "print(f\"Original text:\\n{random_sentence}\\\n",
        "      \\n\\nVectorized version:\")\n",
        "text_vectorizer([random_sentence])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Original text:\n",
            "Black Eye 9: A space battle occurred at Star O784 involving 3 fleets totaling 3942 ships with 14 destroyed      \n",
            "\n",
            "Vectorized version:\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(1, 15), dtype=int64, numpy=\n",
              "array([[ 159,  898,  491,    3,  759,  442, 1068,   17,  874, 1629, 1129,\n",
              "         118, 1524, 1457, 6327]])>"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PErGKRbPJF89"
      },
      "source": [
        "Looking good!\n",
        "\n",
        "Finally, we can check the unique tokens in our vocabulary using the `get_vocabulary()` method."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5nwNdgAZIhna",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f23e1a41-5b2d-4042-cfec-c6b22226a798"
      },
      "source": [
        "# Get the unique words in the vocabulary\n",
        "words_in_vocab = text_vectorizer.get_vocabulary()\n",
        "top_5_words = words_in_vocab[:5] # most common tokens (notice the [UNK] token for \"unknown\" words)\n",
        "bottom_5_words = words_in_vocab[-5:] # least common tokens\n",
        "print(f\"Number of words in vocab: {len(words_in_vocab)}\")\n",
        "print(f\"Top 5 most common words: {top_5_words}\") \n",
        "print(f\"Bottom 5 least common words: {bottom_5_words}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of words in vocab: 10000\n",
            "Top 5 most common words: ['', '[UNK]', 'the', 'a', 'in']\n",
            "Bottom 5 least common words: ['pages', 'paeds', 'pads', 'padres', 'paddytomlinson1']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AHyCdO0uEOkH"
      },
      "source": [
        "### Creating an Embedding using an Embedding Layer\n",
        "\n",
        "We've got a way to map our text to numbers. How about we go a step further and turn those numbers into an embedding?\n",
        "\n",
        "The powerful thing about an embedding is it can be learned during training. This means rather than just being static (e.g. `1` = I, `2` = love, `3` = TensorFlow), a word's numeric representation can be improved as a model goes through data samples.\n",
        "\n",
        "We can see what an embedding of a word looks like by using the [`tf.keras.layers.Embedding`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding) layer. \n",
        "\n",
        "The main parameters we're concerned about here are:\n",
        "* `input_dim` - The size of the vocabulary (e.g. `len(text_vectorizer.get_vocabulary()`).\n",
        "* `output_dim` - The size of the output embedding vector, for example, a value of `100` outputs a  feature vector of size 100 for each word.\n",
        "* `embeddings_initializer` - How to initialize the embeddings matrix, default is `\"uniform\"` which randomly initalizes embedding matrix with uniform distribution. This can be changed for using pre-learned embeddings.\n",
        "* `input_length` - Length of sequences being passed to embedding layer.\n",
        "\n",
        "Knowing these, let's make an embedding layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OsB4StymSk_s",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cffa28e2-f368-4f82-ff97-aab8b54241d3"
      },
      "source": [
        "tf.random.set_seed(42)\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "embedding = layers.Embedding(input_dim=max_vocab_length, # set input shape\n",
        "                             output_dim=128, # set size of embedding vector\n",
        "                             embeddings_initializer=\"uniform\", # default, intialize randomly\n",
        "                             input_length=max_length, # how long is each input\n",
        "                             name=\"embedding_1\") \n",
        "\n",
        "embedding"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.layers.embeddings.Embedding at 0x7f5eb57cee50>"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bfML_IzlSUho"
      },
      "source": [
        "Excellent, notice how `embedding` is a TensoFlow layer? This is important because we can use it as part of a model, meaning its parameters (word representations) can be updated and improved as the model learns.\n",
        "\n",
        "How about we try it out on a sample sentence?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1Re6Eew6SZnG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "83f28dec-93fb-483a-a997-a4552ef74426"
      },
      "source": [
        "# Get a random sentence from training set\n",
        "random_sentence = random.choice(train_sentences)\n",
        "print(f\"Original text:\\n{random_sentence}\\\n",
        "      \\n\\nEmbedded version:\")\n",
        "\n",
        "# Embed the random sentence (turn it into numerical representation)\n",
        "sample_embed = embedding(text_vectorizer([random_sentence]))\n",
        "sample_embed"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Original text:\n",
            "UNR issues Severe Thunderstorm Warning [wind: 60 MPH hail: 0.75 IN] for Weston [WY] and Custer Fall River Lawrence Meade Pennington [SÛ_      \n",
            "\n",
            "Embedded version:\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(1, 15, 128), dtype=float32, numpy=\n",
              "array([[[-0.0235487 , -0.02911143, -0.01600165, ..., -0.03135703,\n",
              "         -0.03661771,  0.02812702],\n",
              "        [ 0.03848192, -0.04844777,  0.0345685 , ..., -0.01402212,\n",
              "         -0.04769395, -0.0404261 ],\n",
              "        [-0.04240407,  0.04235772,  0.04278237, ..., -0.01318695,\n",
              "          0.04171768,  0.01777187],\n",
              "        ...,\n",
              "        [-0.00522641,  0.04871375, -0.03742788, ...,  0.00540795,\n",
              "         -0.04380312, -0.01817607],\n",
              "        [ 0.03406706, -0.00160446,  0.00894339, ..., -0.0356751 ,\n",
              "          0.00541915,  0.00282475],\n",
              "        [ 0.02248487, -0.02848336,  0.04786098, ...,  0.03069806,\n",
              "         -0.04317403, -0.04145076]]], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e4Sn8o9pTBE5"
      },
      "source": [
        "Each token in the sentence gets turned into a length 128 feature vector."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g_VBepuSTBDW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7e62e9cf-bf1c-4e21-a304-ba0b277bb220"
      },
      "source": [
        "# Check out a single token's embedding\n",
        "sample_embed[0][0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(128,), dtype=float32, numpy=\n",
              "array([-2.3548698e-02, -2.9111434e-02, -1.6001653e-02, -2.4854636e-02,\n",
              "       -1.8839194e-02, -1.7742872e-02,  3.5991777e-02,  4.7734752e-03,\n",
              "       -3.1464353e-02, -1.0192860e-02,  3.9272513e-02,  1.6213503e-02,\n",
              "        3.5752203e-02, -7.9760700e-04, -4.6503343e-02,  4.1901264e-02,\n",
              "        2.7158771e-02, -2.9694129e-02,  6.3859299e-04, -3.6073186e-02,\n",
              "        3.7186686e-02,  8.1444494e-03, -3.4610189e-02, -1.2373447e-02,\n",
              "        3.3506799e-02,  3.4542195e-03, -3.4555770e-02,  3.0121803e-03,\n",
              "        1.2546945e-02,  1.8180419e-02, -2.8727353e-02,  3.0131452e-03,\n",
              "        2.2011306e-02,  1.5216086e-02,  8.3960593e-05, -4.9976040e-02,\n",
              "       -4.1987814e-02, -1.4751814e-02,  3.1978119e-02,  3.0810181e-02,\n",
              "        1.3748173e-02,  1.3646554e-02, -1.8768311e-03,  5.6033619e-03,\n",
              "       -3.2450367e-02, -3.2819200e-02,  6.4723380e-03,  2.4402250e-02,\n",
              "       -4.9929023e-02,  8.7605603e-03,  3.7449453e-02, -3.0369056e-02,\n",
              "        2.8607275e-02, -8.9427829e-03, -2.6780851e-03,  1.9382443e-02,\n",
              "       -4.4139970e-02, -4.8123684e-02,  3.2326613e-02,  1.0355391e-02,\n",
              "       -6.2159896e-03,  3.3066813e-02,  4.1976977e-02,  9.8001361e-03,\n",
              "        9.7909793e-03,  1.8213544e-02,  1.6274918e-02, -1.7997943e-02,\n",
              "        1.4698040e-02,  1.0068141e-02, -2.3385560e-02,  1.7339502e-02,\n",
              "        3.5935570e-02, -4.9711645e-02,  3.2845590e-02,  3.8101044e-02,\n",
              "        3.9486382e-02, -3.1647660e-02, -4.8475552e-02,  4.4873584e-02,\n",
              "        2.7549271e-02, -4.1145109e-02, -3.3895336e-02, -3.6730655e-03,\n",
              "        4.9198270e-03,  9.6562132e-03, -2.2904599e-02, -1.3657093e-02,\n",
              "        1.5388299e-02,  8.1878789e-03,  1.8028166e-02,  3.1150069e-02,\n",
              "        4.7483686e-02, -3.7815310e-02, -4.5389161e-03,  4.1796099e-02,\n",
              "        4.3265197e-02,  3.1167094e-02, -4.9614847e-02, -5.8911927e-03,\n",
              "        4.3997217e-02, -2.2734845e-02, -4.1017674e-02,  1.7939974e-02,\n",
              "        2.3607183e-02,  1.5478458e-02,  7.7072531e-04, -4.3312550e-02,\n",
              "       -4.2333078e-02, -2.2680223e-02,  3.2546792e-02, -4.9846746e-02,\n",
              "        1.3042022e-02, -3.2268692e-02, -1.8501390e-02,  5.7965517e-03,\n",
              "       -6.9886930e-03, -1.9324971e-02, -4.5883238e-02,  3.7569497e-02,\n",
              "        1.4392149e-02, -1.0649189e-03,  2.4147406e-03, -2.7852738e-02,\n",
              "       -3.7008919e-02, -3.1357028e-02, -3.6617707e-02,  2.8127018e-02],\n",
              "      dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z0NTsDklR0xw"
      },
      "source": [
        "These values might not mean much to us but they're what our computer sees each word as. When our model looks for patterns in different samples, these values will be updated as necessary.\n",
        "\n",
        "> 🔑 **Note:** The previous two concepts (tokenization and embeddings) are the foundation for many NLP tasks. So if you're not sure about anything, be sure to research and conduct your own experiments to further help your understanding."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZJENUdF3F7Rn"
      },
      "source": [
        "## Modelling a text dataset\n",
        "\n",
        "![](https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/images/08-inputs-and-outputs-with-shapes-and-models-were-going-to-build.png)\n",
        "*Once you've got your inputs and outputs prepared, it's a matter of figuring out which machine learning model to build in between them to bridge the gap.*\n",
        "\n",
        "Now that we've got a way to turn our text data into numbers, we can start to build machine learning models to model it.\n",
        "\n",
        "To get plenty of practice, we're going to build a series of different models, each as its own experiment. We'll then compare the results of each model and see which one performed best.\n",
        "\n",
        "More specifically, we'll be building the following:\n",
        "* **Model 0**: Naive Bayes (baseline)\n",
        "* **Model 1**: Feed-forward neural network (dense model)\n",
        "* **Model 2**: LSTM model\n",
        "* **Model 3**: GRU model\n",
        "* **Model 4**: Bidirectional-LSTM model\n",
        "* **Model 5**: 1D Convolutional Neural Network\n",
        "* **Model 6**: TensorFlow Hub Pretrained Feature Extractor\n",
        "* **Model 7**: Same as model 6 with 10% of training data\n",
        "\n",
        "Model 0 is the simplest to acquire a baseline which we'll expect each other of the other deeper models to beat.\n",
        "\n",
        "Each experiment will go through the following steps:\n",
        "* Construct the model\n",
        "* Train the model\n",
        "* Make predictions with the model\n",
        "* Track prediction evaluation metrics for later comparison\n",
        "\n",
        "Let's get started."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q4i5BiQfF--y"
      },
      "source": [
        "### Model 0: Getting a baseline\n",
        "\n",
        "As with all machine learning modelling experiments, it's important to create a baseline model so you've got a benchmark for future experiments to build upon.\n",
        "\n",
        "To create our baseline, we'll create a Scikit-Learn Pipeline using the TF-IDF (term frequency-inverse document frequency) formula to convert our words to numbers and then model them with the [Multinomial Naive Bayes algorithm](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html#sklearn.naive_bayes.MultinomialNB). This was chosen via referring to the [Scikit-Learn machine learning map](https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html).\n",
        "\n",
        "> 📖 **Reading:** The ins and outs of TF-IDF algorithm is beyond the scope of this notebook, however, the curious reader is encouraged to check out the [Scikit-Learn documentation for more](https://scikit-learn.org/stable/modules/feature_extraction.html#tfidf-term-weighting)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xFqjqWcXtOOs",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0cec96b9-d78a-49af-9197-86a03b8043c8"
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "from sklearn.pipeline import Pipeline\n",
        "\n",
        "# Create tokenization and modelling pipeline\n",
        "model_0 = Pipeline([\n",
        "                    (\"tfidf\", TfidfVectorizer()), # convert words to numbers using tfidf\n",
        "                    (\"clf\", MultinomialNB()) # model the text\n",
        "])\n",
        "\n",
        "# Fit the pipeline to the training data\n",
        "model_0.fit(train_sentences, train_labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Pipeline(memory=None,\n",
              "         steps=[('tfidf',\n",
              "                 TfidfVectorizer(analyzer='word', binary=False,\n",
              "                                 decode_error='strict',\n",
              "                                 dtype=<class 'numpy.float64'>,\n",
              "                                 encoding='utf-8', input='content',\n",
              "                                 lowercase=True, max_df=1.0, max_features=None,\n",
              "                                 min_df=1, ngram_range=(1, 1), norm='l2',\n",
              "                                 preprocessor=None, smooth_idf=True,\n",
              "                                 stop_words=None, strip_accents=None,\n",
              "                                 sublinear_tf=False,\n",
              "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
              "                                 tokenizer=None, use_idf=True,\n",
              "                                 vocabulary=None)),\n",
              "                ('clf',\n",
              "                 MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True))],\n",
              "         verbose=False)"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ybOvOuVJbNjg"
      },
      "source": [
        "The benefit of using a shallow model like Multinomial Naive Bayes is that training is very fast.\n",
        "\n",
        "Let's evaluate our model and find our baseline metric."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "soPfnpmQuUIP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b947adf2-aaaf-4d9d-afc0-81359138cc5a"
      },
      "source": [
        "baseline_score = model_0.score(val_sentences, val_labels)\n",
        "print(f\"Our baseline model achieves an accuracy of: {baseline_score*100:.2f}%\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Our baseline model achieves an accuracy of: 79.27%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hUv5dyuibf3M"
      },
      "source": [
        "How about we make some predictions with our baseline model?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7n89JxrJufcf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "69a3f67c-1da2-434e-fd29-73d723cd181e"
      },
      "source": [
        "# Make predictions\n",
        "baseline_preds = model_0.predict(val_sentences)\n",
        "baseline_preds[:20]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1])"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K354svk_bmdf"
      },
      "source": [
        "### Creating an evaluation function for our model experiments\n",
        "\n",
        "We could evaluate these as they are but since we're going to be evaluating several models in the same way going forward, let's create a helper function which takes an array of predictions and ground truth labels and computes the following:\n",
        "* Accuracy\n",
        "* Precision\n",
        "* Recall\n",
        "* F1-score\n",
        "\n",
        "> 🔑 **Note:** Since we're dealing with a classification problem, the above metrics are the most appropriate. If we were working with a regression problem, other metrics such as MAE (mean absolute error) would be a better choice."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gLmNlDjIxGgJ"
      },
      "source": [
        "# Function to evaluate: accuracy, precision, recall, f1-score\n",
        "from sklearn.metrics import accuracy_score, precision_recall_fscore_support\n",
        "\n",
        "def calculate_results(y_true, y_pred):\n",
        "  \"\"\"\n",
        "  Calculates model accuracy, precision, recall and f1 score of a binary classification model.\n",
        "\n",
        "  Args:\n",
        "  -----\n",
        "  y_true = true labels in the form of a 1D array\n",
        "  y_pred = predicted labels in the form of a 1D array\n",
        "\n",
        "  Returns a dictionary of accuracy, precision, recall, f1-score.\n",
        "  \"\"\"\n",
        "  # Calculate model accuracy\n",
        "  model_accuracy = accuracy_score(y_true, y_pred) * 100\n",
        "  # Calculate model precision, recall and f1 score using \"weighted\" average\n",
        "  model_precision, model_recall, model_f1, _ = precision_recall_fscore_support(y_true, y_pred, average=\"weighted\")\n",
        "  model_results = {\"accuracy\": model_accuracy,\n",
        "                  \"precision\": model_precision,\n",
        "                  \"recall\": model_recall,\n",
        "                  \"f1\": model_f1}\n",
        "  return model_results"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sgy1omMhwr52",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ff1ea5fd-b76d-447a-e1f9-af4b46e8aaba"
      },
      "source": [
        "# Get baseline results\n",
        "baseline_results = calculate_results(y_true=val_labels,\n",
        "                                     y_pred=baseline_preds)\n",
        "baseline_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 79.26509186351706,\n",
              " 'f1': 0.7862189758049549,\n",
              " 'precision': 0.8111390004213173,\n",
              " 'recall': 0.7926509186351706}"
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "noRJNm7dGNyh"
      },
      "source": [
        "### Model 1: A simple dense model\n",
        "\n",
        "The first \"deep\" model we're going to build is a single layer dense model. In fact, it's barely going to have a single layer. \n",
        "\n",
        "It'll take our text and labels as input, tokenize the text, create an embedding, find the average of the embedding (using Global Average Pooling) and then pass the average through a fully connected layer with one output unit and a sigmoid activation function.\n",
        "\n",
        "If the previous sentence sounds like a mouthful, it'll make sense when we code it out (remember, if in doubt, code it out).\n",
        "\n",
        "And since we're going to be building a number of TensorFlow deep learning models, we'll import our `create_tensorboard_callback()` function from `helper_functions.py` to keep track of the results of each. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PVMPUd3HTit5"
      },
      "source": [
        "# Create tensorboard callback (need to create a new one for each model)\n",
        "from helper_functions import create_tensorboard_callback\n",
        "\n",
        "# Create directory to save TensorBoard logs\n",
        "SAVE_DIR = \"model_logs\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pib8hHtu7vt1"
      },
      "source": [
        "Now we've got a TensorBoard callback function ready to go, let's build our first deep model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a_rVtJA7yVBI"
      },
      "source": [
        "# Build model with the Functional API\n",
        "from tensorflow.keras import layers\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\") # inputs are 1-dimensional strings\n",
        "x = text_vectorizer(inputs) # turn the input text into numbers\n",
        "x = embedding(x) # create an embedding of the numerized numbers\n",
        "x = layers.GlobalAveragePooling1D()(x) # lower the dimensionality of the embedding (try running the model without this layer and see what happens)\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x) # create the output layer, want binary outputs so use sigmoid activation\n",
        "model_1 = tf.keras.Model(inputs, outputs, name=\"model_1_dense\") # construct the model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JYzsu36Y8JUe"
      },
      "source": [
        "Looking good. Our model takes a 1-dimensional string as input (in our case, a Tweet), it then tokenizes the string using `text_vectorizer` and creates an embedding using `embedding`.\n",
        "\n",
        "We then (optionally) pool the outputs of the embedding layer to reduce the dimensionality of the tensor we pass to the output layer.\n",
        "\n",
        "> 🛠 **Exercise:** Try building `model_1` with and without a `GlobalAveragePooling1D()` layer after the `embedding` layer. What happens? Why do you think this is?\n",
        "\n",
        "Finally, we pass the output of the pooling layer to a dense layer with sigmoid activation (we use sigmoid since our problem is binary classification).\n",
        "\n",
        "Before we can fit our model to the data, we've got to compile it. Since we're working with binary classification, we'll use `\"binary_crossentropy\"` as our loss function and the Adam optimizer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ubq0ctLD8CQq"
      },
      "source": [
        "# Compile model\n",
        "model_1.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "crgltz1O9uku"
      },
      "source": [
        "Model compiled. Let's get a summary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QkJa-t8aTw1H",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a0c4ada2-1984-41fa-9653-dbfc6bfdfd42"
      },
      "source": [
        "# Get a summary of the model\n",
        "model_1.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_1_dense\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_1 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization_1 (TextVe (None, 15)                0         \n",
            "_________________________________________________________________\n",
            "embedding_1 (Embedding)      (None, 15, 128)           1280000   \n",
            "_________________________________________________________________\n",
            "global_average_pooling1d (Gl (None, 128)               0         \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 1)                 129       \n",
            "=================================================================\n",
            "Total params: 1,280,129\n",
            "Trainable params: 1,280,129\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bH0JLyR09yYt"
      },
      "source": [
        "Most of the trainable parameters are contained within the embedding layer. Recall we created an embedding of size 128 (`output_dim=128`) for a vocabulary of size 10,000 (`input_dim=10000`), hence the 1,280,000 trainable parameters.\n",
        "\n",
        "Alright, our model is compiled, let's fit it to our training data for 5 epochs. We'll also pass our TensorBoard callback function to make sure our model's training metrics are logged."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1YRYpJIfTvHV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cc6ad75c-338c-4e79-d6a0-a1a2bf588652"
      },
      "source": [
        "# Fit the model\n",
        "model_1_history = model_1.fit(train_sentences, # input sentences can be a list of strings due to text preprocessing layer built-in model\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(dir_name=SAVE_DIR, \n",
        "                                                                     experiment_name=\"simple_dense_model\")])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/simple_dense_model/20210923-052559\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 5s 17ms/step - loss: 0.6094 - accuracy: 0.6916 - val_loss: 0.5357 - val_accuracy: 0.7572\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 3s 13ms/step - loss: 0.4410 - accuracy: 0.8189 - val_loss: 0.4691 - val_accuracy: 0.7848\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 3s 13ms/step - loss: 0.3463 - accuracy: 0.8605 - val_loss: 0.4590 - val_accuracy: 0.7900\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 3s 14ms/step - loss: 0.2848 - accuracy: 0.8923 - val_loss: 0.4641 - val_accuracy: 0.7927\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 3s 14ms/step - loss: 0.2380 - accuracy: 0.9118 - val_loss: 0.4767 - val_accuracy: 0.7874\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kZR5_j9C_LW-"
      },
      "source": [
        "Nice! Since we're using such a simple model, each epoch processes very quickly.\n",
        "\n",
        "Let's check our model's performance on the validation set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zSTS87YGzuBG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3dab9b26-7bd0-42c8-8ad8-c55ae73d0e09"
      },
      "source": [
        "# Check the results\n",
        "model_1.evaluate(val_sentences, val_labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 0s 7ms/step - loss: 0.4767 - accuracy: 0.7874\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.4766846001148224, 0.787401556968689]"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5M2CTAetBVfW",
        "outputId": "263af483-3739-4ff7-fa21-52b9eab7a81b"
      },
      "source": [
        "embedding.weights"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<tf.Variable 'embedding_1/embeddings:0' shape=(10000, 128) dtype=float32, numpy=\n",
              " array([[ 0.00073166,  0.01504797, -0.03425457, ..., -0.04403538,\n",
              "         -0.01042282,  0.01876436],\n",
              "        [ 0.04135862, -0.03945084, -0.03811942, ...,  0.00464737,\n",
              "          0.03163553,  0.029283  ],\n",
              "        [ 0.00684031,  0.05363134, -0.00241555, ..., -0.07082176,\n",
              "         -0.04750705,  0.01448254],\n",
              "        ...,\n",
              "        [-0.03301444, -0.0052493 , -0.04209725, ...,  0.02028764,\n",
              "          0.00308807,  0.02215792],\n",
              "        [ 0.00692343,  0.05942352, -0.01975194, ..., -0.06199061,\n",
              "         -0.01018393,  0.03510419],\n",
              "        [-0.0372346 ,  0.06267187, -0.07451146, ..., -0.02367217,\n",
              "         -0.0864333 ,  0.01742156]], dtype=float32)>]"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "M3rfhJFSBrga",
        "outputId": "e16f8a95-540b-40a7-98f4-8fe1a9dff5ac"
      },
      "source": [
        "embed_weights = model_1.get_layer(\"embedding_1\").get_weights()[0]\n",
        "print(embed_weights.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(10000, 128)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I9dg2aba_VxK"
      },
      "source": [
        "And since we tracked our model's training logs with TensorBoard, how about we visualize them?\n",
        "\n",
        "We can do so by uploading our TensorBoard log files (contained in the `model_logs` directory) to [TensorBoard.dev](https://tensorboard.dev/).\n",
        "\n",
        "> 🔑 **Note:** Remember, whatever you upload to TensorBoard.dev becomes public. If there are training logs you don't want to share, don't upload them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t6UrSgRVU6pl"
      },
      "source": [
        "# # View tensorboard logs of transfer learning modelling experiments (should be 4 models)\n",
        "# # Upload TensorBoard dev records\n",
        "# !tensorboard dev upload --logdir ./model_logs \\\n",
        "#   --name \"First deep model on text data\" \\\n",
        "#   --description \"Trying a dense model with an embedding layer\" \\\n",
        "#   --one_shot # exits the uploader when upload has finished"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DVyJl-VE1ACz"
      },
      "source": [
        "# If you need to remove previous experiments, you can do so using the following command\n",
        "# !tensorboard dev delete --experiment_id EXPERIMENT_ID_TO_DELETE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PkinGcjQ_yI9"
      },
      "source": [
        "The TensorBoard.dev experiment for our first deep model can be viewed here: https://tensorboard.dev/experiment/5d1Xm10aT6m6MgyW3HAGfw/\n",
        "\n",
        "![](https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/images/08-tensorboard-dense-model-training-curves.png)\n",
        "\n",
        "*What the training curves of our model look like on TensorBoard. From looking at the curves can you tell if the model is overfitting or underfitting?*\n",
        "\n",
        "Beautiful! Those are some colorful training curves. Would you say the model is overfitting or underfitting?\n",
        "\n",
        "We've built and trained our first deep model, the next step is to make some predictions with it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5X7kbEmAzzxM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5d3a68f3-5b96-4dd4-e8e0-bf351ff68c9f"
      },
      "source": [
        "# Make predictions (these come back in the form of probabilities)\n",
        "model_1_pred_probs = model_1.predict(val_sentences)\n",
        "model_1_pred_probs[:10] # only print out the first 10 prediction probabilities"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.4048821 ],\n",
              "       [0.7443312 ],\n",
              "       [0.997895  ],\n",
              "       [0.10889997],\n",
              "       [0.11143532],\n",
              "       [0.93556094],\n",
              "       [0.9134595 ],\n",
              "       [0.9925345 ],\n",
              "       [0.97156817],\n",
              "       [0.26570338]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YWU5e1NLAKJ9"
      },
      "source": [
        "Since our final layer uses a sigmoid activation function, we get our predictions back in the form of probabilities.\n",
        "\n",
        "To convert them to prediction classes, we'll use `tf.round()`, meaning prediction probabilities below 0.5 will be rounded to 0 and those above 0.5 will be rounded to 1.\n",
        "\n",
        "> 🔑 **Note:** In practice, the output threshold of a sigmoid prediction probability doesn't necessarily have to 0.5. For example, through testing, you may find that a cut off of 0.25 is better for your chosen evaluation metrics. A common example of this threshold cutoff is the [precision-recall tradeoff](https://www.machinelearningaptitude.com/topics/machine-learning/what-is-precision-recall-tradeoff/#:~:text=precision%2Drecall%20tradeoff%20occur%20due,the%20threshold%20of%20the%20classifier.&text=When%20threshold%20is%20decreased%20to,but%20precision%20decreases%20to%200.4.)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qf-R_1vsz47P",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3062f826-9564-48d7-bbec-a803b41e9cba"
      },
      "source": [
        "# Turn prediction probabilities into single-dimension tensor of floats\n",
        "model_1_preds = tf.squeeze(tf.round(model_1_pred_probs)) # squeeze removes single dimensions\n",
        "model_1_preds[:20]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(20,), dtype=float32, numpy=\n",
              "array([0., 1., 1., 0., 0., 1., 1., 1., 1., 0., 0., 1., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 1.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zc3ryY0yCHcI"
      },
      "source": [
        "Now we've got our model's predictions in the form of classes, we can use our `calculate_results()` function to compare them to the ground truth validation labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iDEEhYTF0X1y",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e31e8bca-894a-44ad-88cc-fdea0dbc453a"
      },
      "source": [
        "# Calculate model_1 metrics\n",
        "model_1_results = calculate_results(y_true=val_labels, \n",
        "                                    y_pred=model_1_preds)\n",
        "model_1_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 78.74015748031496,\n",
              " 'f1': 0.7846966492209201,\n",
              " 'precision': 0.7914920592553047,\n",
              " 'recall': 0.7874015748031497}"
            ]
          },
          "metadata": {},
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gnkK6Uc7CYlX"
      },
      "source": [
        "How about we compare our first deep model to our baseline model?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jp88ystW1m0d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ba27df1c-624b-4a24-da50-012a56653038"
      },
      "source": [
        "# Is our simple Keras model better than our baseline model?\n",
        "import numpy as np\n",
        "np.array(list(model_1_results.values())) > np.array(list(baseline_results.values()))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([False, False, False, False])"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lUINrCdRCpFf"
      },
      "source": [
        "Since we'll be doing this kind of comparison (baseline compared to new model) quite a few times, let's create a function to help us out. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wo3norTG3GrE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "14b74922-fe0a-44cb-8932-327e05402960"
      },
      "source": [
        "# Create a helper function to compare our baseline results to new model results\n",
        "def compare_baseline_to_new_results(baseline_results, new_model_results):\n",
        "  for key, value in baseline_results.items():\n",
        "    print(f\"Baseline {key}: {value:.2f}, New {key}: {new_model_results[key]:.2f}, Difference: {new_model_results[key]-value:.2f}\")\n",
        "\n",
        "compare_baseline_to_new_results(baseline_results=baseline_results, \n",
        "                                new_model_results=model_1_results)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Baseline accuracy: 79.27, New accuracy: 78.74, Difference: -0.52\n",
            "Baseline precision: 0.81, New precision: 0.79, Difference: -0.02\n",
            "Baseline recall: 0.79, New recall: 0.79, Difference: -0.01\n",
            "Baseline f1: 0.79, New f1: 0.78, Difference: -0.00\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6e-1LuioSLAM"
      },
      "source": [
        "## Visualizing learned embeddings\n",
        "\n",
        "Our first model (`model_1`) contained an embedding layer (`embedding`) which learned a way of representing words as feature vectors by passing over the training data.\n",
        "\n",
        "Hearing this for the first few times may sound confusing.\n",
        "\n",
        "So to further help understand what a text embedding is, let's visualize the embedding our model learned.\n",
        "\n",
        "To do so, let's remind ourselves of the words in our vocabulary.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-DkcfRQBVXuJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9e10f4fd-50ef-4ccd-a99b-eff8a64082f8"
      },
      "source": [
        "# Get the vocabulary from the text vectorization layer\n",
        "words_in_vocab = text_vectorizer.get_vocabulary()\n",
        "len(words_in_vocab), words_in_vocab[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(10000, ['', '[UNK]', 'the', 'a', 'in', 'to', 'of', 'and', 'i', 'is'])"
            ]
          },
          "metadata": {},
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KzmAPJXQEx6r"
      },
      "source": [
        "And now let's get our embedding layer's weights (these are the numerical representations of each word)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8EUR9PwrZphh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "eb047f23-dbef-46ea-ce98-2ea91b309d09"
      },
      "source": [
        "model_1.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_1_dense\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_1 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization_1 (TextVe (None, 15)                0         \n",
            "_________________________________________________________________\n",
            "embedding_1 (Embedding)      (None, 15, 128)           1280000   \n",
            "_________________________________________________________________\n",
            "global_average_pooling1d (Gl (None, 128)               0         \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 1)                 129       \n",
            "=================================================================\n",
            "Total params: 1,280,129\n",
            "Trainable params: 1,280,129\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9xJ5LrInWDLo",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ffc27236-56f5-4ecf-e107-37a51722194a"
      },
      "source": [
        "# Get the weight matrix of embedding layer \n",
        "# (these are the numerical patterns between the text in the training dataset the model has learned)\n",
        "embed_weights = model_1.get_layer(\"embedding_1\").get_weights()[0]\n",
        "print(embed_weights.shape) # same size as vocab size and embedding_dim (each word is a embedding_dim size vector)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(10000, 128)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jzOJhJHPW1ju"
      },
      "source": [
        "Now we've got these two objects, we can use the [Embedding Projector tool](http://projector.tensorflow.org/_) to visualize our embedding. \n",
        "\n",
        "To use the Embedding Projector tool, we need two files:\n",
        "* The embedding vectors (same as embedding weights).\n",
        "* The meta data of the embedding vectors (the words they represent - our vocabulary).\n",
        "\n",
        "Right now, we've got of these files as Python objects. To download them to file, we're going to [use the code example available on the TensorFlow word embeddings tutorial page](https://www.tensorflow.org/tutorials/text/word_embeddings#retrieve_the_trained_word_embeddings_and_save_them_to_disk).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4e9rfcK6WxQE"
      },
      "source": [
        "# # Code below is adapted from: https://www.tensorflow.org/tutorials/text/word_embeddings#retrieve_the_trained_word_embeddings_and_save_them_to_disk\n",
        "# import io\n",
        "\n",
        "# # Create output writers\n",
        "# out_v = io.open(\"embedding_vectors.tsv\", \"w\", encoding=\"utf-8\")\n",
        "# out_m = io.open(\"embedding_metadata.tsv\", \"w\", encoding=\"utf-8\")\n",
        "\n",
        "# # Write embedding vectors and words to file\n",
        "# for num, word in enumerate(words_in_vocab):\n",
        "#   if num == 0: \n",
        "#      continue # skip padding token\n",
        "#   vec = embed_weights[num]\n",
        "#   out_m.write(word + \"\\n\") # write words to file\n",
        "#   out_v.write(\"\\t\".join([str(x) for x in vec]) + \"\\n\") # write corresponding word vector to file\n",
        "# out_v.close()\n",
        "# out_m.close()\n",
        "\n",
        "# # Download files locally to upload to Embedding Projector\n",
        "# try:\n",
        "#   from google.colab import files\n",
        "# except ImportError:\n",
        "#   pass\n",
        "# else:\n",
        "#   files.download(\"embedding_vectors.tsv\")\n",
        "#   files.download(\"embedding_metadata.tsv\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BVM7ifzpZaxJ"
      },
      "source": [
        "Once you've downloaded the embedding vectors and metadata, you can visualize them using Embedding Vector tool:\n",
        "1. Go to  http://projector.tensorflow.org/\n",
        "2. Click on \"Load data\"\n",
        "3. Upload the two files you downloaded (`embedding_vectors.tsv` and `embedding_metadata.tsv`)\n",
        "4. Explore\n",
        "5. Optional: You can share the data you've created by clicking \"Publish\"\n",
        "\n",
        "What do you find?\n",
        "\n",
        "Are words with similar meanings close together?\n",
        "\n",
        "Remember, they might not be. The embeddings we downloaded are how our model interprets words, not necessarily how we interpret them. \n",
        "\n",
        "Also, since the embedding has been learned purely from Tweets, it may contain some strange values as Tweets are a very unique style of natural language.\n",
        "\n",
        "> 🤔 **Question:** Do you have to visualize embeddings every time?\n",
        "\n",
        "No. Although helpful for gaining an intuition of what natural language embeddings are, it's not completely necessary. Especially as the dimensions of your vocabulary and embeddings grow, trying to comprehend them would become an increasingly difficult task."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AcRdDiEtGQj4"
      },
      "source": [
        "## Recurrent Neural Networks (RNN's)\n",
        "\n",
        "For our next series of modelling experiments we're going to be using a special kind of neural network called a **Recurrent Neural Network (RNN)**.\n",
        "\n",
        "The premise of an RNN is simple: use information from the past to help you with the future (this is where the term recurrent comes from). In other words, take an input (`X`) and compute an output (`y`) based on all previous inputs.\n",
        "\n",
        "This concept is especially helpful when dealing with sequences such as passages of natural language text (such as our Tweets).\n",
        "\n",
        "For example, when you read this sentence, you take into context the previous words when deciphering the meaning of the current word dog. \n",
        "\n",
        "See what happened there? \n",
        "\n",
        "I put the word \"dog\" at the end which is a valid word but it doesn't make sense in the context of the rest of the sentence.\n",
        "\n",
        "When an RNN looks at a sequence of text (already in numerical form), the patterns it learns are continually updated based on the order of the sequence. \n",
        "\n",
        "For a simple example, take two sentences:\n",
        "1. Massive earthquake last week, no?\n",
        "2. No massive earthquake last week.\n",
        "\n",
        "Both contain exactly the same words but have different meaning. The order of the words determines the meaning (one could argue punctuation marks also dictate the meaning but for simplicity sake, let's stay focused on the words).\n",
        "\n",
        "Recurrent neural networks can be used for a number of sequence-based problems:\n",
        "* **One to one:** one input, one output, such as image classification.\n",
        "* **One to many:** one input, many outputs, such as image captioning (image input, a sequence of text as caption output).\n",
        "* **Many to one:** many inputs, one outputs, such as text classification (classifying a Tweet as real diaster or not real diaster).\n",
        "* **Many to many:** many inputs, many outputs, such as machine translation (translating English to Spanish) or speech to text (audio wave as input, text as output).\n",
        "\n",
        "When you come across RNN's in the wild, you'll most likely come across variants of the following:\n",
        "* Long short-term memory cells (LSTMs).\n",
        "* Gated recurrent units (GRUs).\n",
        "* Bidirectional RNN's (passes forward and backward along a sequence, left to right and right to left).\n",
        "\n",
        "Going into the details of each these is beyond the scope of this notebook (we're going to focus on using them instead), the main thing you should know for now is that they've proven very effective at modelling sequences.\n",
        "\n",
        "For a deeper understanding of what's happening behind the scenes of the code we're about to write, I'd recommend the following resources:\n",
        "\n",
        "> 📖 **Resources:**\n",
        "> * [MIT Deep Learning Lecture on Recurrent Neural Networks](https://youtu.be/SEnXr6v2ifU) - explains the background of recurrent neural networks and introduces LSTMs.\n",
        "> * [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) by Andrej Karpathy - demonstrates the power of RNN's with examples generating various sequences.\n",
        "> * [Understanding LSTMs](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Chris Olah - an in-depth (and technical) look at the mechanics of the LSTM cell, possibly the most popular RNN building block.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tDERKwP_XWro"
      },
      "source": [
        "### Model 2: LSTM\n",
        "\n",
        "With all this talk of what RNN's are and what they're good for, I'm sure you're eager to build one.\n",
        "\n",
        "We're going to start with an LSTM-powered RNN.\n",
        "\n",
        "To harness the power of the LSTM cell (LSTM cell and LSTM layer are often used interchangably) in TensorFlow, we'll use [`tensorflow.keras.layers.LSTM()`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM).\n",
        "\n",
        "![](https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/images/08-RNN-architecture-coloured-block-edition.png)\n",
        "*Coloured block example of the structure of an recurrent neural network.*\n",
        "\n",
        "Our model is going to take on a very similar structure to `model_1`:\n",
        "\n",
        "```\n",
        "Input (text) -> Tokenize -> Embedding -> Layers -> Output (label probability)\n",
        "```\n",
        "\n",
        "The main difference will be that we're going to add an LSTM layer between our embedding and output.\n",
        "\n",
        "And to make sure we're not getting reusing trained embeddings (this would involve data leakage between models, leading to an uneven comparison later on), we'll create another embedding layer (`model_2_embedding`) for our model. The `text_vectorizer` layer can be reused since it doesn't get updated during training.\n",
        "\n",
        "> 🔑 **Note:** The reason we use a new embedding layer for each model is since the embedding layer is a *learned* representation of words (as numbers), if we were to use the same embedding layer (`embedding_1`) for each model, we'd be mixing what one model learned with the next. And because we want to compare our models later on, starting them with their own embedding layer each time is a better idea."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pi3vjpFU46hi",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "373cad05-9cf7-4c40-b41b-622f51563819"
      },
      "source": [
        "# Set random seed and create embedding layer (new embedding layer for each model)\n",
        "tf.random.set_seed(42)\n",
        "from tensorflow.keras import layers\n",
        "model_2_embedding = layers.Embedding(input_dim=max_vocab_length,\n",
        "                                     output_dim=128,\n",
        "                                     embeddings_initializer=\"uniform\",\n",
        "                                     input_length=max_length,\n",
        "                                     name=\"embedding_2\")\n",
        "\n",
        "\n",
        "# Create LSTM model\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = text_vectorizer(inputs)\n",
        "x = model_2_embedding(x)\n",
        "print(x.shape)\n",
        "# x = layers.LSTM(64, return_sequences=True)(x) # return vector for each word in the Tweet (you can stack RNN cells as long as return_sequences=True)\n",
        "x = layers.LSTM(64)(x) # return vector for whole sequence\n",
        "print(x.shape)\n",
        "# x = layers.Dense(64, activation=\"relu\")(x) # optional dense layer on top of output of LSTM cell\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model_2 = tf.keras.Model(inputs, outputs, name=\"model_2_LSTM\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(None, 15, 128)\n",
            "(None, 64)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e1wfTARuwWDg"
      },
      "source": [
        "> 🔑 **Note:** Reading the documentation for the [TensorFlow LSTM layer](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM), you'll find a plethora of parameters. Many of these have been tuned to make sure they compute as fast as possible. The main ones you'll be looking to adjust are `units` (number of hidden units) and `return_sequences` (set this to `True` when stacking LSTM or other recurrent layers).\n",
        "\n",
        "Now we've got our LSTM model built, let's compile it using `\"binary_crossentropy\"` loss and the Adam optimizer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pWdt3bFRwG6w"
      },
      "source": [
        "# Compile model\n",
        "model_2.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I2e_t8RFxgXG"
      },
      "source": [
        "And before we fit our model to the data, let's get a summary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IAjdfDfLwK_R",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f345c914-32a3-437a-89c9-07ae4989ec62"
      },
      "source": [
        "model_2.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_2_LSTM\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_2 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization_1 (TextVe (None, 15)                0         \n",
            "_________________________________________________________________\n",
            "embedding_2 (Embedding)      (None, 15, 128)           1280000   \n",
            "_________________________________________________________________\n",
            "lstm (LSTM)                  (None, 64)                49408     \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 1)                 65        \n",
            "=================================================================\n",
            "Total params: 1,329,473\n",
            "Trainable params: 1,329,473\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S5NLw3wD0aMz"
      },
      "source": [
        "Looking good! You'll notice a fair few more trainable parameters within our LSTM layer than `model_1`. \n",
        "\n",
        "If you'd like to know where this number comes from, I recommend going through the above resources as well the following on calculating the number of parameters in an LSTM cell:\n",
        "* [Stack Overflow answer to calculate the number of parameters in an LSTM cell](https://stackoverflow.com/questions/38080035/how-to-calculate-the-number-of-parameters-of-an-lstm-network) by Marcin Możejko\n",
        "* [Calculating number of parameters in a LSTM unit and layer](https://medium.com/@priyadarshi.cse/calculating-number-of-parameters-in-a-lstm-unit-layer-7e491978e1e4) by Shridhar Priyadarshi\n",
        "\n",
        "Now our first RNN model's compiled let's fit it to our training data, validating it on the validation data and tracking its training parameters using our TensorBoard callback."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YgZ7ojDvwKcq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "34da9032-cfe3-46b2-8cab-80d6abc3a593"
      },
      "source": [
        "# Fit model\n",
        "model_2_history = model_2.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(SAVE_DIR, \n",
        "                                                                     \"LSTM\")])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/LSTM/20210923-052618\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 12s 34ms/step - loss: 0.5100 - accuracy: 0.7416 - val_loss: 0.4566 - val_accuracy: 0.7822\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 4s 19ms/step - loss: 0.3176 - accuracy: 0.8717 - val_loss: 0.5138 - val_accuracy: 0.7756\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 4s 18ms/step - loss: 0.2201 - accuracy: 0.9152 - val_loss: 0.5858 - val_accuracy: 0.7677\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 4s 19ms/step - loss: 0.1556 - accuracy: 0.9428 - val_loss: 0.6041 - val_accuracy: 0.7743\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 4s 20ms/step - loss: 0.1076 - accuracy: 0.9594 - val_loss: 0.8746 - val_accuracy: 0.7507\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1gikGe_Z16PP"
      },
      "source": [
        "Nice! We've got our first trained RNN model using LSTM cells. Let's make some predictions with it.\n",
        "\n",
        "The same thing will happen as before, due to the sigmoid activiation function in the final layer, when we call the `predict()` method on our model, it'll return prediction probabilities rather than classes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4c_lVbKLemrU",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "08161e1d-fb72-4046-8ba1-1937487eebc4"
      },
      "source": [
        "# Make predictions on the validation dataset\n",
        "model_2_pred_probs = model_2.predict(val_sentences)\n",
        "model_2_pred_probs.shape, model_2_pred_probs[:10] # view the first 10"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((762, 1), array([[0.00712602],\n",
              "        [0.7873681 ],\n",
              "        [0.9996376 ],\n",
              "        [0.05679193],\n",
              "        [0.0025822 ],\n",
              "        [0.9996238 ],\n",
              "        [0.9217023 ],\n",
              "        [0.9997993 ],\n",
              "        [0.9994954 ],\n",
              "        [0.6645735 ]], dtype=float32))"
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fQ6ope-ddpOo"
      },
      "source": [
        "We can turn these prediction probabilities into prediction classes by rounding to the nearest integer (by default, prediction probabilities under 0.5 will go to 0 and those over 0.5 will go to 1)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iFnIhtyE7hlb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "284865b4-bfd5-470d-b3da-c3bef6f7ea92"
      },
      "source": [
        "# Round out predictions and reduce to 1-dimensional array\n",
        "model_2_preds = tf.squeeze(tf.round(model_2_pred_probs))\n",
        "model_2_preds[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zTBy4poXd_7p"
      },
      "source": [
        "Beautiful, now let's use our `caculate_results()` function to evaluate our LSTM model and our `compare_baseline_to_new_results()` function to compare it to our baseline model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3iHXv04y76vj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9265fa0c-a725-4333-f91f-4d96fa5a5c06"
      },
      "source": [
        "# Calculate LSTM model results\n",
        "model_2_results = calculate_results(y_true=val_labels,\n",
        "                                    y_pred=model_2_preds)\n",
        "model_2_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 75.06561679790026,\n",
              " 'f1': 0.7489268622514025,\n",
              " 'precision': 0.7510077975908164,\n",
              " 'recall': 0.7506561679790026}"
            ]
          },
          "metadata": {},
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZdQGn2L68B5Q",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "686d82ce-8eb5-40d3-9db2-85ffc4fa6484"
      },
      "source": [
        "# Compare model 2 to baseline\n",
        "compare_baseline_to_new_results(baseline_results, model_2_results)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Baseline accuracy: 79.27, New accuracy: 75.07, Difference: -4.20\n",
            "Baseline precision: 0.81, New precision: 0.75, Difference: -0.06\n",
            "Baseline recall: 0.79, New recall: 0.75, Difference: -0.04\n",
            "Baseline f1: 0.79, New f1: 0.75, Difference: -0.04\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q0pAtADt8ju7"
      },
      "source": [
        "### Model 3: GRU\n",
        "\n",
        "Another popular and effective RNN component is the GRU or gated recurrent unit.\n",
        "\n",
        "The GRU cell has similar features to an LSTM cell but has less parameters.\n",
        "\n",
        "> 📖 **Resource:** A full explanation of the GRU cell is beyond the scope of this noteook but I'd suggest the following resources to learn more:\n",
        "* [Gated Recurrent Unit](https://en.wikipedia.org/wiki/Gated_recurrent_unit) Wikipedia page\n",
        "* [Understanding GRU networks](https://towardsdatascience.com/understanding-gru-networks-2ef37df6c9be) by Simeon Kostadinov\n",
        "\n",
        "To use the GRU cell in TensorFlow, we can call the [`tensorflow.keras.layers.GRU()`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/GRU) class.\n",
        "\n",
        "The architecture of the GRU-powered model will follow the same structure we've been using:\n",
        "\n",
        "```\n",
        "Input (text) -> Tokenize -> Embedding -> Layers -> Output (label probability)\n",
        "```\n",
        "\n",
        "Again, the only difference will be the layer(s) we use between the embedding and the output."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SoSCGq3H47Yo"
      },
      "source": [
        "# Set random seed and create embedding layer (new embedding layer for each model)\n",
        "tf.random.set_seed(42)\n",
        "from tensorflow.keras import layers\n",
        "model_3_embedding = layers.Embedding(input_dim=max_vocab_length,\n",
        "                                     output_dim=128,\n",
        "                                     embeddings_initializer=\"uniform\",\n",
        "                                     input_length=max_length,\n",
        "                                     name=\"embedding_3\")\n",
        "\n",
        "# Build an RNN using the GRU cell\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = text_vectorizer(inputs)\n",
        "x = model_3_embedding(x)\n",
        "# x = layers.GRU(64, return_sequences=True) # stacking recurrent cells requires return_sequences=True\n",
        "x = layers.GRU(64)(x) \n",
        "# x = layers.Dense(64, activation=\"relu\")(x) # optional dense layer after GRU cell\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model_3 = tf.keras.Model(inputs, outputs, name=\"model_3_GRU\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JLT5maFWhKH1"
      },
      "source": [
        "TensorFlow makes it easy to use powerful components such as the GRU cell in our models. And now our third model is built, let's compile it, just as before."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lBL1mb31hHDS"
      },
      "source": [
        "# Compile GRU model\n",
        "model_3.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yvnksvkmha2A"
      },
      "source": [
        "What does a summary of our model look like?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JVnB5yQeiAWs",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "068184f9-c913-46fd-83d9-8547e8f4f1f6"
      },
      "source": [
        "# Get a summary of the GRU model\n",
        "model_3.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_3_GRU\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_3 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization_1 (TextVe (None, 15)                0         \n",
            "_________________________________________________________________\n",
            "embedding_3 (Embedding)      (None, 15, 128)           1280000   \n",
            "_________________________________________________________________\n",
            "gru (GRU)                    (None, 64)                37248     \n",
            "_________________________________________________________________\n",
            "dense_2 (Dense)              (None, 1)                 65        \n",
            "=================================================================\n",
            "Total params: 1,317,313\n",
            "Trainable params: 1,317,313\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KcXzKqgXhdez"
      },
      "source": [
        "Notice the difference in number of trainable parameters between `model_2` (LSTM) and `model_3` (GRU). The difference comes from the LSTM cell having more trainable parameters than the GRU cell.\n",
        "\n",
        "We'll fit our model just as we've been doing previously. We'll also track our models results using our `create_tensorboard_callback()` function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gvamg5JOh_jC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "629279a6-1dd9-43fb-f0ae-975be030ebe7"
      },
      "source": [
        "# Fit model\n",
        "model_3_history = model_3.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(SAVE_DIR, \"GRU\")])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/GRU/20210923-052650\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 12s 24ms/step - loss: 0.5242 - accuracy: 0.7314 - val_loss: 0.4553 - val_accuracy: 0.7769\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 3s 15ms/step - loss: 0.3195 - accuracy: 0.8694 - val_loss: 0.4937 - val_accuracy: 0.7808\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 2s 10ms/step - loss: 0.2197 - accuracy: 0.9181 - val_loss: 0.5607 - val_accuracy: 0.7743\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 2s 10ms/step - loss: 0.1599 - accuracy: 0.9441 - val_loss: 0.6220 - val_accuracy: 0.7782\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 2s 10ms/step - loss: 0.1221 - accuracy: 0.9584 - val_loss: 0.6205 - val_accuracy: 0.7677\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hM4mQj1Sh7Gn"
      },
      "source": [
        "Due to the optimized default settings of the GRU cell in TensorFlow, training doesn't take long at all. \n",
        "\n",
        "Time to make some predictions on the validation samples."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W5TUVHCl9pe-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "21116150-7412-49c4-ae12-679f3c688c44"
      },
      "source": [
        "# Make predictions on the validation data\n",
        "model_3_pred_probs = model_3.predict(val_sentences)\n",
        "model_3_pred_probs.shape, model_3_pred_probs[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((762, 1), array([[0.33325258],\n",
              "        [0.87741184],\n",
              "        [0.9980252 ],\n",
              "        [0.11561754],\n",
              "        [0.01235959],\n",
              "        [0.9925639 ],\n",
              "        [0.6214262 ],\n",
              "        [0.99813336],\n",
              "        [0.9982377 ],\n",
              "        [0.50181067]], dtype=float32))"
            ]
          },
          "metadata": {},
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hasS7dzRiYQh"
      },
      "source": [
        "Again we get an array of prediction probabilities back which we can convert to prediction classes by rounding them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "haILbddg98CY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8d567022-4f1f-46cb-aae1-70242ce55e56"
      },
      "source": [
        "# Convert prediction probabilities to prediction classes\n",
        "model_3_preds = tf.squeeze(tf.round(model_3_pred_probs))\n",
        "model_3_preds[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 61
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_7yAgh-viglB"
      },
      "source": [
        "Now we've got predicted classes, let's evaluate them against the ground truth labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h9OZbQu1-LPp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3b32ee9f-3291-4f8c-873d-c7780488cab8"
      },
      "source": [
        "# Calcuate model_3 results\n",
        "model_3_results = calculate_results(y_true=val_labels, \n",
        "                                    y_pred=model_3_preds)\n",
        "model_3_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 76.77165354330708,\n",
              " 'f1': 0.7667932666650168,\n",
              " 'precision': 0.7675450859410361,\n",
              " 'recall': 0.7677165354330708}"
            ]
          },
          "metadata": {},
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o9t7wcALiuRk"
      },
      "source": [
        "Finally we can compare our GRU model's results to our baseline."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_7AE6vtn-RQZ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ff8ad89a-089a-4b29-be3f-b0c6f87c19ad"
      },
      "source": [
        "# Compare to baseline\n",
        "compare_baseline_to_new_results(baseline_results, model_3_results)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Baseline accuracy: 79.27, New accuracy: 76.77, Difference: -2.49\n",
            "Baseline precision: 0.81, New precision: 0.77, Difference: -0.04\n",
            "Baseline recall: 0.79, New recall: 0.77, Difference: -0.02\n",
            "Baseline f1: 0.79, New f1: 0.77, Difference: -0.02\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oLm6r4nQ-Wdr"
      },
      "source": [
        "### Model 4: Bidirectonal RNN model \n",
        "\n",
        "Look at us go! We've already built two RNN's with GRU and LSTM cells. Now we're going to look into another kind of RNN, the bidirectional RNN.\n",
        "\n",
        "A standard RNN will process a sequence from left to right, where as a bidirectional RNN will process the sequence from left to right and then again from right to left.\n",
        "\n",
        "Intuitively, this can be thought of as if you were reading a sentence for the first time in the normal fashion (left to right) but for some reason it didn't make sense so you traverse back through the words and go back over them again (right to left).\n",
        "\n",
        "In practice, many sequence models often see and improvement in performance when using bidirectional RNN's.\n",
        "\n",
        "However, this improvement in performance often comes at the cost of longer training times and increased model parameters (since the model goes left to right and right to left, the number of trainable parameters doubles).\n",
        "\n",
        "Okay enough talk, let's build a bidirectional RNN.\n",
        "\n",
        "Once again, TensorFlow helps us out by providing the [`tensorflow.keras.layers.Bidirectional`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Bidirectional) class. We can use the `Bidirectional` class to wrap our existing RNNs, instantly making them bidirectional."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NAU9dvGm47_2"
      },
      "source": [
        "# Set random seed and create embedding layer (new embedding layer for each model)\n",
        "tf.random.set_seed(42)\n",
        "from tensorflow.keras import layers\n",
        "model_4_embedding = layers.Embedding(input_dim=max_vocab_length,\n",
        "                                     output_dim=128,\n",
        "                                     embeddings_initializer=\"uniform\",\n",
        "                                     input_length=max_length,\n",
        "                                     name=\"embedding_4\")\n",
        "\n",
        "# Build a Bidirectional RNN in TensorFlow\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = text_vectorizer(inputs)\n",
        "x = model_4_embedding(x)\n",
        "# x = layers.Bidirectional(layers.LSTM(64, return_sequences=True))(x) # stacking RNN layers requires return_sequences=True\n",
        "x = layers.Bidirectional(layers.LSTM(64))(x) # bidirectional goes both ways so has double the parameters of a regular LSTM layer\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model_4 = tf.keras.Model(inputs, outputs, name=\"model_4_Bidirectional\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Hm5cwmNm-g4"
      },
      "source": [
        "> 🔑 **Note:** You can use the `Bidirectional` wrapper on any RNN cell in TensorFlow. For example, `layers.Bidirectional(layers.GRU(64))` creates a bidirectional GRU cell.\n",
        "\n",
        "Our bidirectional model is built, let's compile it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wP1jeF0am9x0"
      },
      "source": [
        "# Compile\n",
        "model_4.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NtpYyjsbnEwN"
      },
      "source": [
        "And of course, we'll check out a summary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-sUd9AQ6nFXI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dc1d6295-7753-41b9-df0d-4f4986cb3cbf"
      },
      "source": [
        "# Get a summary of our bidirectional model\n",
        "model_4.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_4_Bidirectional\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_4 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization_1 (TextVe (None, 15)                0         \n",
            "_________________________________________________________________\n",
            "embedding_4 (Embedding)      (None, 15, 128)           1280000   \n",
            "_________________________________________________________________\n",
            "bidirectional (Bidirectional (None, 128)               98816     \n",
            "_________________________________________________________________\n",
            "dense_3 (Dense)              (None, 1)                 129       \n",
            "=================================================================\n",
            "Total params: 1,378,945\n",
            "Trainable params: 1,378,945\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TvItfzeZnIE-"
      },
      "source": [
        "Notice the increased number of trainable parameters in `model_4` (bidirectional LSTM) compared to `model_2` (regular LSTM). This is due to the bidirectionality we added to our RNN.\n",
        "\n",
        "Time to fit our bidirectional model and track its performance."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bAKY_QbHXPHB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1e576aae-b7a6-4bc5-b229-99e0885d9bd2"
      },
      "source": [
        "# Fit the model (takes longer because of the bidirectional layers)\n",
        "model_4_history = model_4.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(SAVE_DIR, \"bidirectional_RNN\")])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/bidirectional_RNN/20210923-052719\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 8s 21ms/step - loss: 0.5093 - accuracy: 0.7481 - val_loss: 0.4606 - val_accuracy: 0.7795\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 3s 14ms/step - loss: 0.3135 - accuracy: 0.8708 - val_loss: 0.5144 - val_accuracy: 0.7690\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 3s 14ms/step - loss: 0.2150 - accuracy: 0.9178 - val_loss: 0.5626 - val_accuracy: 0.7677\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 3s 14ms/step - loss: 0.1523 - accuracy: 0.9469 - val_loss: 0.6365 - val_accuracy: 0.7769\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 3s 14ms/step - loss: 0.1083 - accuracy: 0.9639 - val_loss: 0.6509 - val_accuracy: 0.7664\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zkt8GVRHoJz6"
      },
      "source": [
        "Due to the bidirectionality of our model we see a slight increase in training time.\n",
        "\n",
        "Not to worry, it's not too dramatic of an increase.\n",
        "\n",
        "Let's make some predictions with it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uFc7QHRtXmn7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "67a915b9-2e27-492e-86a7-6b114354cfcf"
      },
      "source": [
        "# Make predictions with bidirectional RNN on the validation data\n",
        "model_4_pred_probs = model_4.predict(val_sentences)\n",
        "model_4_pred_probs[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.04000043],\n",
              "       [0.827929  ],\n",
              "       [0.99842227],\n",
              "       [0.1353109 ],\n",
              "       [0.00311337],\n",
              "       [0.99220747],\n",
              "       [0.9552836 ],\n",
              "       [0.99945647],\n",
              "       [0.99898285],\n",
              "       [0.28141677]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L_9HmNIYobDB"
      },
      "source": [
        "And we'll convert them to prediction classes and evaluate them against the ground truth labels and baseline model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G5z8bMdaXw51",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ecbb2c9e-5b45-4872-a126-6f6b4de3f091"
      },
      "source": [
        "# Convert prediction probabilities to labels\n",
        "model_4_preds = tf.squeeze(tf.round(model_4_pred_probs))\n",
        "model_4_preds[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-a7Ym_vKYAO4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a3d841a2-6cdd-4da2-d623-d046f608841c"
      },
      "source": [
        "# Calculate bidirectional RNN model results\n",
        "model_4_results = calculate_results(val_labels, model_4_preds)\n",
        "model_4_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 76.64041994750657,\n",
              " 'f1': 0.7651213533864446,\n",
              " 'precision': 0.7665895370389821,\n",
              " 'recall': 0.7664041994750657}"
            ]
          },
          "metadata": {},
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hAET-LKpYT18",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d69437c7-e780-41ab-d575-b9cfdefdf82b"
      },
      "source": [
        "# Check to see how the bidirectional model performs against the baseline\n",
        "compare_baseline_to_new_results(baseline_results, model_4_results)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Baseline accuracy: 79.27, New accuracy: 76.64, Difference: -2.62\n",
            "Baseline precision: 0.81, New precision: 0.77, Difference: -0.04\n",
            "Baseline recall: 0.79, New recall: 0.77, Difference: -0.03\n",
            "Baseline f1: 0.79, New f1: 0.77, Difference: -0.02\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wcvt_7emuKlR"
      },
      "source": [
        "## Convolutional Neural Networks for Text\n",
        "\n",
        "You might've used convolutional neural networks (CNNs) for images before but they can also be used for sequences.\n",
        "\n",
        "The main difference between using CNNs for images and sequences is the shape of the data. Images come in 2-dimensions (height x width) where as sequences are often 1-dimensional (a string of text).\n",
        "\n",
        "So to use CNNs with sequences, we use a 1-dimensional convolution instead of a 2-dimensional convolution.\n",
        "\n",
        "A typical CNN architecture for sequences will look like the following: \n",
        "\n",
        "```\n",
        "Inputs (text) -> Tokenization -> Embedding -> Layers -> Outputs (class probabilities)\n",
        "```\n",
        "\n",
        "You might be thinking \"that just looks like the architecture layout we've been using for the other models...\"\n",
        "\n",
        "And you'd be right.\n",
        "\n",
        "The difference again is in the layers component. Instead of using an LSTM or GRU cell, we're going to use a [`tensorflow.keras.layers.Conv1D()`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv1D) layer followed by a [`tensorflow.keras.layers.GlobablMaxPool1D()`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/GlobalMaxPool1D) layer.\n",
        "\n",
        "> 📖 **Resource:** The intuition here is explained succinctly in the paper [*Understanding Convolutional Neural Networks for Text Classification*](https://www.aclweb.org/anthology/W18-5408.pdf), where they state that CNNs classify text through the following steps:\n",
        "1. 1-dimensional convolving filters are used as ngram detectors, each filter specializing in a closely-related family of ngrams (an ngram is a collection of n-words, for example, an ngram of 5 might result in \"hello, my name is Daniel\").\n",
        "2. Max-pooling over time extracts the relevant ngrams for making a decision.\n",
        "3. The rest of the network classifies the text based on this information.\n",
        "\n",
        "> \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lgXEorf9GWY1"
      },
      "source": [
        "### Model 5: Conv1D\n",
        "\n",
        "Before we build a full 1-dimensional CNN model, let's see a 1-dimensional convolutional layer (also called a **temporal convolution**) in action.\n",
        "\n",
        "We'll first create an embedding of a sample of text and experiment passing it through a `Conv1D()` layer and `GlobalMaxPool1D()` layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "563hl7nPWP_3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "32a7e900-e4b4-4b1b-9b4e-617ca9f7eb92"
      },
      "source": [
        "# Test out the embedding, 1D convolutional and max pooling\n",
        "embedding_test = embedding(text_vectorizer([\"this is a test sentence\"])) # turn target sentence into embedding\n",
        "conv_1d = layers.Conv1D(filters=32, kernel_size=5, activation=\"relu\") # convolve over target sequence 5 words at a time\n",
        "conv_1d_output = conv_1d(embedding_test) # pass embedding through 1D convolutional layer\n",
        "max_pool = layers.GlobalMaxPool1D() \n",
        "max_pool_output = max_pool(conv_1d_output) # get the most important features\n",
        "embedding_test.shape, conv_1d_output.shape, max_pool_output.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(TensorShape([1, 15, 128]), TensorShape([1, 11, 32]), TensorShape([1, 32]))"
            ]
          },
          "metadata": {},
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-WzTeShEemJ2"
      },
      "source": [
        "Notice the output shapes of each layer.\n",
        "\n",
        "The embedding has an output shape dimension of the parameters we set it to (`input_length=15` and `output_dim=128`).\n",
        "\n",
        "The 1-dimensional convolutional layer has an output which has been compressed inline with its parameters. And the same goes for the max pooling layer output.\n",
        "\n",
        "Our text starts out as a string but gets converted to a feature vector of length 64 through various transformation steps (from tokenization to embedding to 1-dimensional convolution to max pool).\n",
        "\n",
        "Let's take a peak at what each of these transformations looks like."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gRcxYgs-dxM8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "33616ed5-7333-4891-d3fa-d798c64fd789"
      },
      "source": [
        "# See the outputs of each layer\n",
        "embedding_test[:1], conv_1d_output[:1], max_pool_output[:1]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(<tf.Tensor: shape=(1, 15, 128), dtype=float32, numpy=\n",
              " array([[[ 0.02534914, -0.03109061,  0.00285616, ..., -0.00783159,\n",
              "          -0.02685575, -0.0443413 ],\n",
              "         [-0.0658626 ,  0.09451495, -0.01477603, ..., -0.00657781,\n",
              "          -0.04238792,  0.07777896],\n",
              "         [-0.04803652, -0.00709756, -0.02330894, ..., -0.0180733 ,\n",
              "           0.02351036,  0.02676384],\n",
              "         ...,\n",
              "         [ 0.00073166,  0.01504797, -0.03425457, ..., -0.04403538,\n",
              "          -0.01042282,  0.01876436],\n",
              "         [ 0.00073166,  0.01504797, -0.03425457, ..., -0.04403538,\n",
              "          -0.01042282,  0.01876436],\n",
              "         [ 0.00073166,  0.01504797, -0.03425457, ..., -0.04403538,\n",
              "          -0.01042282,  0.01876436]]], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1, 11, 32), dtype=float32, numpy=\n",
              " array([[[0.08324985, 0.00648716, 0.        , 0.03983572, 0.        ,\n",
              "          0.01144416, 0.00416251, 0.0228839 , 0.        , 0.00900978,\n",
              "          0.        , 0.        , 0.03401771, 0.06408274, 0.08103722,\n",
              "          0.00409014, 0.01579616, 0.        , 0.07930177, 0.        ,\n",
              "          0.        , 0.        , 0.14525084, 0.        , 0.        ,\n",
              "          0.        , 0.03682078, 0.06534287, 0.        , 0.        ,\n",
              "          0.05094624, 0.        ],\n",
              "         [0.        , 0.05387188, 0.        , 0.11491331, 0.        ,\n",
              "          0.        , 0.1623708 , 0.        , 0.        , 0.00171254,\n",
              "          0.14336711, 0.        , 0.        , 0.        , 0.        ,\n",
              "          0.01197936, 0.        , 0.        , 0.13551372, 0.0040106 ,\n",
              "          0.10309819, 0.09445544, 0.08390297, 0.        , 0.04213036,\n",
              "          0.04487597, 0.06560461, 0.        , 0.02272684, 0.        ,\n",
              "          0.        , 0.        ],\n",
              "         [0.03683221, 0.04895764, 0.        , 0.1532475 , 0.        ,\n",
              "          0.        , 0.        , 0.        , 0.        , 0.        ,\n",
              "          0.        , 0.04650313, 0.00496456, 0.07349401, 0.01608641,\n",
              "          0.        , 0.02779119, 0.        , 0.0808056 , 0.01403176,\n",
              "          0.        , 0.03768815, 0.1038278 , 0.        , 0.03361662,\n",
              "          0.        , 0.02577607, 0.00140354, 0.        , 0.        ,\n",
              "          0.03211498, 0.        ],\n",
              "         [0.0088782 , 0.10450974, 0.        , 0.06974535, 0.02328686,\n",
              "          0.        , 0.04052207, 0.        , 0.        , 0.02733764,\n",
              "          0.08674346, 0.        , 0.        , 0.06129852, 0.02007267,\n",
              "          0.        , 0.        , 0.        , 0.03364263, 0.        ,\n",
              "          0.04525332, 0.05219702, 0.06375706, 0.        , 0.        ,\n",
              "          0.00774407, 0.00273467, 0.        , 0.        , 0.00499633,\n",
              "          0.        , 0.        ],\n",
              "         [0.        , 0.02369069, 0.        , 0.05827617, 0.05297644,\n",
              "          0.        , 0.        , 0.        , 0.        , 0.        ,\n",
              "          0.01719718, 0.02936822, 0.00466103, 0.06879887, 0.01944808,\n",
              "          0.01585533, 0.01294545, 0.        , 0.06866529, 0.        ,\n",
              "          0.00623766, 0.0351405 , 0.02407533, 0.        , 0.05979815,\n",
              "          0.        , 0.01170142, 0.        , 0.        , 0.        ,\n",
              "          0.04444929, 0.        ],\n",
              "         [0.03544863, 0.        , 0.        , 0.05054973, 0.06105441,\n",
              "          0.        , 0.00997427, 0.01403005, 0.        , 0.01680727,\n",
              "          0.0314851 , 0.03889389, 0.        , 0.07710679, 0.0059097 ,\n",
              "          0.        , 0.00263033, 0.        , 0.08935824, 0.        ,\n",
              "          0.        , 0.05331149, 0.0522795 , 0.        , 0.06658384,\n",
              "          0.01881707, 0.02448696, 0.        , 0.        , 0.        ,\n",
              "          0.02008456, 0.        ],\n",
              "         [0.03544863, 0.        , 0.        , 0.05054973, 0.06105442,\n",
              "          0.        , 0.00997426, 0.01403006, 0.        , 0.01680727,\n",
              "          0.03148509, 0.03889391, 0.        , 0.07710679, 0.0059097 ,\n",
              "          0.        , 0.00263035, 0.        , 0.08935823, 0.        ,\n",
              "          0.        , 0.05331149, 0.05227951, 0.        , 0.06658384,\n",
              "          0.01881707, 0.02448694, 0.        , 0.        , 0.        ,\n",
              "          0.02008457, 0.        ],\n",
              "         [0.03544864, 0.        , 0.        , 0.05054973, 0.06105441,\n",
              "          0.        , 0.00997426, 0.01403005, 0.        , 0.01680726,\n",
              "          0.0314851 , 0.03889389, 0.        , 0.07710679, 0.0059097 ,\n",
              "          0.        , 0.00263034, 0.        , 0.08935826, 0.        ,\n",
              "          0.        , 0.0533115 , 0.0522795 , 0.        , 0.06658384,\n",
              "          0.01881707, 0.02448694, 0.        , 0.        , 0.        ,\n",
              "          0.02008457, 0.        ],\n",
              "         [0.03544863, 0.        , 0.        , 0.05054973, 0.06105442,\n",
              "          0.        , 0.00997426, 0.01403005, 0.        , 0.01680727,\n",
              "          0.0314851 , 0.0388939 , 0.        , 0.07710679, 0.0059097 ,\n",
              "          0.        , 0.00263034, 0.        , 0.08935825, 0.        ,\n",
              "          0.        , 0.05331149, 0.05227951, 0.        , 0.06658386,\n",
              "          0.01881707, 0.02448695, 0.        , 0.        , 0.        ,\n",
              "          0.02008456, 0.        ],\n",
              "         [0.03544863, 0.        , 0.        , 0.05054973, 0.0610544 ,\n",
              "          0.        , 0.00997427, 0.01403005, 0.        , 0.01680727,\n",
              "          0.0314851 , 0.0388939 , 0.        , 0.0771068 , 0.0059097 ,\n",
              "          0.        , 0.00263034, 0.        , 0.08935825, 0.        ,\n",
              "          0.        , 0.05331149, 0.05227951, 0.        , 0.06658386,\n",
              "          0.01881707, 0.02448695, 0.        , 0.        , 0.        ,\n",
              "          0.02008456, 0.        ],\n",
              "         [0.03544863, 0.        , 0.        , 0.05054973, 0.06105442,\n",
              "          0.        , 0.00997426, 0.01403006, 0.        , 0.01680726,\n",
              "          0.03148509, 0.0388939 , 0.        , 0.0771068 , 0.0059097 ,\n",
              "          0.        , 0.00263034, 0.        , 0.08935824, 0.        ,\n",
              "          0.        , 0.05331149, 0.05227952, 0.        , 0.06658386,\n",
              "          0.01881706, 0.02448695, 0.        , 0.        , 0.        ,\n",
              "          0.02008456, 0.        ]]], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1, 32), dtype=float32, numpy=\n",
              " array([[0.08324985, 0.10450974, 0.        , 0.1532475 , 0.06105442,\n",
              "         0.01144416, 0.1623708 , 0.0228839 , 0.        , 0.02733764,\n",
              "         0.14336711, 0.04650313, 0.03401771, 0.0771068 , 0.08103722,\n",
              "         0.01585533, 0.02779119, 0.        , 0.13551372, 0.01403176,\n",
              "         0.10309819, 0.09445544, 0.14525084, 0.        , 0.06658386,\n",
              "         0.04487597, 0.06560461, 0.06534287, 0.02272684, 0.00499633,\n",
              "         0.05094624, 0.        ]], dtype=float32)>)"
            ]
          },
          "metadata": {},
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kMcrthJwg3B2"
      },
      "source": [
        "Alright, we've seen the outputs of several components of a CNN for sequences, let's put them together and construct a full model, compile it (just as we've done with our other models) and get a summary. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G9aphPWCYkWN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6d723bc4-84d1-49eb-9ef9-0887364f029b"
      },
      "source": [
        "# Set random seed and create embedding layer (new embedding layer for each model)\n",
        "tf.random.set_seed(42)\n",
        "from tensorflow.keras import layers\n",
        "model_5_embedding = layers.Embedding(input_dim=max_vocab_length,\n",
        "                                     output_dim=128,\n",
        "                                     embeddings_initializer=\"uniform\",\n",
        "                                     input_length=max_length,\n",
        "                                     name=\"embedding_5\")\n",
        "\n",
        "# Create 1-dimensional convolutional layer to model sequences\n",
        "from tensorflow.keras import layers\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = text_vectorizer(inputs)\n",
        "x = model_5_embedding(x)\n",
        "x = layers.Conv1D(filters=32, kernel_size=5, activation=\"relu\")(x)\n",
        "x = layers.GlobalMaxPool1D()(x)\n",
        "# x = layers.Dense(64, activation=\"relu\")(x) # optional dense layer\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model_5 = tf.keras.Model(inputs, outputs, name=\"model_5_Conv1D\")\n",
        "\n",
        "# Compile Conv1D model\n",
        "model_5.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "# Get a summary of our 1D convolution model\n",
        "model_5.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_5_Conv1D\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_5 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization_1 (TextVe (None, 15)                0         \n",
            "_________________________________________________________________\n",
            "embedding_5 (Embedding)      (None, 15, 128)           1280000   \n",
            "_________________________________________________________________\n",
            "conv1d_1 (Conv1D)            (None, 11, 32)            20512     \n",
            "_________________________________________________________________\n",
            "global_max_pooling1d_1 (Glob (None, 32)                0         \n",
            "_________________________________________________________________\n",
            "dense_4 (Dense)              (None, 1)                 33        \n",
            "=================================================================\n",
            "Total params: 1,300,545\n",
            "Trainable params: 1,300,545\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o1Y4BpMGh0jG"
      },
      "source": [
        "Woohoo! Looking great! Notice how the number of trainable parameters for the 1-dimensional convolutional layer is similar to that of the LSTM layer in `model_2`.\n",
        "\n",
        "Let's fit our 1D CNN model to our text data. In line with previous experiments, we'll save its results using our `create_tensorboard_callback()` function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9fzlaKm1ZrMX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "544b3d97-2d9c-40a3-b0e5-51feac67a4fa"
      },
      "source": [
        "# Fit the model\n",
        "model_5_history = model_5.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(SAVE_DIR, \n",
        "                                                                     \"Conv1D\")])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/Conv1D/20210923-052810\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 5s 10ms/step - loss: 0.5652 - accuracy: 0.7141 - val_loss: 0.4733 - val_accuracy: 0.7795\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 2s 7ms/step - loss: 0.3380 - accuracy: 0.8615 - val_loss: 0.4758 - val_accuracy: 0.7730\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 2s 8ms/step - loss: 0.2070 - accuracy: 0.9234 - val_loss: 0.5457 - val_accuracy: 0.7730\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 2s 7ms/step - loss: 0.1314 - accuracy: 0.9578 - val_loss: 0.6163 - val_accuracy: 0.7730\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 2s 7ms/step - loss: 0.0933 - accuracy: 0.9691 - val_loss: 0.6779 - val_accuracy: 0.7782\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d2up-1tLiXKD"
      },
      "source": [
        "Nice! Thanks to GPU acceleration, our 1D convolutional model trains nice and fast. Let's make some predictions with it and evaluate them just as before."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZHYw5GkxZ2OK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "44a902e9-9710-4854-d61d-649b7a6f5ef3"
      },
      "source": [
        "# Make predictions with model_5\n",
        "model_5_pred_probs = model_5.predict(val_sentences)\n",
        "model_5_pred_probs[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.225345  ],\n",
              "       [0.7534112 ],\n",
              "       [0.9995602 ],\n",
              "       [0.05562792],\n",
              "       [0.01449848],\n",
              "       [0.9858518 ],\n",
              "       [0.98418933],\n",
              "       [0.99758804],\n",
              "       [0.99862623],\n",
              "       [0.26914373]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v9YqTtjiaauS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0bb1e353-5ecc-4cdb-cd73-07911d7857c2"
      },
      "source": [
        "# Convert model_5 prediction probabilities to labels\n",
        "model_5_preds = tf.squeeze(tf.round(model_5_pred_probs))\n",
        "model_5_preds[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 77
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wMY3s1Pnaj34",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d45f024e-6352-4583-8946-bf3da8249a21"
      },
      "source": [
        "# Calculate model_5 evaluation metrics \n",
        "model_5_results = calculate_results(y_true=val_labels, \n",
        "                                    y_pred=model_5_preds)\n",
        "model_5_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 77.82152230971128,\n",
              " 'f1': 0.7758810170952618,\n",
              " 'precision': 0.7807522349051432,\n",
              " 'recall': 0.7782152230971129}"
            ]
          },
          "metadata": {},
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wRfF4B6_at8k",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8de35775-b434-4465-e63c-0066a7fa3f69"
      },
      "source": [
        "# Compare model_5 results to baseline \n",
        "compare_baseline_to_new_results(baseline_results, model_5_results)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Baseline accuracy: 79.27, New accuracy: 77.82, Difference: -1.44\n",
            "Baseline precision: 0.81, New precision: 0.78, Difference: -0.03\n",
            "Baseline recall: 0.79, New recall: 0.78, Difference: -0.01\n",
            "Baseline f1: 0.79, New f1: 0.78, Difference: -0.01\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g_roVSSRt-7h"
      },
      "source": [
        "## Using Pretrained Embeddings (transfer learning for NLP)\n",
        "\n",
        "For all of the previous deep learning models we've built and trained, we've created and used our own embeddings from scratch each time.\n",
        "\n",
        "However, a common practice is to leverage pretrained embeddings through **transfer learning**. This is one of the main benefits of using deep models: being able to take what one (often larger) model has learned (often on a large amount of data) and adjust it for our own use case.\n",
        "\n",
        "For our next model, instead of using our own embedding layer, we're going to replace it with a pretrained embedding layer.\n",
        "\n",
        "More specifically, we're going to be using the [Universal Sentence Encoder](https://www.aclweb.org/anthology/D18-2029.pdf) from [TensorFlow Hub](https://tfhub.dev/google/universal-sentence-encoder/4) (a great resource containing a plethora of pretrained model resources for a variety of tasks).\n",
        "\n",
        "> 🔑 **Note:** There are many different pretrained text embedding options on TensorFlow Hub, however, some require different levels of text preprocessing than others. Best to experiment with a few and see which best suits your use case.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R-NQ2MA5GZBo"
      },
      "source": [
        "### Model 6: TensorFlow Hub Pretrained Sentence Encoder\n",
        "\n",
        "The main difference between the embedding layer we created and the Universal Sentence Encoder is that rather than create a word-level embedding, the Universal Sentence Encoder, as you might've guessed, creates a whole sentence-level embedding.\n",
        "\n",
        "Our embedding layer also outputs an a 128 dimensional vector for each word, where as, the Universal Sentence Encoder outputs a 512 dimensional vector for each sentence.\n",
        "\n",
        "![](https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/images/08-USE-tensorflow-hub-encoder-decoder-model.png)\n",
        "*The feature extractor model we're building through the eyes of an **encoder/decoder** model.*\n",
        "\n",
        "> 🔑 **Note:** An **encoder** is the name for a model which converts raw data such as text into a numerical representation (feature vector), a **decoder** converts the numerical representation to a desired output.\n",
        "\n",
        "As usual, this is best demonstrated with an example.\n",
        "\n",
        "We can load in a TensorFlow Hub module using the [`hub.load()`](https://www.tensorflow.org/hub/api_docs/python/hub/load) method and passing it the target URL of the module we'd like to use, in our case, it's \"https://tfhub.dev/google/universal-sentence-encoder/4\".\n",
        "\n",
        "Let's load the Universal Sentence Encoder model and test it on a couple of sentences."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7piW5jtxbUkV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fc509df1-e7a6-4145-82ca-674785ac5257"
      },
      "source": [
        "# Example of pretrained embedding with universal sentence encoder - https://tfhub.dev/google/universal-sentence-encoder/4\n",
        "import tensorflow_hub as hub\n",
        "embed = hub.load(\"https://tfhub.dev/google/universal-sentence-encoder/4\") # load Universal Sentence Encoder\n",
        "embed_samples = embed([sample_sentence,\n",
        "                      \"When you call the universal sentence encoder on a sentence, it turns it into numbers.\"])\n",
        "\n",
        "print(embed_samples[0][:50])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tf.Tensor(\n",
            "[-0.01157024  0.0248591   0.0287805  -0.01271502  0.03971543  0.08827759\n",
            "  0.02680986  0.05589837 -0.01068731 -0.0059729   0.00639324 -0.01819523\n",
            "  0.00030817  0.09105891  0.05874644 -0.03180627  0.01512476 -0.05162928\n",
            "  0.00991369 -0.06865346 -0.04209306  0.0267898   0.03011008  0.00321069\n",
            " -0.00337969 -0.04787359  0.02266718 -0.00985924 -0.04063614 -0.01292095\n",
            " -0.04666384  0.056303   -0.03949255  0.00517685  0.02495828 -0.07014439\n",
            "  0.02871508  0.04947682 -0.00633971 -0.08960191  0.02807117 -0.00808362\n",
            " -0.01360601  0.05998649 -0.10361786 -0.05195372  0.00232955 -0.02332528\n",
            " -0.03758105  0.0332773 ], shape=(50,), dtype=float32)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vvArnKkGb4vu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "67d73b29-193e-4e95-bd8d-e071b551ec3b"
      },
      "source": [
        "# Each sentence has been encoded into a 512 dimension vector\n",
        "embed_samples[0].shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([512])"
            ]
          },
          "metadata": {},
          "execution_count": 81
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZxYFDkGD-XjF"
      },
      "source": [
        "Passing our sentences to the Universal Sentence Encoder (USE) encodes them from strings to 512 dimensional vectors, which make no sense to us but hopefully make sense to our machine learning models.\n",
        "\n",
        "Speaking of models, let's build one with the USE as our embedding layer.\n",
        "\n",
        "We can convert the TensorFlow Hub USE module into a Keras layer using the [`hub.KerasLayer`](https://www.tensorflow.org/hub/api_docs/python/hub/KerasLayer) class.\n",
        "\n",
        "> 🔑 **Note:** Due to the size of the USE TensorFlow Hub module, it may take a little while to download. Once it's downloaded though, it'll be cached and ready to use. And as with many TensorFlow Hub modules, there is a [\"lite\" version of the USE](https://tfhub.dev/google/universal-sentence-encoder-lite/2) which takes up less space but sacrifices some performance and requires more preprocessing steps. However, depending on your available compute power, the lite version may be better for your application use case."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZcbBj0aXqrs9"
      },
      "source": [
        "# We can use this encoding layer in place of our text_vectorizer and embedding layer\n",
        "sentence_encoder_layer = hub.KerasLayer(\"https://tfhub.dev/google/universal-sentence-encoder/4\",\n",
        "                                        input_shape=[], # shape of inputs coming to our model \n",
        "                                        dtype=tf.string, # data type of inputs coming to the USE layer\n",
        "                                        trainable=False, # keep the pretrained weights (we'll create a feature extractor)\n",
        "                                        name=\"USE\") "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WvjQl4p7BO_A"
      },
      "source": [
        "Beautiful! Now we've got the USE as a Keras layer, we can use it in a Keras Sequential model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M_pjIvPuYltA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "141cb32c-b1a7-46e8-8b95-12c7ec4d589e"
      },
      "source": [
        "# Create model using the Sequential API\n",
        "model_6 = tf.keras.Sequential([\n",
        "  sentence_encoder_layer, # take in sentences and then encode them into an embedding\n",
        "  layers.Dense(64, activation=\"relu\"),\n",
        "  layers.Dense(1, activation=\"sigmoid\")\n",
        "], name=\"model_6_USE\")\n",
        "\n",
        "# Compile model\n",
        "model_6.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "model_6.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_6_USE\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "USE (KerasLayer)             (None, 512)               256797824 \n",
            "_________________________________________________________________\n",
            "dense_5 (Dense)              (None, 64)                32832     \n",
            "_________________________________________________________________\n",
            "dense_6 (Dense)              (None, 1)                 65        \n",
            "=================================================================\n",
            "Total params: 256,830,721\n",
            "Trainable params: 32,897\n",
            "Non-trainable params: 256,797,824\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yukgxOgCCR2Z"
      },
      "source": [
        "Notice the number of paramters in the USE layer, these are the pretrained weights its learned on various text sources (Wikipedia, web news, web question-answer forums, etc, see the [Universal Sentence Encoder paper](https://www.aclweb.org/anthology/D18-2029.pdf) for more).\n",
        "\n",
        "The trainable parameters are only in our output layers, in other words, we're keeping the USE weights frozen and using it as a feature-extractor. We could fine-tune these weights by setting `trainable=True` when creating the `hub.KerasLayer` instance.\n",
        "\n",
        "Now we've got a feature extractor model ready, let's train it and track its results to TensorBoard using our `create_tensorboard_callback()` function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uX9S0YvafybG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f4c3c8d1-b3cf-4c87-dd75-2ed786b5e5d5"
      },
      "source": [
        "# Train a classifier on top of pretrained embeddings\n",
        "model_6_history = model_6.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(SAVE_DIR, \n",
        "                                                                     \"tf_hub_sentence_encoder\")])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/tf_hub_sentence_encoder/20210923-052854\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 10s 32ms/step - loss: 0.5008 - accuracy: 0.7892 - val_loss: 0.4478 - val_accuracy: 0.7966\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 4s 19ms/step - loss: 0.4144 - accuracy: 0.8133 - val_loss: 0.4369 - val_accuracy: 0.8058\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 4s 19ms/step - loss: 0.3998 - accuracy: 0.8212 - val_loss: 0.4329 - val_accuracy: 0.8110\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 4s 18ms/step - loss: 0.3925 - accuracy: 0.8266 - val_loss: 0.4288 - val_accuracy: 0.8110\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 4s 17ms/step - loss: 0.3860 - accuracy: 0.8276 - val_loss: 0.4309 - val_accuracy: 0.8123\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KeI0kvVVDmbl"
      },
      "source": [
        "USE model trained! Let's make some predictions with it an evaluate them as we've done with our other models."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xeyNXqU-gM2p",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cbbc27f6-ee98-4633-f703-7a186fc6c932"
      },
      "source": [
        "# Make predictions with USE TF Hub model\n",
        "model_6_pred_probs = model_6.predict(val_sentences)\n",
        "model_6_pred_probs[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.14443193],\n",
              "       [0.7271502 ],\n",
              "       [0.9856655 ],\n",
              "       [0.19740924],\n",
              "       [0.73417026],\n",
              "       [0.6859663 ],\n",
              "       [0.9808888 ],\n",
              "       [0.97411025],\n",
              "       [0.91573215],\n",
              "       [0.08070081]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 85
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gbn1Z0FfgVdx",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "101ab428-2f85-42a0-b308-feb7888f8b5d"
      },
      "source": [
        "# Convert prediction probabilities to labels\n",
        "model_6_preds = tf.squeeze(tf.round(model_6_pred_probs))\n",
        "model_6_preds[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 1., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 86
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N2Ow2de3okcb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0b82286e-5589-4ddc-8982-d1dd0da78f49"
      },
      "source": [
        "# Calculate model 6 performance metrics\n",
        "model_6_results = calculate_results(val_labels, model_6_preds)\n",
        "model_6_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 81.23359580052494,\n",
              " 'f1': 0.810686575717776,\n",
              " 'precision': 0.8148798668657973,\n",
              " 'recall': 0.8123359580052494}"
            ]
          },
          "metadata": {},
          "execution_count": 87
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-BHnRHHHgp1r",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0ab489c1-4f16-428d-b391-46a95f10c044"
      },
      "source": [
        "# Compare TF Hub model to baseline\n",
        "compare_baseline_to_new_results(baseline_results, model_6_results)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Baseline accuracy: 79.27, New accuracy: 81.23, Difference: 1.97\n",
            "Baseline precision: 0.81, New precision: 0.81, Difference: 0.00\n",
            "Baseline recall: 0.79, New recall: 0.81, Difference: 0.02\n",
            "Baseline f1: 0.79, New f1: 0.81, Difference: 0.02\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LHwu4QjijYWG"
      },
      "source": [
        "### Model 7: TensorFlow Hub Pretrained Sentence Encoder 10% of the training data\n",
        "\n",
        "One of the benefits of using transfer learning methods, such as, the pretrained embeddings within the USE is the ability to get great results on a small amount of data (the USE paper even mentions this in the abstract).\n",
        "\n",
        "To put this to the test, we're going to make a small subset of the training data (10%), train a model and evaluate it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W5Sal8DpjzWm"
      },
      "source": [
        "### NOTE: Making splits like this will lead to data leakage ###\n",
        "### (some of the training examples in the validation set) ###\n",
        "\n",
        "### WRONG WAY TO MAKE SPLITS (train_df_shuffled has already been split) ### \n",
        "\n",
        "# # Create subsets of 10% of the training data\n",
        "# train_10_percent = train_df_shuffled[[\"text\", \"target\"]].sample(frac=0.1, random_state=42)\n",
        "# train_sentences_10_percent = train_10_percent[\"text\"].to_list()\n",
        "# train_labels_10_percent = train_10_percent[\"target\"].to_list()\n",
        "# len(train_sentences_10_percent), len(train_labels_10_percent)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XHgowC3GUPJH"
      },
      "source": [
        "# One kind of correct way (there are more) to make data subset\n",
        "# (split the already split train_sentences/train_labels)\n",
        "train_sentences_90_percent, train_sentences_10_percent, train_labels_90_percent, train_labels_10_percent = train_test_split(np.array(train_sentences),\n",
        "                                                                                                                            train_labels,\n",
        "                                                                                                                            test_size=0.1,\n",
        "                                                                                                                            random_state=42)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "j8jaydmiVnJP",
        "outputId": "17634293-52c5-4a51-e4ef-b5521d339bb3"
      },
      "source": [
        "# Check length of 10 percent datasets\n",
        "print(f\"Total training examples: {len(train_sentences)}\")\n",
        "print(f\"Length of 10% training examples: {len(train_sentences_10_percent)}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total training examples: 6851\n",
            "Length of 10% training examples: 686\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7E2jr7rSEYT8"
      },
      "source": [
        "Because we've selected a random subset of the training samples, the classes should be roughly balanced (as they are in the full training dataset)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V0lEpFT0k0RB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "140c4560-e0a5-472e-bdc2-47374fe7f95e"
      },
      "source": [
        "# Check the number of targets in our subset of data \n",
        "# (this should be close to the distribution of labels in the original train_labels)\n",
        "pd.Series(train_labels_10_percent).value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    415\n",
              "1    271\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 92
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ghl1qeGOEnXG"
      },
      "source": [
        "To make sure we're making an appropriate comparison between our model's ability to learn from the full training set and 10% subset, we'll clone our USE model (`model_6`) using the [`tf.keras.models.clone_model()`](https://www.tensorflow.org/api_docs/python/tf/keras/models/clone_model) method.\n",
        "\n",
        "Doing this will create the same architecture but reset the learned weights of the clone target (pretrained weights from the USE will remain but all others will be reset)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PGmxeAOBjdg2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2c321240-d2c1-4afe-e8a0-fa35fe64038e"
      },
      "source": [
        "# Clone model_6 but reset weights\n",
        "model_7 = tf.keras.models.clone_model(model_6)\n",
        "\n",
        "# Compile model\n",
        "model_7.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "# Get a summary (will be same as model_6)\n",
        "model_7.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_6_USE\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "USE (KerasLayer)             (None, 512)               256797824 \n",
            "_________________________________________________________________\n",
            "dense_5 (Dense)              (None, 64)                32832     \n",
            "_________________________________________________________________\n",
            "dense_6 (Dense)              (None, 1)                 65        \n",
            "=================================================================\n",
            "Total params: 256,830,721\n",
            "Trainable params: 32,897\n",
            "Non-trainable params: 256,797,824\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LxFkEM_aFoLK"
      },
      "source": [
        "Notice the layout of `model_7` is the same as `model_6`. Now let's train the newly created model on our 10% training data subset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LklU2maOkgUF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "65c50e4b-73f1-4012-af7a-f6ec5e2a1591"
      },
      "source": [
        "# Fit the model to 10% of the training data\n",
        "model_7_history = model_7.fit(x=train_sentences_10_percent,\n",
        "                              y=train_labels_10_percent,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(SAVE_DIR, \"10_percent_tf_hub_sentence_encoder\")])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/10_percent_tf_hub_sentence_encoder/20210923-052925\n",
            "Epoch 1/5\n",
            "22/22 [==============================] - 6s 147ms/step - loss: 0.6716 - accuracy: 0.6574 - val_loss: 0.6526 - val_accuracy: 0.6903\n",
            "Epoch 2/5\n",
            "22/22 [==============================] - 1s 47ms/step - loss: 0.5972 - accuracy: 0.8032 - val_loss: 0.5944 - val_accuracy: 0.7362\n",
            "Epoch 3/5\n",
            "22/22 [==============================] - 1s 45ms/step - loss: 0.5178 - accuracy: 0.8149 - val_loss: 0.5398 - val_accuracy: 0.7625\n",
            "Epoch 4/5\n",
            "22/22 [==============================] - 1s 31ms/step - loss: 0.4526 - accuracy: 0.8265 - val_loss: 0.5084 - val_accuracy: 0.7677\n",
            "Epoch 5/5\n",
            "22/22 [==============================] - 1s 46ms/step - loss: 0.4094 - accuracy: 0.8382 - val_loss: 0.4915 - val_accuracy: 0.7703\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Qpyqdh-F6Eh"
      },
      "source": [
        "Due to the smaller amount of training data, training happens even quicker than before.\n",
        "\n",
        "Let's evaluate our model's performance after learning on 10% of the training data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ot6MRnznlgCL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3e9b410c-117c-4292-b33e-206ebf4de1a9"
      },
      "source": [
        "# Make predictions with the model trained on 10% of the data\n",
        "model_7_pred_probs = model_7.predict(val_sentences)\n",
        "model_7_pred_probs[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.24043235],\n",
              "       [0.76837844],\n",
              "       [0.90137184],\n",
              "       [0.29067948],\n",
              "       [0.57149994],\n",
              "       [0.8356514 ],\n",
              "       [0.8062943 ],\n",
              "       [0.83358175],\n",
              "       [0.85545677],\n",
              "       [0.11749928]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 95
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vj_4aZellpRu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6ce5ceab-3cbf-4d0d-c762-f4c3893affe1"
      },
      "source": [
        "# Convert prediction probabilities to labels\n",
        "model_7_preds = tf.squeeze(tf.round(model_7_pred_probs))\n",
        "model_7_preds[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 1., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 96
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T_lTXrDblyva",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ad1a4228-9944-4e17-ecae-6c864c3a51fa"
      },
      "source": [
        "# Calculate model results\n",
        "model_7_results = calculate_results(val_labels, model_7_preds)\n",
        "model_7_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 77.03412073490814,\n",
              " 'f1': 0.7667059443150692,\n",
              " 'precision': 0.7755630249535594,\n",
              " 'recall': 0.7703412073490814}"
            ]
          },
          "metadata": {},
          "execution_count": 97
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G84ezltll6DT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3d966d5b-0c50-48e9-bb29-1e55063a06e8"
      },
      "source": [
        "# Compare to baseline\n",
        "compare_baseline_to_new_results(baseline_results, model_7_results)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Baseline accuracy: 79.27, New accuracy: 77.03, Difference: -2.23\n",
            "Baseline precision: 0.81, New precision: 0.78, Difference: -0.04\n",
            "Baseline recall: 0.79, New recall: 0.77, Difference: -0.02\n",
            "Baseline f1: 0.79, New f1: 0.77, Difference: -0.02\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iBs9V61EGh0J"
      },
      "source": [
        "## Comparing the performance of each of our models\n",
        "\n",
        "Woah. We've come a long way! From training a baseline to several deep models.\n",
        "\n",
        "Now it's time to compare our model's results.\n",
        "\n",
        "But just before we do, it's worthwhile mentioning, this type of practice is a standard deep learning workflow. Training various different models, then comparing them to see which one performed best and continuing to train it if necessary.\n",
        "\n",
        "The important thing to note is that for all of our modelling experiments we used the same training data (except for `model_7` where we used 10% of the training data).\n",
        "\n",
        "To visualize our model's performances, let's create a pandas DataFrame we our results dictionaries and then plot it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ex0NSaz7lRf-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        },
        "outputId": "421ea5ed-744d-4bc8-a0fb-b79e367f5ec3"
      },
      "source": [
        "# Combine model results into a DataFrame\n",
        "all_model_results = pd.DataFrame({\"baseline\": baseline_results,\n",
        "                                  \"simple_dense\": model_1_results,\n",
        "                                  \"lstm\": model_2_results,\n",
        "                                  \"gru\": model_3_results,\n",
        "                                  \"bidirectional\": model_4_results,\n",
        "                                  \"conv1d\": model_5_results,\n",
        "                                  \"tf_hub_sentence_encoder\": model_6_results,\n",
        "                                  \"tf_hub_10_percent_data\": model_7_results})\n",
        "all_model_results = all_model_results.transpose()\n",
        "all_model_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>accuracy</th>\n",
              "      <th>precision</th>\n",
              "      <th>recall</th>\n",
              "      <th>f1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>baseline</th>\n",
              "      <td>79.265092</td>\n",
              "      <td>0.811139</td>\n",
              "      <td>0.792651</td>\n",
              "      <td>0.786219</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>simple_dense</th>\n",
              "      <td>78.740157</td>\n",
              "      <td>0.791492</td>\n",
              "      <td>0.787402</td>\n",
              "      <td>0.784697</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>lstm</th>\n",
              "      <td>75.065617</td>\n",
              "      <td>0.751008</td>\n",
              "      <td>0.750656</td>\n",
              "      <td>0.748927</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>gru</th>\n",
              "      <td>76.771654</td>\n",
              "      <td>0.767545</td>\n",
              "      <td>0.767717</td>\n",
              "      <td>0.766793</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>bidirectional</th>\n",
              "      <td>76.640420</td>\n",
              "      <td>0.766590</td>\n",
              "      <td>0.766404</td>\n",
              "      <td>0.765121</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>conv1d</th>\n",
              "      <td>77.821522</td>\n",
              "      <td>0.780752</td>\n",
              "      <td>0.778215</td>\n",
              "      <td>0.775881</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>tf_hub_sentence_encoder</th>\n",
              "      <td>81.233596</td>\n",
              "      <td>0.814880</td>\n",
              "      <td>0.812336</td>\n",
              "      <td>0.810687</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>tf_hub_10_percent_data</th>\n",
              "      <td>77.034121</td>\n",
              "      <td>0.775563</td>\n",
              "      <td>0.770341</td>\n",
              "      <td>0.766706</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                          accuracy  precision    recall        f1\n",
              "baseline                 79.265092   0.811139  0.792651  0.786219\n",
              "simple_dense             78.740157   0.791492  0.787402  0.784697\n",
              "lstm                     75.065617   0.751008  0.750656  0.748927\n",
              "gru                      76.771654   0.767545  0.767717  0.766793\n",
              "bidirectional            76.640420   0.766590  0.766404  0.765121\n",
              "conv1d                   77.821522   0.780752  0.778215  0.775881\n",
              "tf_hub_sentence_encoder  81.233596   0.814880  0.812336  0.810687\n",
              "tf_hub_10_percent_data   77.034121   0.775563  0.770341  0.766706"
            ]
          },
          "metadata": {},
          "execution_count": 99
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v-s2DSLpmM1F"
      },
      "source": [
        "# Reduce the accuracy to same scale as other metrics\n",
        "all_model_results[\"accuracy\"] = all_model_results[\"accuracy\"]/100"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wp69bR8umD5g",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 546
        },
        "outputId": "cb7ae0e4-2a9c-4ef3-d23a-a9a93a6a992d"
      },
      "source": [
        "# Plot and compare all of the model results\n",
        "all_model_results.plot(kind=\"bar\", figsize=(10, 7)).legend(bbox_to_anchor=(1.0, 1.0));"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqkAAAIRCAYAAABpvyTfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde5xWZb3+8esaDiIKJDgeEUHkNCKKIppaVB7SSjwrmGnuip/u1DLdZVlqdNpa2m97+O2NZy3cpm5LPJRZKbazUkBRQVBUQlFxVASUFJDv749njT4MAzPoMPc9rM/79XpezFrP4pmL58XMXLPWfd/LESEAAAAgJzWpAwAAAACNUVIBAACQHUoqAAAAskNJBQAAQHYoqQAAAMhOx1SfePPNN4++ffum+vQAAAAtNnXq1FcjojZ1jjJJVlL79u2rKVOmpPr0AAAALWb7H6kzlA2X+wEAAJAdSioAAACyQ0kFAABAdpKNSQUAAGjPpk6dukXHjh2vkjRUnPhbVyslPbFixYov77777q80dQAlFQAA4APo2LHjVVtttdWQ2trahTU1NZE6T3uycuVK19fX17388stXSRrd1DG0fgAAgA9maG1t7WIK6rqrqamJ2traRaqchW76mDbMAwAAsCGpoaB+cMV7t8YuSkkFAABAdhiTCgAA0Ar6nn3X7q35enP//bNTW/P12hvOpAIAAGCtli9f3uafk5IKAADQju2///79d9pppyE77rjjTj/72c82l6Rbb721e11d3ZBBgwbVffSjHx0oSYsWLao56qij+g4cOLBu4MCBddddd91HJKlr167DG17r2muv3ezII4/sK0lHHnlk3+OOO67PsGHDBp9yyim977vvvq677rrr4CFDhtQNHz588PTp0zeSpBUrVmjcuHG9BwwYsNPAgQPrfvSjH20xadKkbvvvv3//htf99a9/3f2AAw7or3XA5X4AAIB2bOLEiXO33HLLd998800PHz687thjj33j1FNP7Xv//ffPGjx48LIFCxZ0kKSzzz576+7du7/71FNPzZSk+vr6Ds299ksvvdR52rRpszp27KjXX3+95uGHH57VqVMn/eY3v+n2zW9+s/c999zzzEUXXVQ7b968zjNnzpzRqVMnLViwoENtbe27X/va1/q8+OKLHbfZZpsV11xzTa+TTjrp1XX5d1FSAQAA2rELLrhgy7vuuusjkvTyyy93uuSSS2pHjhy5ZPDgwcskacstt3xXkh544IHuN91007MNf6+2tvbd5l77iCOOWNixY6Uuvv766x2OPfbYfnPnzu1iO5YvX25J+tOf/tT95JNPru/UqZOqP98xxxzz2pVXXtnzq1/96mvTpk3b9LbbbntuXf5dlFQAAIB26s477+w2efLkblOmTJnVrVu3lSNHjhw0fPjwpbNnz+7S0tew/d7H//znP1393Kabbrqy4eNvfetb244aNWrJvffe+8zs2bM7f+pTnxq0ttc95ZRTXvvsZz+7Y5cuXeKQQw5Z2FBiW4oxqQAAAO3UG2+80aFHjx7vduvWbeUjjzzSZfr06Zu8/fbbNQ899FC3WbNmdZakhsv9o0aNWvzzn/98i4a/23C5v1evXsunTZvW5d1339Xtt9++2Zo+1+LFizv07t17mSRNmDBh84b9++233+IJEyZs3jC5quHz9e3bd/mWW265/KKLLtp63Lhx63SpX+JMKgAAQKtIsWTUkUceueiKK66o3WGHHXbaYYcd3t5ll13e2mKLLVZccsklcw8//PAdV65cqV69ei1/8MEHn/7JT37y0kknndRnwIABO9XU1MR3vvOdF0888cQ3vv/9788/9NBDd+zZs+eKXXbZZelbb73V5EnMb33rWy9/+ctf7nfBBRdsc8ABB7zRsP+MM86of+qppzYaPHjwTh07dowTTzyx/jvf+U69JI0ZM+a1yy+/vONuu+329rr+2xyR5kYJI0aMiClTpiT53AAAAOvC9tSIGFG9b/r06XN32WWXdT5DWCYnnHBCn+HDhy8944wzmnyfpk+fvvkuu+zSt6nnOJMKAMD6dn6PZp5f1DY5gDa00047Ddl4441XTpgw4fkP8vc3/JLa3DcGiW8OAIAPrO/ZdzV7zNxmprDsfP3Ozb7GzT9Z0ewxQ2Y92ewxQFuZMWPGh/oPycQpAAAAZKdFJdX2QbZn255j++wmnu9j+z7bj9h+zPZnWj8qAAAAyqLZkmq7g6TLJR0sqU7SWNt1jQ77rqSbI2K4pDGS/l9rBwUAAEB5tORM6khJcyLi2YhYJukmSYc2OiYkdS8+7iHpxdaLCAAAgLJpycSpbSVVz8p6QdKejY45X9LvbZ8maRNJ+zf1QrbHSRonSX369FnXrAAAAPk6v8furft6i9p83VVJeuCBB7pec801va677romZ+XPnTu308knn7zd7373u2eber61tNbEqbGSrouI3pI+I+kXtld77Yi4IiJGRMSI2traVvrUAAAAWJMVK5pfGaLaxz/+8aVrKqhS5U5S67ugSi0rqfMlbVe13bvYV+1Lkm6WpIj4q6QukjYXAAAA1pvZs2d37tev306jR4/ut8MOO+x00EEH7bBkyZKabbfddudTTjll27q6uiHXXHPNZrfddlv3XXfddXBdXd2Qgw8+eIdFixbVSNLkyZO7Dh8+fPCgQYPqdt555yELFy6sufPOO7t98pOf3FGS7rrrrk0HDx5cN3jw4LohQ4bULVy4sGb27NmdBwwYsJMkLV261EcddVTfgQMH1g0ZMqTujjvu6CZJl1xySa8DDzyw/8c+9rEB22+//dCTTz6597r+21pyuf9hSQNs91OlnI6RdFyjY+ZJ2k/SdbaHqFJS69c1zAfR3Pp0za1NJ7VsfbrHT3y8pZEAAADazNy5c7tMmDBh7oEHHvjW0Ucf3fenP/1prST16tVrxcyZM5986aWXOh5yyCH9H3jggae6d+++8pxzztnqBz/4wZY//OEPX/785z/ff+LEic+MGjVq6euvv16z6aabrqx+7YsuumirSy655B8HHnjgW4sWLarp2rXryldeeeW95y+44IItbOupp56a+cgjj3T5zGc+M+CZZ555QpJmzpzZdfr06TM33njjlTvuuOPQs846a8GOO+64vKX/rmZLakSssH2qpHskdZB0TUTMsD1e0pSImCTpTElX2j5DlUlUX4xU91tFy3CTAwAANghbbbXVsgMPPPAtSfrCF77w2iWXXLKFJJ1wwgkLJen+++/f5JlnnukycuTIwZK0fPly77777m8+9thjXbbYYovlo0aNWipJPXv2XNn4tffaa683zzrrrO2OOeaY18eOHbuwf//+qxzz4IMPbnraaae9IknDhw9/e5tttln2+OOPd5Gkfffdd3GvXr3elaQdd9zx7WeeeWajVi2pkhQRd0u6u9G+c6s+nilpn5Z+UgAAALQO201ud+vWbaUkRYT23XffxXfcccdz1cc99NBDGzf32j/+8Y9fPuywwxbdfvvtPT72sY8Nvuuuu57u2rXramW2KZ07d37vhGWHDh1i+fLlXtvxjW34t0VtJU8OHrLW53O6FV1r3KJPan4YBEMgAABI76WXXur8hz/8YZP999//rYkTJ/bce++935w5c2bXhuc/8YlPvHXmmWf2eeKJJzYaOnToO4sXL66ZO3dup2HDhr39yiuvdJo8eXLXUaNGLV24cOFql/tnzJix0ciRI/85cuTIf06dOrXrE0880WXkyJFLG57fZ5993vzlL3/Zc/To0Usee+yxjV566aXOw4YNe/vvf/97V31IlFR8YM0Vdymv8g4AwHqVaMmovn37vn3ppZduMW7cuK4DBgx4+6yzzqq/6qqrtmh4fptttlkxYcKEuWPGjNlh2bJllqTzzjtv/rBhw96ZOHHiM6effnqft99+u6ZLly4rH3jggaeqX/vCCy/c4sEHH+xuOwYNGvTPo446atG8efM6NTz/zW9+85UTTjhh+4EDB9Z16NBBEyZMmLvxxhu3ypBPSioAAEA71rFjR91+++2rXMqfP3/+Kpc7R48evWT06NGrnTkaNWrU0unTp8+q3ve5z31uyec+97klknT99devthTVoEGDlj399NMzJKlr165x6623zm18zOmnn/6apNcatu+777456/avoqQCAFoTkzIBtBJKKgCgRRjvDuSn+qzmhoaSCgDICuPdS4Kz7mhGa90WFQAAAGg1nEkFAACtjjtC4sOipAIl1+wPkn//bLOvwQ8SAEBro6QCWLuWjBvr16fZQ9rbOMPmzwId1+xr7NyC94XyDnw4OX1v2fn6nXdvzdd7/MTHk6y7eskll/SaMmXKJjfccMO8b3zjG9tsuumm744fP35BW+egpAJAQu3pbnYA8rZy5UpFhDp06JA6Sqtg4hQAAEA7NXv27M59+/Ydevjhh/cdOHDgTt/85je3Hjp06JCBAwfWnXHGGds0HHfZZZf1GjhwYN2gQYPqDjvssH6SdOONN/YYNmzY4CFDhtTtvffeA59//vmsTl5mFQYAAADrZt68eRtdffXVzy1atOj1W265ZbPHHnvsyYjQ/vvvv+Nvf/vbTWtra1f87Gc/2/qvf/3rrK233nrFggULOkjSAQcc8OaYMWNm1dTU6OKLL958/PjxW1155ZUvpP73NKCkAgAAtGNbb731sv322++tcePG9X7ggQe619XV1UnS0qVLa2bNmtVl2rRpNYcccsjCrbfeeoUkbbnllu9K0nPPPdf5sMMO611fX99p2bJlNdttt907Kf8djXG5HwAAoB3r2rXrSkmKCH39619/adasWTNnzZo1c968eU+cccYZr67p75166ql9/vVf//WVp556auZll132j3feeSerXphVGAAAAHwwBx988OJf/OIXmy9atKhGkp577rlO8+fP7/jpT3968R133LHZyy+/3EGSGi73L1mypEOfPn2WS9J1113XK13ypnG5H6XRsvuOr31ZIZYUAgCsSaoloxocccQRi2fMmNFljz32GCxVzrBOnDjxuREjRrx95plnvvSxj31scE1NTQwdOnTp//zP/8w955xzXhw7dmz/Hj16rNh3332XzJs3b6OU+RujpAKtLKc1+wAAG7ZBgwYte/rpp2c0bH/ve9975Xvf+94rjY877bTTXjvttNNeq953/PHHv3H88ce/0fjY008//TVJr0nSxRdf/OJ6iN0iXO4HAABAdiipAAAAyA4lFQAAANmhpAIAACA7lFQAAABkh5IKAACA7LAEFQAAQCt4cvCQ3Vvz9YbMerLZdVd/+MMfbnHNNdfUDhgw4O0FCxZ0mjlzZtezzz57/vjx4xe0ZpYUKKkAAADt1NVXX137hz/84akuXbrEnDlzOt96662bpc7UWrjcDwAA0A4dd9xxfV544YWNDj744AFXXXVVz1GjRi3t1KlTpM7VWjiTCgAA0A7deOON8yZPntxj8uTJT2299dYrUudpbZxJBQAAQHYoqQAAAMgOJRUAAADZYUwqAABAK2jJklHry7x58zrusccedW+99VYH2zFhwoQtn3zyySd69uy5MlWmD4uSCgAA0E7Nnz//8YaPFyxY8FjKLK2Ny/0AAADIDiUVAAAA2WlRSbV9kO3ZtufYPruJ539u+9Hi8ZTtN1o/KgAAQFZWrly50qlDtFfFe7fGMbPNllTbHSRdLulgSXWSxtquqz4mIs6IiF0jYldJl0q67UOlBgAAyN8T9fX1PSiq627lypWur6/vIemJNR3TkolTIyXNiYhnJcn2TZIOlTRzDcePlXTeOmYFAABoV1asWPHll19++aqXX355qBhCua5WSnpixYoVX17TAS0pqdtKer5q+wVJezZ1oO3tJfWT9Kd1CAkAANDu7L777q9IGp06x4aqtVv/GEm3RsS7TT1pe5ztKban1NfXt/KnBgAAwIaiJSV1vqTtqrZ7F/uaMkbSf6/phSLiiogYEREjamtrW54SAAAApdKSkvqwpAG2+9nurEoRndT4INuDJW0m6a+tGxEAAABl02xJjYgVkk6VdI+kJyXdHBEzbI+3XT0OY4ykmyIi1k9UAAAAlEWLbosaEXdLurvRvnMbbZ/ferEAAABQZiyXAAAAgOxQUgEAAJAdSioAAACyQ0kFAABAdiipAAAAyA4lFQAAANmhpAIAACA7lFQAAABkh5IKAACA7FBSAQAAkB1KKgAAALJDSQUAAEB2KKkAAADIDiUVAAAA2aGkAgAAIDuUVAAAAGSHkgoAAIDsUFIBAACQHUoqAAAAskNJBQAAQHYoqQAAAMgOJRUAAADZoaQCAAAgO5RUAAAAZIeSCgAAgOxQUgEAAJAdSioAAACyQ0kFAABAdiipAAAAyA4lFQAAANmhpAIAACA7lFQAAABkh5IKAACA7FBSAQAAkB1KKgAAALJDSQUAAEB2WlRSbR9ke7btObbPXsMxx9ieaXuG7RtbNyYAAADKpGNzB9juIOlySQdIekHSw7YnRcTMqmMGSPq2pH0iYqHtLdZXYAAAAGz4WnImdaSkORHxbEQsk3STpEMbHfMVSZdHxEJJiohXWjcmAAAAyqQlJXVbSc9Xbb9Q7Ks2UNJA23+x/TfbBzX1QrbH2Z5ie0p9ff0HSwwAAIANXmtNnOooaYCkT0gaK+lK2x9pfFBEXBERIyJiRG1tbSt9agAAAGxoWlJS50varmq7d7Gv2guSJkXE8oh4TtJTqpRWAAAAYJ21pKQ+LGmA7X62O0saI2lSo2N+o8pZVNneXJXL/8+2Yk4AAACUSLMlNSJWSDpV0j2SnpR0c0TMsD3e9ujisHskvWZ7pqT7JP1bRLy2vkIDAABgw9bsElSSFBF3S7q70b5zqz4OSd8oHgAAAMCHwh2nAAAAkB1KKgAAALJDSQUAAEB2KKkAAADIDiUVAAAA2aGkAgAAIDuUVAAAAGSHkgoAAIDsUFIBAACQHUoqAAAAskNJBQAAQHYoqQAAAMgOJRUAAADZoaQCAAAgO5RUAAAAZIeSCgAAgOxQUgEAAJAdSioAAACyQ0kFAABAdiipAAAAyA4lFQAAANmhpAIAACA7lFQAAABkh5IKAACA7FBSAQAAkB1KKgAAALJDSQUAAEB2KKkAAADIDiUVAAAA2aGkAgAAIDuUVAAAAGSHkgoAAIDsUFIBAACQHUoqAAAAskNJBQAAQHZaVFJtH2R7tu05ts9u4vkv2q63/Wjx+HLrRwUAAEBZdGzuANsdJF0u6QBJL0h62PakiJjZ6NBfRcSp6yEjAAAASqYlZ1JHSpoTEc9GxDJJN0k6dP3GAgAAQJm1pKRuK+n5qu0Xin2NHWn7Mdu32t6uVdIBAACglFpr4tQdkvpGxDBJ90q6vqmDbI+zPcX2lPr6+lb61AAAANjQtKSkzpdUfWa0d7HvPRHxWkS8U2xeJWn3pl4oIq6IiBERMaK2tvaD5AUAAEAJtKSkPixpgO1+tjtLGiNpUvUBtreu2hwt6cnWiwgAAICyaXZ2f0SssH2qpHskdZB0TUTMsD1e0pSImCTpdNujJa2Q9LqkL67HzAAAANjANVtSJSki7pZ0d6N951Z9/G1J327daAAAACgr7jgFAACA7FBSAQAAkB1KKgAAALJDSQUAAEB2KKkAAADIDiUVAAAA2aGkAgAAIDuUVAAAAGSHkgoAAIDsUFIBAACQHUoqAAAAskNJBQAAQHYoqQAAAMgOJRUAAADZoaQCAAAgO5RUAAAAZIeSCgAAgOxQUgEAAJAdSioAAACyQ0kFAABAdiipAAAAyA4lFQAAANmhpAIAACA7lFQAAABkh5IKAACA7FBSAQAAkB1KKgAAALJDSQUAAEB2KKkAAADIDiUVAAAA2aGkAgAAIDuUVAAAAGSHkgoAAIDsUFIBAACQHUoqAAAAskNJBQAAQHZaVFJtH2R7tu05ts9ey3FH2g7bI1ovIgAAAMqm2ZJqu4OkyyUdLKlO0ljbdU0c103S1yT9vbVDAgAAoFxaciZ1pKQ5EfFsRCyTdJOkQ5s47geSLpD0divmAwAAQAm1pKRuK+n5qu0Xin3vsb2bpO0i4q61vZDtcban2J5SX1+/zmEBAABQDh964pTtGkkXSzqzuWMj4oqIGBERI2praz/spwYAAMAGqiUldb6k7aq2exf7GnSTNFTS/bbnStpL0iQmTwEAAOCDaklJfVjSANv9bHeWNEbSpIYnI2JRRGweEX0joq+kv0kaHRFT1ktiAAAAbPCaLakRsULSqZLukfSkpJsjYobt8bZHr++AAAAAKJ+OLTkoIu6WdHejfeeu4dhPfPhYAAAAKDPuOAUAAIDsUFIBAACQHUoqAAAAskNJBQAAQHYoqQAAAMgOJRUAAADZoaQCAAAgO5RUAAAAZIeSCgAAgOxQUgEAAJAdSioAAACyQ0kFAABAdiipAAAAyA4lFQAAANmhpAIAACA7lFQAAABkh5IKAACA7FBSAQAAkB1KKgAAALJDSQUAAEB2KKkAAADIDiUVAAAA2aGkAgAAIDuUVAAAAGSHkgoAAIDsUFIBAACQHUoqAAAAskNJBQAAQHYoqQAAAMgOJRUAAADZoaQCAAAgO5RUAAAAZIeSCgAAgOxQUgEAAJAdSioAAACy06KSavsg27Ntz7F9dhPPn2z7cduP2v5f23WtHxUAAABl0WxJtd1B0uWSDpZUJ2lsEyX0xojYOSJ2lXShpItbPSkAAABKoyVnUkdKmhMRz0bEMkk3STq0+oCIWFy1uYmkaL2IAAAAKJuOLThmW0nPV22/IGnPxgfZ/qqkb0jqLOlTrZIOAAAApdRqE6ci4vKI6C/pW5K+29QxtsfZnmJ7Sn19fWt9agAAAGxgWlJS50varmq7d7FvTW6SdFhTT0TEFRExIiJG1NbWtjwlAAAASqUlJfVhSQNs97PdWdIYSZOqD7A9oGrzs5Kebr2IAAAAKJtmx6RGxArbp0q6R1IHSddExAzb4yVNiYhJkk61vb+k5ZIWSjpxfYYGAADAhq0lE6cUEXdLurvRvnOrPv5aK+cCAABAiXHHKQAAAGSHkgoAAIDsUFIBAACQHUoqAAAAskNJBQAAQHYoqQAAAMgOJRUAAADZoaQCAAAgO5RUAAAAZIeSCgAAgOxQUgEAAJAdSioAAACyQ0kFAABAdiipAAAAyA4lFQAAANmhpAIAACA7lFQAAABkh5IKAACA7FBSAQAAkB1KKgAAALJDSQUAAEB2KKkAAADIDiUVAAAA2aGkAgAAIDuUVAAAAGSHkgoAAIDsUFIBAACQHUoqAAAAskNJBQAAQHYoqQAAAMgOJRUAAADZoaQCAAAgO5RUAAAAZIeSCgAAgOxQUgEAAJAdSioAAACy06KSavsg27Ntz7F9dhPPf8P2TNuP2f6j7e1bPyoAAADKotmSaruDpMslHSypTtJY23WNDntE0oiIGCbpVkkXtnZQAAAAlEdLzqSOlDQnIp6NiGWSbpJ0aPUBEXFfRCwtNv8mqXfrxgQAAECZtKSkbivp+artF4p9a/IlSb9t6gnb42xPsT2lvr6+5SkBAABQKq06ccr28ZJGSPppU89HxBURMSIiRtTW1rbmpwYAAMAGpGMLjpkvabuq7d7FvlXY3l/SOZJGRcQ7rRMPAAAAZdSSM6kPSxpgu5/tzpLGSJpUfYDt4ZImSBodEa+0fkwAAACUSbMlNSJWSDpV0j2SnpR0c0TMsD3e9ujisJ9K2lTSLbYftT1pDS8HAAAANKsll/sVEXdLurvRvnOrPt6/lXMBAACgxLjjFAAAALJDSQUAAEB2KKkAAADIDiUVAAAA2aGkAgAAIDuUVAAAAGSHkgoAAIDsUFIBAACQHUoqAAAAskNJBQAAQHYoqQAAAMgOJRUAAADZoaQCAAAgO5RUAAAAZIeSCgAAgOxQUgEAAJAdSioAAACyQ0kFAABAdiipAAAAyA4lFQAAANmhpAIAACA7lFQAAABkh5IKAACA7FBSAQAAkB1KKgAAALJDSQUAAEB2KKkAAADIDiUVAAAA2aGkAgAAIDuUVAAAAGSHkgoAAIDsUFIBAACQHUoqAAAAskNJBQAAQHYoqQAAAMhOi0qq7YNsz7Y9x/bZTTz/cdvTbK+wfVTrxwQAAECZNFtSbXeQdLmkgyXVSRpru67RYfMkfVHSja0dEAAAAOXTsQXHjJQ0JyKelSTbN0k6VNLMhgMiYm7x3Mr1kBEAAAAl05LL/dtKer5q+4Vi3zqzPc72FNtT6uvrP8hLAAAAoATadOJURFwRESMiYkRtbW1bfmoAAAC0Iy0pqfMlbVe13bvYBwAAAKwXLSmpD0saYLuf7c6SxkiatH5jAQAAoMyaLakRsULSqZLukfSkpJsjYobt8bZHS5LtPWy/IOloSRNsz1ifoQEAALBha8nsfkXE3ZLubrTv3KqPH1ZlGAAAAADwoXHHKQAAAGSHkgoAAIDsUFIBAACQHUoqAAAAskNJBQAAQHYoqQAAAMgOJRUAAADZoaQCAAAgO5RUAAAAZIeSCgAAgOxQUgEAAJAdSioAAACyQ0kFAABAdiipAAAAyA4lFQAAANmhpAIAACA7lFQAAABkh5IKAACA7FBSAQAAkB1KKgAAALJDSQUAAEB2KKkAAADIDiUVAAAA2aGkAgAAIDuUVAAAAGSHkgoAAIDsUFIBAACQHUoqAAAAskNJBQAAQHYoqQAAAMgOJRUAAADZoaQCAAAgO5RUAAAAZIeSCgAAgOxQUgEAAJAdSioAAACy06KSavsg27Ntz7F9dhPPb2T7V8Xzf7fdt7WDAgAAoDyaLam2O0i6XNLBkuokjbVd1+iwL0laGBE7Svq5pAtaOygAAADKoyVnUkdKmhMRz0bEMkk3STq00TGHSrq++PhWSfvZduvFBAAAQJk4ItZ+gH2UpIMi4svF9hck7RkRp1Yd80RxzAvF9jPFMa82eq1xksYVm4MkzW6tf8iHtLmkV5s9qnx4X1bHe9I03pem8b40jfdldbwnTcvpfdk+ImpThyiTjm35ySLiCklXtOXnbAnbUyJiROocueF9WR3vSdN4X5rG+9I03pfV8Z40jfel3FpyuX++pO2qtnsX+5o8xnZHST0kvdYaAQEAAFA+LSmpD0saYLuf7aJyw4AAABsMSURBVM6Sxkia1OiYSZJOLD4+StKforlxBAAAAMAaNHu5PyJW2D5V0j2SOki6JiJm2B4vaUpETJJ0taRf2J4j6XVVimx7kt0QhEzwvqyO96RpvC9N431pGu/L6nhPmsb7UmLNTpwCAAAA2hp3nAIAAEB2KKkAAADIDiUVAAAA2aGkAgAAIDttuph/bmzvK2lARFxru1bSphHxXOpcKdnuKulMSX0i4iu2B0gaFBF3Jo6WjO0Rks6RtL0qXzOWFBExLGkwZMV2z7U9HxGvt1WWXNi+VNIaZ+dGxOltGCcrtjtI+kNEfDJ1ltwUP3d+IqlOUpeG/RGxQ7JQSKK0JdX2eZJGqHJ71msldZL0S0n7pMyVgWslTZX00WJ7vqRbJJW2pEqaKOnfJD0uaWXiLNmwvUTvF5DOqnwNvRUR3dOlSmqqKu+Hm3guJJXxB+yU4s99VCkcvyq2j5Y0M0miTETEu7ZX2u4REYtS58nMtZLOk/RzSZ+UdJK48ltKpS2pkg6XNFzSNEmKiBdtd0sbKQv9I+JY22MlKSKW2m7qh26Z1BfrAaNKRLz39VL8HzlU0l7pEqUVEf1SZ8hNRFwvSbZPkbRvRKwotv9L0p9TZsvEm5Iet32vpLcadpb5DHNh44j4o21HxD8knW97qqRzUwdD2ypzSV0WEWE7JMn2JqkDZWKZ7Y1VnCGz3V/SO2kjJXee7ask/VFV70VE3JYuUl6KO8z9prhCcXbqPKnZ3kzSAK16qfKBdImS20xSd1Vu9iJJmxb7yu624oFVvWO7RtLTxc2E5qvyfwYlU+aSerPtCZI+Yvsrkv5F0pWJM+XgPEm/k7Sd7YmqXKb7YtJE6Z0kabAql7MbLveHSv7DxfYRVZs1qgyfeTtRnGzY/rKkr0nqLelRVc4u/1XSp1LmSuzfJT1i+z5VhkN8XNL5SRNlICKuL04K9ImI2anzZORrkrpKOl3SD1S55H9C0kRIotR3nLJ9gKQDVfmmeU9E3Js4UhZs91LlB6sl/S0iXk0cKSnbsyNiUOocubF9bdXmCklzJV0ZEa+kSZQH249L2kOVr51dbQ+W9OOIOKKZv7pBs72VpD2Lzb9HxMsp8+TA9iGSfiapc0T0s72rpPERMTpxtKRsHx0RtzS3Dxu+UpdUrM72PpIejYi3bB8vaTdJ/1GMCyqlooz9NCJKPdGjWjEz+fSI+HnqLLmx/XBE7GH7UUl7RsQ7tmdExE6ps+XE9uCImJU6R0rFOMtPSbo/IoYX+56IiKFpk6Vle1pE7NbcPmz4Snu5v7hUeYGkLVQ5Y9iwrFBZZyY3+E9Ju9jeRdI3JF0t6QZJo5KmSmsvSY/afk6VMamlX4KqmJk8VpXZt1jVC7Y/Iuk3ku61vVBSaX/JW4vfS+qTOkRiyyNiUaO5qaVdQcT2wZI+I2lb25dUPdVdlas1KJnSllRJF0o6JCKeTB0kMyuKCWWHSro8Iq62/aXUoRI7KHWATP3F9mWqLCtUPTN5WrpI6UXE4cWH5xdjMHuoMs67dBoVjVWekvSRtsySqRm2j5PUoVgb9HRJDybOlNKLqixbNlqVJd0aLJF0RpJESKq0l/tt/yUiyr4m6mpsT1blB+pJqkxueEXS9IjYOWmwhGz/IiK+0Ny+sikKmPT+WqkNZ5jLPEFI0nvDIbZU1YmAiJiXLlEaxVq6Z6rpFUIuiojN2zhSVoqbp5yjqrkRkn4QEaWegGi7U0QsT50D6ZW5pP6HpK1UuSTHskKFYnLDcZIejog/2+4j6RMRcUPiaMk0HgtVFJDHI6IuYazkbJ+pVRevD0mLJU2JiEeTBUvM9mmqrJKxQFWrQZRxeIjtP0n6bkSsdnbQ9nOsLYumcMcpNChzSb22id0REf/S5mGQJdvflvQdSRtLWtqwW9IySVdExLdTZcuB7RtVWXZqkirvy+ckPSapr6RbIuLCdOnSsT1HlQlTr6XOklpxq9i3I2JpsweXiO07tPbbxZZ9dv//6v07Th2i4o5TEcFi/iVT2pKKpjGhbHW2f1L2QtoU2w9I+kxEvFlsbyrpLlXG8E4t65nmYhjEAQ13V8J731fuioiy3xhEkmS7YSLqEapc0ftlsT1W0oKIKPX4S9tTI2J32483DDVr2Jc6G9pW6SZO2f5mRFxo+1I18Zsst6NjQlkT7rS9CctyrWYLrTrWcLmkLSPin7bLXEaelXS/7bu06lCii9NFSu4QST8vfrH5laTflbnER8RkSbJ9UUSMqHrqDttTEsXKCXecgqQSllRJDeWLbwRNW0BBXU31slxnSrpKLMslSRMl/d327cX2IZJuLG4xXOY1ZecVj87Fo/Qi4iTbnSQdrMrZwstt3xsRX04cLbVNbO8QEc9Kku1+krhF9+p3nPqUpBOTJkISXO7HKphQtrqGiVO2z5U0v1iWi4WlJdkeocqtcyXpLxHBL3+FYviDGoZDoDJrW5XhICdJ+jiz+32QpCtUOftuSdtLGhcRv08aDMhE6UoqA9bXjgllq2NZLqwL20Ml/UJSz2LXq5JOiIgZ6VKlVSzSfqykT0i6X9LNkn5f5kv+DWxvJGlwsTmrzON2+fmMxspYUtd6ibZhrBDQgGW5sC5sPyjpnIi4r9j+hKQfR8TeSYMlZPu/VRmL+tsyl7DGijPLp6jyy69UKfATyrpGKBPK0FjpSmo12xtL6hMRs1NnyYXtgaqMwdwyIobaHiZpdET8MHE0oF2wPT0idmluH2D7KkmdJF1f7PqCpHfLPlbX9pRGE8qa3IcNX03qAKnYPkTSoypuV2h7V9uT0qbKwpWSvq3KTG1FxGOSxiRNlIjtJbYXN/FYYntx6nzI1rO2v2e7b/H4ripjDkvL9hG2n7a9iK+hVewRESdGxJ+Kx0mS9kgdKgOb2H5v4X4mlJVXGWf3Nzhf0khVLq8oIh4tvhDKrmtEPGS7el8px41FRLfUGdAu/Yuk70tqmGz452JfmbG0XdPetd0/Ip6RpKKYvZs4Uw7OUGUZt1UmlKWNhBTKXFKXR8SiRmWsvGMf3veq7f4q3gvbR0l6KW0koP2IiIWqLJ2D97G0XdP+TdJ9jcrYSWkjpRcRvytujdrkhDLbB0TEvWnSoS2Vdkyq7asl/VHS2ZKOVOWHSqeIODlpsMSK3+SvkLS3pIWSnpN0fETMTZkLyJ3t/xsRX1/TDOUyz0xmabs1K2b3Dyo2ZzOxrHksAVgeZS6pXSWdI+lAVX6DvUfSDyLi7aTBMlEsyF4TEUtSZwHaA9u7R8TUNa0gUuaVQ1jarmm2vyppYkS8UWxvJmlsRPy/tMnyZvuRiBieOgfWv9KW1Gq2O0jaJCJKO5Df9jfW9nzJb+kItJjtr0XEfzS3D7D9aETs2mgfBawZnEktjzLP7r/RdvfijOHjkmba/rfUuRLqVjxGqLJu37bF42RV7lUPoGWaun3jF9s6RE5s97b9a9uvFI//sd07da4MdHDVxIjihAm30gUKZZ44VRcRi21/XtJvVRmbOlXST9PGSiMivi9Jth+QtFvDZX7b50u6K2E0oF2wPVaVmz70a7ScXTdJr6dJlY1rJd0o6ehi+/hi3wHJEuXhd5J+ZXtCsf1/in2lZnujxmNzG+2b2/apkEKZS2qn4m4fh0m6LCKW22bsg7SlpGVV28uKfQDW7kFVVsLYXNJFVfuXSHosSaJ81EZE9bjU62x/PVmafHxLlWJ6SrF9r6Sr0sXJxl+1+hW89/ZFxBFtnghJlLmkTlDlt7Hpkh6wvb2k0o5JrXKDpIds/7rYPkzSdeniAO1DRPxD0j+KqzMvNkzCLO5s11vlPvvzmu3jJf13sT1W0msJ82QhIlaqcoe//0ydJQfFLai3lbSx7eGqTGqWpO6SuiYLhmSYOFXFdseIKOXC9dVs7ybpY8XmAxHxSNVzmxXrQAJogu0pkvaOiGXFdmdJf4mI0t5JqDgJcKmkj6qyPNeDkk6LiOeTBkvM9j6q3Fhme1VOGlmVVQ92WNvf21DZPlGV8dsjJE2pemqJpOtYsqx8Sl1SbX9W0k6SujTsi4jx6RLlj1mVwNqtYcb29IjYJVWm1GxfL+nrDb/g2u4p6WcsQeVZqtxdaaqq7jQVEaU+y2z7yIj4n9Q5kF5pL/fb/i9VLh98UpUxQEdJeihpqPbBzR8ClFq97dERMUmSbB8q6dXEmVIbVn0FJiJeLy7nlt2iiPht6hAZutP2cZL6qqqncBKpfEpbUlW5HDfM9mMR8X3bF6kyyx9rV95T70DLnCxpou3LVfl6eUHSCWkjJVdTPVSoOJNa5p8/De6z/VNJt2nVO3FNSxcpC7dLWqTKGWbuwFViZf4m8c/iz6W2t1FlEP/WCfMA2ABExDOS9rK9abH9ZuJIObhI0l9t31JsHy3pRwnz5GLP4s8RVftC0qcSZMlJ74g4KHUIpFfmknqn7Y9IulCV39Yklv5oCS73A2the0tJP5a0TUQcbLtO0kcj4urE0ZKJiBuKCWUN5euIiJiZMlMOIuKTqTNk6kHbO0fE46mDIK3STpwqloU5RZVZ7CHpz5L+s2HZmDKzva+kARFxre1aSZtGxHPFcz0jouwLkwNrZPu3qixUf05E7GK7o6RHImLnxNGQGX6haZrtmZJ2lPScKpf7G1Y9GJY0GNpcmUvqzaosa/HLYtdxknpExDHpUqVn+zxVLj0NioiBxVCIWyJin8TRgHbB9sMRsUf1PdibmvEP8AtN04oly1ZTrEWMEqlJHSChoRHxpYi4r3h8RdLQ1KEycLik0ZLekqSIeFGV2zoCaJm3bPdSMcnQ9l6qTAIBGts8Im6WtFKSinW63137X9nwFWV0O0mfKj5eqnL3ldIq85jUabb3ioi/SZLtPbXq4sFltSwiouEWsbY3SR0IaGe+IWmSpP62/yKpVpUl7oDG+IWmCdVX9FQ509xJlaueXNErmdKVVNuPq/INoZMqg7PnFdvbS5qVMlsmbrY9QdJHbH9F0r9IujJxJqBdsN1B0qjiMUiVsXSzI2J50mDIFb/QNO1wScMlTZMqV/Rsc0WvhEo3JnVNY10aMOZFsn2ApANV+QF7T0TcmzgS0G7YfigiRqbOgfahGIfa5C80tg8o4/ffhq+hhjscFlf0/srEqfIpXUkFgPXJ9s9VuVLzKxVjuyUWaMe6K+ttqG2fJWmApAMk/USVK3o3RsSlSYOhzVFSIUmyvURN302qYemP7m0cCWiXbN/XxO6IiLIv0I51VL1CRNlwRQ8SJRUAgCyV+ExqP0kvNaxbXqxrvmVEzE0aDG2udBOn0Dzbu0naV5Uzq/8bEY8kjgRkz/bxEfFL299o6vmIuLitMwHt1C2S9q7afrfYt0eaOEiFdcewCtvnSrpeUi9Jm0u6zvZ306YC2oWG5dq6reEBrKu5qQMk0jEiljVsFB93TpgHiXC5H6uwPVvSLo0uszwaEYPSJgOADYvtrpLOlNQnIr5ie4Aqd/u7M3G0pGzfK+nSiJhUbB8q6fSI2C9tMrQ1LvejsRcldZH0drG9kaT56eIA7YPtS9b2fESc3lZZ0G5cK2mqpI8W2/NVuaxd6pIq6WRJE21fVmy/IOkLCfMgEUoqGlskaUbxm2yosgTIQw0/gPlBC6zR1OLPfSTVqbIElSQdLWlmkkTIXf+IONb2WEmKiKW2nTpUSsUNMU6JiL1sbypJEfFm4lhIhJKKxn5dPBrcnygH0K5ExPWSZPsUSfsW92GX7f+S9OeU2ZCtZcWQqobbovaX9E7aSGlFxLu29y0+ppyWHCUVq2j4QQvgA9tMUndJrxfbmxb7gMbOk/Q7SdvZnqjKWfgvJk2Uh0dsT1Jl6EP1DTFuSxcJKVBSsQrbn5P0A0nbq/L/g8X8gXXz76r8kL1Pla+fj0s6P2kiZCki7rU9TdJeqvxf+VpEvJo4Vg66SHpNUvUNMEISJbVkmN2PVdieI+kISY8H/zmAD8T2VpL2LDb/HhEvp8yDPNk+XNKfImJRsf0RSZ+IiN+kTQbkgXVS0djzkp6goALrxvbg4s/dJG2jytfS85K2KfYBjZ3XUFAlKSLeUGUIQKnZHmj7j7afKLaHsV53OXEmFauwvYcql/snq2oAP3fLAdbO9hURMa64zF/9jbVhyMyn1vBXUVK2H4uIYY32PR4RO6fKlAPbkyX9m6QJETG82PdERAxNmwxtjTOpaOxHkpaqMiaIu+UALRQR44oPPyPpLlWWc3tD0qRiH9DYFNsX2+5fPC7W+0uZlVnXiHio0b4VSZIgKSZOobFt+G0V+FCul7RYUsPi/sdJukHSMckSIVenSfqe3l9T915JX00XJxuvFstxNSzNdZSkl9JGQgpc7scqbF8o6Q8R8fvUWYD2yPbMiKhrbh+AptneQdIVkvaWtFDSc5I+HxH/SBoMbY6SilXYXiJpE1XGoy4XS1AB68T2LyVdFhF/K7b3lPTViDghbTLkxvZASWdJ6quqK5uMX66wvYmkmohYkjoL0qCkAkArsP24KpcnO0kaJGlesb29pFmcSUVjtqdL+i9VxqG+27A/Iko9LtV2L1VWOdhXla+h/5U0PiJeSxoMbY6SCkmV5XMiYtaalsqJiGltnQloT2xvv7bnuVSJxmxPjYjdU+fIje17JT0g6ZfFrs+rsn7s/ulSIQVKKiSttnxOg/f+c3D5CQBal+3zJb0i6ddadcm/19f0d8qgqeWmWJqrnCipWIXtYyT9LiIW2/6epN0k/YAzqQDQumw/18TuiIgd2jxMRoqluB6SdHOx6yhJIyPirHSpkAIlFatoWFza9r6qLOr/M0nnRsSezfxVAAA+tKoJvA3jdDtIeqv4mIm8JcJi/mis4ZvCZyVdGRF3SeqcMA8AbJBsd7X9XdtXFNsDbH8uda7UIqJbRNRERKfiUVPs6xYR3W3vlDoj2gYlFY3Ntz1B0rGS7ra9kfh/AgDrw7WSlqmyHqgkzZf0w3Rx2o1fpA6AtkH5QGPHSLpH0qcj4g1JPVW5hzIAoHX1j4gLVVmTWhGxVJW1qbF2vEclwW1RsYrim+RtVdsvidvRAcD6sMz2xnr/9p/9VTXLH2vEZJqSoKQCAJDG+ZJ+J2k72xMl7SPppKSJgIwwux8AgESKuyvtpcol7L9FxKuJI2XP9t8iYq/UObD+UVIBAEjA9h8jYr/m9pWJ7R6SDpK0bbFrvqR7ijkSKBkmTgEA0IZsd7HdU9Lmtjez3bN49NX75ax0bJ8gaZqkT0jqWjw+KWlq8RxKhjOpAAC0Idtfk/R1SduocqawYbb6YlXWp74sVbaUbM+WtGfjs6a2N5P094gYmCYZUqGkAgCQgO3TIuLS1DlyYfspSXtExKJG+3tImhIRA9IkQyrM7gcAIIGIuNT23pL6qurncUTckCxUWj+SNM327yU9X+zrI+kAVW7TjZLhTCoAAAnY/oWk/pIe1fu3pI6IOD1dqrSKS/uf1uoTpxamS4VUKKkAACRg+0lJdcEPYqBJzO4HACCNJyRtlTpEe2D78dQZ0PYYkwoAQBqbS5pp+yFV3Q41Ikani5SO7SPW9JQo86VESQUAII3zUwfIzK8kTZTU1PCHLm2cBRlgTCoAAInY3l7SgIj4g+2ukjpExJLUuVKwPVXSiRHxRBPPPR8R2yWIhYQYkwoAQAK2vyLpVkkTil3bSvpNukTJfV2VGxo05fC2DII8UFIBAEjjq5L2UVHMIuJpSVskTZRQRPw5Iuat4bkpDR/b/nbbpUJKlFQAANJ4JyKWNWzY7qimx2NiVUenDoC2QUkFACCNyba/I2lj2wdIukXSHYkztQdOHQBtg4lTAAAkYLtG0pckHahK8bpH0lUs7r92tqdFxG6pc2D9o6QCAJCY7Z6SekfEY6mz5M72IxExPHUOrH9c7gcAIAHb99vuXhTUqZKutP3z1LnagVtSB0DboKQCAJBGj4hYLOkISTdExJ6S9kucKTnbO9i+w/artl+xfbvtHRqej4gfp8yHtkNJBQAgjY62t5Z0jKQ7U4fJyI2SblblVqjbqHLm9L+TJkISlFQAANIYr8pkqTkR8XBxtvDpxJly0DUifhERK4rHL8VtUUuJiVMAAGTI9rcj4iepc7SVYmyuJH1L0kJJN6mybuyxkjaLCBbxLxlKKgAAGSrbUku2n1OllDa1DmpExA5N7McGrGPqAAAAoEmlWrQ+IvqlzoC8UFIBAMhTKS912j6hqf0RcUNbZ0FalFQAAPJUqjOpVfao+riLKstyTZNESS0ZSioAAHkq5aL1EXFa9bbtj6gyiQolwxJUAAAkwKL1LfaWJMarlhBnUgEASONGSZdLOrzYHqPKovV7JkuUAdt36P3xuDWS6lRZ3B8lwxJUAAAkYPuxiBjWaN/0iNglVaYc2B5VtblC0j8i4oVUeZAOJRUAgDbEovVAy1BSAQBoQyxav3a2j5B0gaQtVHmPrMr70j1pMLQ5SioAAMiG7TmSDomIJ1NnQVpMnAIAIAEWrV+jBRRUSJxJBQAgCduXVm2+t2h9RByVKFJSxWV+SRolaStJv5H0TsPzEXFbilxIh5IKAEAGGhatj4iDUmdJwfa1a3k6IuJf2iwMskBJBQAgA7Y7SXoiIgalzpIz29+OiJ+kzoH1jzGpAAAkwKL1H9jRkiipJUBJBQAgjZ9Vfcyi9S3X1NJd2ABRUgEASCAiJqfO0E4xTrEkalIHAACgjGwfYftp24tsL7a9xPbi1LnaAc6klgQlFQCANC6UNDoiekRE94joVua7Ktm+oPjz6GYOvaUN4iADzO4H8P/bu3sUq4IgDKD15Y4/4ApMzAY0NRe3MuaKa3AFbsEdiIGZRo7CmwW4Aw1cQBmMwuUx6Iu6G+850MntpMKiq/u7wARJPnb3k9l1rCLJVVWdV9Vldz+eXQ/zuZMKAANtQus/J3lbQuv/eFdVP6rq1tG1h9R1TupuT5n3ykkqAAwktP7vkrzv7qdH315398tZNTGHJhUAFrTX0PokX47H/UkO3X0+qybm8HAKANb0rwdE/5UkF7/vpT5Mctisb1V1Nbs+xnOSCgALSvK1ux/NrmOUJHeq6l5d/03q1WbrZ3d/n1MVM2lSAWBBN429YU+M+wFgTULr2TVNKgAMJLQeTmPcDwADCa2H0wjzB4CxhNbDCYz7AWCg7n7R3Xer6kN3396ss6p6M7s+WIUmFQDmuH/Dt2fDq4BFGfcDwEBJLqrqeVU9SHLYbJ1V1ac5VcF6PJwCgIGE1sNpNKkAACzHnVQAAJajSQUAYDmaVAAAlqNJBQBgOb8Abr44YfS2UKMAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 720x504 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "avbdkiIuKNNr"
      },
      "source": [
        "Looks like our pretrained USE TensorFlow Hub models have the best performance, even the one with only 10% of the training data seems to outperform the other models. This goes to show the power of transfer learning.\n",
        "\n",
        "How about we drill down and get the F1-score's of each model?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yktdOiufmm3p",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 546
        },
        "outputId": "97090bc0-63d5-41e2-f7b1-deddfcd99f53"
      },
      "source": [
        "# Sort model results by f1-score\n",
        "all_model_results.sort_values(\"f1\", ascending=False)[\"f1\"].plot(kind=\"bar\", figsize=(10, 7));"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAIRCAYAAABu0TiPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5xlZX3n+88XEBUFxNBq5C7T6iGKYlow4om34KAZIRIvYIzXyNEjXqJxgqNBgpMYSdSTUSaKJt4RwUlMqygSxUu80lwEAYkdUAEzsVECRCe26O/8sVbJpqjqKvrZ3WtXr8/79dovaq29uurrtmrXt9Z61vOkqpAkSdLm2W7oAJIkSSuZZUqSJKmBZUqSJKmBZUqSJKmBZUqSJKnBDkN94d1337323Xffob68JEnSsp1//vnXVdWqhZ4brEztu+++rFu3bqgvL0mStGxJvrPYc17mkyRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJarDD0AFa7Xv8x4eOAMC3/+w3h44gSZIGsKwzU0kOT3JFkvVJjl/g+b2TnJvkwiQXJ3nC9KNKkiTNniXLVJLtgVOAxwMHAMckOWDeYa8Bzqiqg4Cjgf857aCSJEmzaDlnpg4G1lfVlVW1ETgdOHLeMQXs0n+8K/C96UWUJEmaXcspU3sAV09sX9Pvm3Qi8Iwk1wBnAS9e6BMlOTbJuiTrNmzYsBlxJUmSZsu07uY7Bnh3Ve0JPAF4X5LbfO6qOrWq1lTVmlWrVk3pS0uSJA1nOWXqWmCvie09+32TngecAVBVXwbuBOw+jYCSJEmzbDll6jxgdZL9kuxIN8B87bxjvgs8FiDJ/0VXpryOJ0mStnlLlqmquhk4DjgbuJzurr1Lk5yU5Ij+sFcAz0/ydeCDwLOrqrZUaEmSpFmxrEk7q+osuoHlk/tOmPj4MuDQ6UaTJEmafSt+BnTd1qzMCg/ODC9J2vZZpjQalkxJ0pbgQseSJEkNLFOSJEkNLFOSJEkNHDMljZxjySSpjWemJEmSGlimJEmSGlimJEmSGjhmSpIWMCtjyWZpHNmsvCYwW6+LZJmSJKmBJVNe5pMkSWpgmZIkSWpgmZIkSWpgmZIkSWpgmZIkSWpgmZIkSWrg1AiSJGnqxjRlhGemJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGiyrTCU5PMkVSdYnOX6B59+c5KL+8U9J/m36USVJkmbPDksdkGR74BTgMOAa4Lwka6vqsrljqur3J45/MXDQFsgqSZI0c5ZzZupgYH1VXVlVG4HTgSM3cfwxwAenEU6SJGnWLadM7QFcPbF9Tb/vNpLsA+wHfGaR549Nsi7Jug0bNtzerJIkSTNn2gPQjwY+XFU/W+jJqjq1qtZU1ZpVq1ZN+UtLkiRtfcspU9cCe01s79nvW8jReIlPkiSNyHLK1HnA6iT7JdmRrjCtnX9QkvsDuwFfnm5ESZKk2bVkmaqqm4HjgLOBy4EzqurSJCclOWLi0KOB06uqtkxUSZKk2bPk1AgAVXUWcNa8fSfM2z5xerEkSZJWBmdAlyRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJamCZkiRJarCsMpXk8CRXJFmf5PhFjnlqksuSXJrktOnGlCRJmk07LHVAku2BU4DDgGuA85KsrarLJo5ZDbwKOLSqrk9yjy0VWJIkaZYs58zUwcD6qrqyqjYCpwNHzjvm+cApVXU9QFV9f7oxJUmSZtNyytQewNUT29f0+ybdF7hvki8m+UqSwxf6REmOTbIuyboNGzZsXmJJkqQZMq0B6DsAq4FHAccA70hyt/kHVdWpVbWmqtasWrVqSl9akiRpOMspU9cCe01s79nvm3QNsLaqflpVVwH/RFeuJEmStmnLKVPnAauT7JdkR+BoYO28Yz5Cd1aKJLvTXfa7coo5JUmSZtKSZaqqbgaOA84GLgfOqKpLk5yU5Ij+sLOBHyS5DDgXeGVV/WBLhZYkSZoVS06NAFBVZwFnzdt3wsTHBby8f0iSJI2GM6BLkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1WFaZSnJ4kiuSrE9y/ALPPzvJhiQX9Y/fm35USZKk2bPDUgck2R44BTgMuAY4L8naqrps3qEfqqrjtkBGSZKkmbWcM1MHA+ur6sqq2gicDhy5ZWNJkiStDMspU3sAV09sX9Pvm++3k1yc5MNJ9lroEyU5Nsm6JOs2bNiwGXElSZJmy7QGoH8U2LeqDgTOAd6z0EFVdWpVramqNatWrZrSl5YkSRrOcsrUtcDkmaY9+32/UFU/qKqf9JvvBH51OvEkSZJm23LK1HnA6iT7JdkROBpYO3lAkl+e2DwCuHx6ESVJkmbXknfzVdXNSY4Dzga2B/6mqi5NchKwrqrWAi9JcgRwM/BD4NlbMLMkSdLMWLJMAVTVWcBZ8/adMPHxq4BXTTeaJEnS7HMGdEmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAbLKlNJDk9yRZL1SY7fxHG/naSSrJleREmSpNm1ZJlKsj1wCvB44ADgmCQHLHDczsBLga9OO6QkSdKsWs6ZqYOB9VV1ZVVtBE4HjlzguNcBbwD+Y4r5JEmSZtpyytQewNUT29f0+34hyUOAvarq45v6REmOTbIuyboNGzbc7rCSJEmzpnkAepLtgDcBr1jq2Ko6tarWVNWaVatWtX5pSZKkwS2nTF0L7DWxvWe/b87OwAOAzyb5NvAwYK2D0CVJ0hgsp0ydB6xOsl+SHYGjgbVzT1bVDVW1e1XtW1X7Al8BjqiqdVsksSRJ0gxZskxV1c3AccDZwOXAGVV1aZKTkhyxpQNKkiTNsh2Wc1BVnQWcNW/fCYsc+6j2WJIkSSuDM6BLkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1WFaZSnJ4kiuSrE9y/ALPvyDJJUkuSvKPSQ6YflRJkqTZs2SZSrI9cArweOAA4JgFytJpVfXAqnowcDLwpqknlSRJmkHLOTN1MLC+qq6sqo3A6cCRkwdU1Y0Tm3cBanoRJUmSZtcOyzhmD+Dqie1rgEPmH5TkRcDLgR2Bxyz0iZIcCxwLsPfee9/erJIkSTNnagPQq+qUqtof+EPgNYscc2pVramqNatWrZrWl5YkSRrMcsrUtcBeE9t79vsWczrwWy2hJEmSVorllKnzgNVJ9kuyI3A0sHbygCSrJzZ/E/jW9CJKkiTNriXHTFXVzUmOA84Gtgf+pqouTXISsK6q1gLHJfkN4KfA9cCztmRoSZKkWbGcAehU1VnAWfP2nTDx8UunnEuSJGlFcAZ0SZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBpYpSZKkBssqU0kOT3JFkvVJjl/g+ZcnuSzJxUk+nWSf6UeVJEmaPUuWqSTbA6cAjwcOAI5JcsC8wy4E1lTVgcCHgZOnHVSSJGkWLefM1MHA+qq6sqo2AqcDR04eUFXnVtWP+82vAHtON6YkSdJsWk6Z2gO4emL7mn7fYp4HfKIllCRJ0kqxwzQ/WZJnAGuARy7y/LHAsQB77733NL+0JEnSIJZzZupaYK+J7T37fbeS5DeAVwNHVNVPFvpEVXVqVa2pqjWrVq3anLySJEkzZTll6jxgdZL9kuwIHA2snTwgyUHA2+mK1PenH1OSJGk2LVmmqupm4DjgbOBy4IyqujTJSUmO6A/7c+CuwJlJLkqydpFPJ0mStE1Z1pipqjoLOGvevhMmPv6NKeeSJElaEZwBXZIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqcGyylSSw5NckWR9kuMXeP7Xk1yQ5OYkT55+TEmSpNm0ZJlKsj1wCvB44ADgmCQHzDvsu8CzgdOmHVCSJGmW7bCMYw4G1lfVlQBJTgeOBC6bO6Cqvt0/9/MtkFGSJGlmLecy3x7A1RPb1/T7brckxyZZl2Tdhg0bNudTSJIkzZStOgC9qk6tqjVVtWbVqlVb80tLkiRtEcspU9cCe01s79nvkyRJGr3llKnzgNVJ9kuyI3A0sHbLxpIkSVoZlixTVXUzcBxwNnA5cEZVXZrkpCRHACR5aJJrgKcAb09y6ZYMLUmSNCuWczcfVXUWcNa8fSdMfHwe3eU/SZKkUXEGdEmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAbLKlNJDk9yRZL1SY5f4Pk7JvlQ//xXk+w77aCSJEmzaMkylWR74BTg8cABwDFJDph32POA66vqPwFvBt4w7aCSJEmzaDlnpg4G1lfVlVW1ETgdOHLeMUcC7+k//jDw2CSZXkxJkqTZlKra9AHJk4HDq+r3+u3fBQ6pquMmjvlGf8w1/fY/98dcN+9zHQsc22/eD7hiWv9DGu0OXLfkUePj63JbviYL83VZmK/LwnxdbsvXZGGz9LrsU1WrFnpih62ZoqpOBU7dml9zOZKsq6o1Q+eYNb4ut+VrsjBfl4X5uizM1+W2fE0WtlJel+Vc5rsW2Gtie89+34LHJNkB2BX4wTQCSpIkzbLllKnzgNVJ9kuyI3A0sHbeMWuBZ/UfPxn4TC11/VCSJGkbsORlvqq6OclxwNnA9sDfVNWlSU4C1lXVWuCvgfclWQ/8kK5wrSQzd+lxRvi63JavycJ8XRbm67IwX5fb8jVZ2Ip4XZYcgC5JkqTFOQO6JElSA8uUJElSA8uUJElSA8uUJElSg606aees6Ncb/IeqevTQWWZRkkcAq6vqXUlWAXetqquGzjWUJDsBrwD2rqrnJ1kN3K+qPjZwtEEkeQuw6J0rVfWSrRhHK0D/M/N6uvVd7zS3v6ruM1ioASW5+6aer6ofbq0ssyjJGuDVwD50PSVAVdWBgwbbhFGWqar6WZKfJ9m1qm4YOs8sSfJaYA3dcj/vAu4AvB84dMhcA3sXcD7wa/32tcCZwCjLFLCu/++hdL8cP9RvPwW4bJBEMyLJTdxSNHek+/n5UVXtMlyqmfAu4LXAm4FHA89h3FdGzqf7PlloDdsCRlkyJ3wAeCVwCfDzgbMsyyjLVO/fgUuSnAP8aG6nf1XzJOAg4AKAqvpekp2HjTS4/avqaUmOAaiqH495Ie+qeg9AkhcCj6iqm/vttwFfGDLb0KrqFz8r/ffIkcDDhks0M+5cVZ9Okqr6DnBikvOBE4YONoSq2m/oDDNuQz+H5Yox5jL1t/1Dt7axqipJASS5y9CBZsDGJHemP+OQZH/gJ8NGmgm7AbvQTdQLcNd+n+iuSQAf6c/2Hj90noH9JMl2wLf6SaCvpft+Gb0kuwGrufXlz88Pl2gmvDbJO4FPM/FeW1Uz+zt7tGWqqt7T/4Lcu6quGDrPDDkjyduBuyV5PvBc4B0DZxraa4FPAnsl+QDd5a1nD5poNvwZcGGSc+kuV/w6cOKgiQaW5KiJze3oLpn/x0BxZslLgZ2AlwCvo7vU98xBE82AJL9H99rsCVxEdxbzy8Bjhsw1A54D3J/uMvncZb5ihk+AjHYG9CRPBP4C2LGq9kvyYOCkqjpi4GiDS3IY8Di6X5BnV9U5A0caXJJfonujC/CVqrpu4EgzIcm9gEP6za9W1f8eMs/QkrxrYvNm4NvAO6rq+8Mkmg1JnlJVZy61b2ySXAI8lO495cFJ7g/8aVUdtcQ/3aYluaKq7jd0jttjzGXqfLr2/9mqOqjf942qesCwyTRrkhwKXFRVP0ryDOAhwF/2Yz80Icn9q+qbQ+cYQn+X8Euq6s1DZ5k1SS6oqocstW9skpxXVQ9NchFwSFX9JMmlVfUrQ2cbUv9HyZ9X1Yq5oWW0l/mAn1bVDfPGEa+Iuwa2pP4yxRuAe9CdhZm7JXXMdyP9FfCgJA8CXk63sPd7gUcOmmo2fQrYe+gQQ+jvEj6G7o41AUkeDzwB2CPJ/5h4ahe6M3djd02SuwEfAc5Jcj3gH2ndVYCLklxFN2bKqRFm2KVJng5s38+B8hLgSwNnmgUnA0+sqsuHDjJDbu4H5R8JnFJVf53keUOHGsq8X4q3egq429bMMoO+mOStdNNFTN4lfMFwkQb1PbqpNI6gmw5gzk3A7w+SaIZU1ZP6D0/sxx7uSjc+c+wOHzrA7TXmy3w70U0K9ouxQcDrqmrUg0WTfLGqxjyn1G0k+RzdG9xz6AZZfx/4elU9cNBgA+nnUnoFC9/R+Maq2n0rR5oZ/S9EuGWuqbm/qEc9oDjJHarqp0PnmEX95eF7MnFyo6q+O1yi4SV5X1X97lL7Zsloy5QWluQvgXvRnXZeEbekbmn9IOunA+dV1ReS7A08qqreO3C0QST5DPCaqrrNmdwkV415Dp0kr+DWkzEWcCOwrqouGizYwJwBfWFJXkx3t/C/MnHX2ixfztoa5o+n6wvnJVV1wICxNml0ZSrJR9n0Uhijvptv3t1Ic6qqnrvVw2gm9Uth/EdV/XjoLLMmyWl00yGspStU/wW4GNgXOLOqTh4u3XCS/CO3zID+RPoZ0KtqlJN2zkmynm7g+Q+GzjILkrwK+G/AnYG595cAG4FTq+pVQ2VbyhjL1Nyg4aPozsC8v98+BvjXqhr9dXzdmoPyF9a/Lh+vKicw7SX5PPCEqvr3fvuuwMfpxoCcP8t/WW9JSc6vql9Ncsnc5fG5fUNnG1J/WfiwuVUE1Eny+lkuTgsZ3QD0qvocQJI3VtWaiac+mmTdIv9sm5fkv1bVyYstYjvyZXYclL+wJwJv7gvEh4BP+kuBe3DrsWQ/Be5ZVf8nyZhLpzOgL+xK4LNJPs6th1W8abhIM+FjSe6ykqajGV2ZmnCXJPepqisBkuwHjHnplLmiMNpCuQn/apG6rap6TpI7AI+nO7N7SpJzqur3Bo42pA8AX03y9/32E4HT+mWZVsycOVvA/BnQHwM8a9BEs+G7/WPH/qHO5HQ0rwDeyYxPRzO6y3xzkhwOnEr3l0GAfYBjq+pTgwbTzHFQ/qb1hepw+rsdx3w3H0CSNXRLDgF8sar8A0Wb1F8OZu7y8NjNDUBPcgJwbT8dzUxP8jraMgWQ5I506/8AfHPMYz8cmL84B+UvrJ+Q8WnAo4DPAmcAn/JSn+b4vrJpSR4AvA+4e7/rOuCZVXXpcKmGtxKnoxltmer/mn4h3f9R0P0yePtY50KZGJi/oLmxZtKcJB+kGyv1iTH/IaLFecPPpiX5EvDqqjq3334U3dp8Dx802MBW4nQ0Yy5T76Rbkfo9/a7fBX428vEeACS5M7B3VV0xdJZZkOS+dNfw71lVD0hyIHBEVf33gaNJK0KSdfNu+Flw39gk+XpVPWipfZp92w0dYEAPrapnVdVn+sdz6FbvHrUkTwQuol/SIMmDk6wdNtXg3gG8iu7OLKrqYuDoQRPNgCRHJflWkhuS3JjkpiQ3Dp1LM+kuSX4xQac3/PzClUn+KMm+/eM1dON4R2nuPWSBx8y/t4z5br6fJdm/qv4ZoP9B/9nAmWbBicDBdJc9qaqL+je+Mdupqr42b1FsxwU5ZYSW7/fppgC41Q0/w0aaCc8F/hiYu5nlC/2+UaqqnYfOsLnGXKZeCZw774f7OcNGmgk/raob5hWHcV4LvsV1Sfanfx2SPBn4l2EjzQSnjNCyVNUn+yVlFrzhJ8lhVXXOMOmGU1XX000XoRVutGOm4Bd3892v37zCQbSQ5K+BTwPHA79N94N+h6p6waDBBtSftTwVeDhwPXAV8Iyq+vaQuYbmlBGallm/7X3akvx/VfWyxe52HPtdjivRaMtUkhcBH6iqf+u3dwOOqar/OWyyYSXZCXg18Di6M3ZnA6+rqv8YNNgM6Cde3K6qbho6yyxwyghNS5ILq+qgoXNsLUl+tarOX+wuau+eXnnGXKYuqqoHz9s3qh/opfQrdd+lqmZ64N+WkuTlm3reJR+k6Rjbmak5SV5aVX+51D7NvjHfzbd9JgYG9cVh9NP5JzktyS79WZhLgMuSvHLoXAPZuX+soZuTbI/+8QK6taJGLcmeSf4uyff7x/9KsufQuaQVZKEldZ69tUOo3ZgHoH8S+FCSt/fb/0+/b+wOqKobk/wO8Am6sVPnA38+bKytr6r+GKBfyPchc5f3kpwIfHzAaLPiXcBpwFP67Wf0+w4bLJFmUpI7zh+TOm/ft7d+quEkOYZuUsr95k09szPww2FSqcWYy9Qf0hWoF/bb59Atpjh2d+hnh/8t4K1V9dMk47wWfIt7Ahsntjf2+8ZuVVVNjpt6d5KXDZZGs+zL3PZs7i/2VdVRWz3RsL5Ed0fw7sAbJ/bfBFw8SCI1GW2Zqqqf081q/VdDZ5kxb6f7K/HrwOeT7AOMcszUhPcCX0vyd/32bwHvHi7OzPhBkmcAH+y3jwF+MGAezZh+WZA9gDsnOYjuphaAXYCdBgs2sKr6DvCd/grA9+Zu8OlXn9iTkZ2p2xaMeQD6oXQTVO5DVypDdyfSfTb178YoyQ5jX7w2yUOA/7vf/HxVXTjx3G79fDGj0hfttwC/Rnd795eAF1fV1YMG08xI8iy6MUBrgHUTT90EvHvs02gkWQc8vKo29ts7Al+sqtGvxrHSjLlMfZNuVt7zmZj5vKpG/5d1kt8EfgW409y+qjppuESzbcR3Ir0HeNlckUxyd+AvnBpB8yX57ar6X0PnmDWL3FXu2nwr0Ggv8wE3VNUnhg4xa5K8je70+6PpxpA9GfjaoKFmX5Y+ZJt04OQZuar6YX8pR5rvY0meDuzLxO8d/0hjQ5IjqmotQJIjgesGzqTNMOYydW6SP6dbE2ly9uYLhos0Ex5eVQcmubiq/jjJG+nu6tPixnl6F7abvMTZn5ka83uKFvf3wA10VwJGv9LEhBcAH0hyCt37yDXAM4eNpM0x5je+Q/r/rpnYV8BjBsgyS/5P/98fJ7k33YDiXx4wj2bXG4EvJzmz334K8CcD5tHs2rOqDh86xKypqn8GHpbkrv32vw8cSZtptGWqqh49dIYZ9bEkdwNOpvsrEpwyYimjvMxXVe/tB9DO/QFyVFVdNmQmzawvJXlgVV0ydJBZkuSewJ8C966qxyc5APi1qvrrgaPpdhrzAHS/iRfQ35r7Qro71wr4AvBXY1+bL8kjgNVV9a4kq4C7VtVV/XN3ryon2pMWkeQy4D/RLRL+E265e/rAQYMNLMkn6Ca6fXVVPSjJDsCFVfXAgaPpdhpzmfKbeAFJzqC7bfn9/a6nA7tW1VOHSzWsJK+luxx8v8Ne0psAAA0YSURBVKq6b3/588yqOnTgaNKK0E+jcRv9fEujleS8qnro5LqwC93hp9k35rX5dq+qM4CfA/TzKP1s0/9kFB5QVc+rqnP7x/OBBwwdamBPAo4AfgRQVd+jW/ZB0jL0pWkv4DH9xz9m3L9/5vwoyS/R38SS5GF0A/W1wox2zBR+Ey/mgiQPq6qvACQ5hFtPtjdGG6uq5pbV6ReBlrRMk2d36a4I3IHu7PfYz+6+HFgL7J/ki8AquulotMKMuUz5TTwhySV0xfIOdINFv9tv7wN8c8hsM+CMfkHsuyV5PvBc4B0DZ5JWkicBBwEXQHd2N8moz+4m2R54ZP+4H904siuq6qeDBtNmGe2YKeiWSWGRb+Ikh1XVOYOF28oWG9Mwx7ENOQx4HN33ytlj+t6QWiX5WlUdPLdaQH9298sOQO9el6FzqN2oy9SmjHWJEEmatiR/AKwGDgNeT3d297SqesugwQaW5M10VwM+RD8mE5w8eiWyTC1i8u4KjVOSm1h4dvO527p32cqRpBXLs7u3leTcBXZXVY198ugVxzK1CM9MSdJ0JNkP+Je5+er6+ezuWVXfHjSYNCVjHoAuLVuShwCPoDtT9Y9VdeHAkaSV5Ezg4RPbP+v3PXSYOMNK8oyqen+Sly/0fFW9aWtnUhvn+Vjct4cOoNmQ5ATgPcAvAbsD707ymmFTSSvKDlW1cW6j/3jHAfMMbW56lZ0XeWiFGe1lviQ7Aa8A9q6q5ydZTTfD9ccGjqYZk+QK4EHzLlFcVFX3GzaZtDIkOQd4S1Wt7bePBF5SVY8dNpk0HWO+zPcuuoV8f63fvpbutLNlSvN9D7gTMLc+4R3pvl8kLc8LgA8keWu/fQ3wuwPmGVSS/7Gp56vqJVsri6ZjzGVq/6p6WpJjAKrqx0kydCjNpBuAS/u/rovu9u6vzb0h+sYnLa6fnPKFVfWwJHcFqKp/HzjW0M7v/3socADd1AgATwEuGySRmoy5TG3sL9fMLRGyP91q5tJ8f9c/5nx2oBzSilNVP0vyiP7jsZcoAKrqPQBJXgg8ol8bliRvA74wZDZtnjGXqdcCnwT2SvIBur8Qnj1oIs2kuTc+SZvtwiRr6YZSTE5O+bfDRZoJuwG7AD/st+/a79MKM9oyVVXnJLkAeBjdJHIvrarrBo6lGZTkvwCvo1uncAectFO6ve4E/ACYnIyygLGXqT+jK5rn0r2v/Dpw4qCJtFnGfDffk4DPVNUN/fbdgEdV1UeGTaZZk2Q9cBRwSY31B0bSFpHkXsAh/eZXq+p/D5lHm2fM80y9dq5IAVTVv9Fd+pPmuxr4hkVK2jxJ7pvk00m+0W8fOOa52pLcv//vQ4B7073HXA3cu9+nFWbMZ6Yunr9ieZJLquqBQ2XSbEryULrLfJ9j4iYFZymWlifJ54BXAm+fW/M0yTeq6gHDJhtGklOr6tj+8t7kL+G5IQSuzbfCjPnM1Lokb0qyf/94E7fcripN+hPgx3TjPpylWLr9dqqqr83bd/MgSWZAVR3bf/gE4ON006/8G7C236cVZrQD0IEXA3/ELfN7nAO8aLg4mmH3Hutf0NKUXNdPPzM3Fc2TgX8ZNtJMeA9wIzA3iefTgfcCTx0skTbLaC/zScuV5GTgH6rqU0NnkVaiJPcBTqVb7Ph64Crgd6rqO4MGG1iSy6rqgKX2afaNtkwluS/wB8C+TJyh81q15ktyE93CpD8BfopTI0ibJcldgO2q6qahs8yCJO8H3lpVX+m3DwFeVFXPHDaZbq8xl6mvA2+jGyf1s7n9VeW4KUmaoiS/RHe39CPoLvX9I3BSVf1g0GADSXIJ3etwB+B+wHf77X2Ab3pmauUZc5k6v6p+degcml1J7l9V31zsVuWqumBrZ5JWon5dy88D7+93/Q7dvH6/MVyq4STZZ1PPj/3y50o05jJ1IvB9ujXXJm93/+Fi/0bjMu/25Tm/+IHxkrC0PAtNg+BUNNqWjLlMXbXA7qqq+2z1MJppSZ4KfLKqbkzyR8BDgNd5Zkpann7qma8BZ/S7ngwcXFV/MFwqaXpGW6ak5Zqb4DXdyvevA/4COKGqDlnin0riVjdxzI1P3Z5bFjz2Zg6teKOdtDPJTklek+TUfnt1v6CtNN/cL4DfBN5RVR8Hdhwwj7SiVNXOVbVdVd2hf2zX79u5qnZJ8itDZ5RajLZMAe8CNtLNewJwLfDfh4ujGXZtkrcDTwPOSnJHxv2zI03b+4YOILUY8y+E/avqZLp5g6iqH9PNHyTN91TgbOA/9wti351unTFJ0+F7r1a0MS8nszHJnblleYP9mbirT5rTF+2/ndj+F1wKQ5omB+9qRRtzmToR+CSwV5IPAIcCzxk0kSRJWnFGfTdfPyvvw+hOMX+lqq4bOJIkjU6Sr1TVw4bOIW2u0ZapJJ+uqscutU+StPmS7AocDuzR77oWOLsffyhtE0Y3AD3JnZLcHdg9yW5J7t4/9uWWH3ZJUqMkzwQuAB4F7NQ/Hg2c3z8nbRNGd2YqyUuBlwH3pvsLae4ukhvp5hB661DZJGlbkuQK4JD5Z6GS7AZ8taruO0wyabpGV6bmJHlxVb1l6ByStK1K8k/AQ6vqhnn7dwXWVdXqYZJJ0zXau/mq6i1JHg7sy8TrUFXvHSyUJG1b/gS4IMmngKv7fXsDh9EtzSRtE8Z8Zup9wP7ARdyyXEhV1UuGSyVJ25b+kt5/5rYD0K8fLpU0XWMuU5cDB9RYXwBJkjQVo7ubb8I3gHsNHUKSxijJJUNnkKZltGOmgN2By5J8jYllZKrqiOEiSdK2I8lRiz2Ff8xqGzLmMnXi0AEkaRv3IeADLLz23p22chZpixntmCmAJPsAq6vqH5LsBGxfVTcNnUuStgVJzgeeVVXfWOC5q6tqrwFiSVM32jFTSZ4PfBh4e79rD+AjwyWSpG3Oy+gmRF7Ik7ZmEGlLGm2ZAl4EHEr/g15V3wLuMWgiSdqGVNUXquq7izy3bu7jJK/aeqmk6RtzmfpJVW2c20iyAwtf15ckbVlPGTqA1GLMZepzSf4bcOckhwFnAh8dOJMkjVGWPkSaXaMdgJ5kO+B5wOPofpDPBt7pJJ6StHUluaCqHjJ0DmlzjbZMTUpyd2DPqrp46CySNDZJLqyqg4bOIW2u0V7mS/LZJLv0Rep84B1J3jx0LkkaoTOHDiC1GG2ZAnatqhuBo4D3VtUhwGMHziRJ25wk90ny0STXJfl+kr9Pcp+556vqT4fMJ7Uac5naIckvA08FPjZ0GEnahp0GnEG3hMy96c5EfXDQRNIUjblMnUQ36Hx9VZ3X/5X0rYEzSdK2aKeqel9V3dw/3o/LyWgb4gD0RSR5VVW9fugckrRS9WNSAf4QuB44nW4+v6cBu1WVk3Vqm2CZWoS36kpSmyRX0ZWnheaRqqq6zwL7pRVnh6EDzDAnkZOkBlW139AZpK3BMrU4T9lJ0hQkeeZC+6vqvVs7i7QlWKYW55kpSZqOh058fCe6aWguACxT2iZYphbnJHKSNAVV9eLJ7SR3oxuMLm0TRjs1gpPISdJgfgQ4nkrbjDGfmToNOAV4Ur99NN0kcocMlkiStkFJPsot41C3Aw6gm8RT2iaMdmqEJBdX1YHz9n29qh40VCZJ2hYleeTE5s3Ad6rqmqHySNM2ujLlJHKSJGmaxlimnEROkraiJEcBbwDuQffeG7r3210GDSZNyejKlCRp60qyHnhiVV0+dBZpSxjtAHQnkZOkreZfLVLalo32zFSSt0xs/mISuap68kCRJGmb0l/eA3gkcC/gI8BP5p6vqr8dIpc0baMtU/PNTSJXVYcPnUWStgVJ3rWJp6uqnrvVwkhbkGWql+QOwDeq6n5DZ5GkMUnyqqp6/dA5pM015jFTTiInSbPhKYBlSivWaMsU8BcTHzuJnCQNx4XltaKNtkxV1eeGziBJAm65SiCtSGNe6PioJN9KckOSG5PclOTGoXNJ0gh5Zkor2mjLFHAycERV7VpVu1TVzs7GK0nTk+QN/X+fssShZ26FONIWM9q7+ZJ8saoOHTqHJG2rklwCHAicX1UPGTqPtKWMbszUxCRy65J8CCeRk6Qt5ZN0C8rfdd4wCtfm0zZldGemnEROkrauJJ+qqsfN23dyVf3XoTJJ0zS6MrVcTiInSdOR5IL5l/mSXFxVBw6VSZqmMQ9AX8pSAyYlSZuQ5IX9uKn7Jbl44nEVcMnQ+aRp8czUIpJcWFUHDZ1DklaqJLsCu9HNbn78xFM3VdUPh0klTZ9lahELnZaWJEmaz8t8i3MSOUmStKTRlSknkZMkSdM0ust8TiInSZKmaXSTduIkcpIkaYpGd5mvql5ZVXcDPtOvyTf32Bl429D5JEnSyjK6MjVh9wX2Hb7VU0iSpBVtdJf5krwQ+H+B+yS5eOKpnYEvDZNKkiStVGMcgO4kcpIkaWpGV6YkSZKmacxjpiRJkppZpiRJkhpYpiRJkhpYpiRJkhr8/7jooyj80XUxAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x504 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pv2iE0TPGdNy"
      },
      "source": [
        "Drilling down into a single metric we see our USE TensorFlow Hub models performing  better than all of the other models. Interestingly, the baseline's F1-score isn't too far off the rest of the deeper models.\n",
        "\n",
        "We can also visualize all of our model's training logs using TensorBoard.dev."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Ca8TalwGhPf"
      },
      "source": [
        "# # View tensorboard logs of transfer learning modelling experiments (should be 4 models)\n",
        "# # Upload TensorBoard dev records\n",
        "# !tensorboard dev upload --logdir ./model_logs \\\n",
        "#   --name \"NLP modelling experiments\" \\\n",
        "#   --description \"A series of different NLP modellings experiments with various models\" \\\n",
        "#   --one_shot # exits the uploader when upload has finished"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uIYVXCUJ3FBn"
      },
      "source": [
        "The TensorBoard logs of the different modelling experiments we ran can be viewed here: https://tensorboard.dev/experiment/LkoAakb7QIKBZ0RL97cXbw/"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Os7dv00u21jg"
      },
      "source": [
        "# If you need to remove previous experiments, you can do so using the following command\n",
        "# !tensorboard dev delete --experiment_id EXPERIMENT_ID_TO_DELETE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GGVZhTTiGdd5"
      },
      "source": [
        "## Combining our models (model ensembling/stacking)\n",
        "\n",
        "Many production systems use an **ensemble** (multiple different models combined) of models to make a prediction.\n",
        "\n",
        "The idea behind model stacking is that if several uncorrelated models agree on a prediction, then the prediction must be more robust than a prediction made by a singular model.\n",
        "\n",
        "The keyword in the sentence above is **uncorrelated**, which is another way of saying, different types of models. For example, in our case, we might combine our baseline, our bidirectional model and our TensorFlow Hub USE model.\n",
        "\n",
        "Although these models are all trained on the same data, they all have a different way of finding patterns.\n",
        "\n",
        "If we were to use three similarly trained models, such as three LSTM models, the predictions they output will likely be very similar.\n",
        "\n",
        "Think of it as trying to decide where to eat with your friends. If you all have similar tastes, you'll probably all pick the same restaurant. But if you've all got different tastes and still end up picking the same restaurant, the restaurant must be good.\n",
        "\n",
        "Since we're working with a classification problem, there are a few of ways we can combine our models:\n",
        "1. **Averaging** - Take the output prediction probabilities of each model for each sample, combine them and then average them.\n",
        "2. **Majority vote (mode)** - Make class predictions with each of your models on all samples, the predicted class is the one in majority. For example, if three different models predict `[1, 0, 1]` respectively, the majority class is `1`, therefore, that would be the predicted label.\n",
        "3. **Model stacking** - Take the outputs of each of your chosen models and use them as inputs to another model.\n",
        "\n",
        "> 📖 **Resource:** The above methods for model stacking/ensembling were adapted from Chapter 6 of the [Machine Learning Engineering Book](http://www.mlebook.com/wiki/doku.php) by Andriy Burkov. If you're looking to enter the field of machine learning engineering, not only building models but production-scale machine learning systems, I'd highly recommend reading it in its entirety.\n",
        "\n",
        "Again, the concept of model stacking is best seen in action.\n",
        "\n",
        "We're going to combine our baseline model (`model_0`), LSTM model (`model_2`) and our USE model trained on the full training data (`model_6`) by averaging the combined prediction probabilities of each."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t63u8PCCm-yo",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "47a65c8c-325a-402c-bfa3-0a940416bd54"
      },
      "source": [
        "# Get mean pred probs for 3 models\n",
        "baseline_pred_probs = np.max(model_0.predict_proba(val_sentences), axis=1) # get the prediction probabilities from baseline model\n",
        "combined_pred_probs = baseline_pred_probs + tf.squeeze(model_2_pred_probs, axis=1) + tf.squeeze(model_6_pred_probs)\n",
        "combined_preds = tf.round(combined_pred_probs/3) # average and round the prediction probabilities to get prediction classes\n",
        "combined_preds[:20]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(20,), dtype=float32, numpy=\n",
              "array([0., 1., 1., 0., 0., 1., 1., 1., 1., 1., 0., 0., 0., 1., 0., 0., 0.,\n",
              "       0., 0., 1.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 105
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6abZa7wqlXSI"
      },
      "source": [
        "Wonderful! We've got a combined predictions array of different classes, let's evaluate them against the true labels and add our stacked model's results to our `all_model_results` DataFrame."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ieYvhDiev8Et",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b3b965dd-54c8-4796-edcc-b31e344b38bb"
      },
      "source": [
        "# Calculate results from averaging the prediction probabilities\n",
        "ensemble_results = calculate_results(val_labels, combined_preds)\n",
        "ensemble_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 78.08398950131233,\n",
              " 'f1': 0.7805169025578647,\n",
              " 'precision': 0.7805216999297674,\n",
              " 'recall': 0.7808398950131233}"
            ]
          },
          "metadata": {},
          "execution_count": 106
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "132EHlUUpRrP"
      },
      "source": [
        "# Add our combined model's results to the results DataFrame\n",
        "all_model_results.loc[\"ensemble_results\"] = ensemble_results"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pm2P1zsvpZ3D"
      },
      "source": [
        "# Convert the accuracy to the same scale as the rest of the results\n",
        "all_model_results.loc[\"ensemble_results\"][\"accuracy\"] = all_model_results.loc[\"ensemble_results\"][\"accuracy\"]/100"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "trmdZ6eEpwHI",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 328
        },
        "outputId": "b2704163-e293-4b88-f289-efcc04d9b4b2"
      },
      "source": [
        "all_model_results"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>accuracy</th>\n",
              "      <th>precision</th>\n",
              "      <th>recall</th>\n",
              "      <th>f1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>baseline</th>\n",
              "      <td>0.792651</td>\n",
              "      <td>0.811139</td>\n",
              "      <td>0.792651</td>\n",
              "      <td>0.786219</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>simple_dense</th>\n",
              "      <td>0.787402</td>\n",
              "      <td>0.791492</td>\n",
              "      <td>0.787402</td>\n",
              "      <td>0.784697</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>lstm</th>\n",
              "      <td>0.750656</td>\n",
              "      <td>0.751008</td>\n",
              "      <td>0.750656</td>\n",
              "      <td>0.748927</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>gru</th>\n",
              "      <td>0.767717</td>\n",
              "      <td>0.767545</td>\n",
              "      <td>0.767717</td>\n",
              "      <td>0.766793</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>bidirectional</th>\n",
              "      <td>0.766404</td>\n",
              "      <td>0.766590</td>\n",
              "      <td>0.766404</td>\n",
              "      <td>0.765121</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>conv1d</th>\n",
              "      <td>0.778215</td>\n",
              "      <td>0.780752</td>\n",
              "      <td>0.778215</td>\n",
              "      <td>0.775881</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>tf_hub_sentence_encoder</th>\n",
              "      <td>0.812336</td>\n",
              "      <td>0.814880</td>\n",
              "      <td>0.812336</td>\n",
              "      <td>0.810687</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>tf_hub_10_percent_data</th>\n",
              "      <td>0.770341</td>\n",
              "      <td>0.775563</td>\n",
              "      <td>0.770341</td>\n",
              "      <td>0.766706</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ensemble_results</th>\n",
              "      <td>0.780840</td>\n",
              "      <td>0.780522</td>\n",
              "      <td>0.780840</td>\n",
              "      <td>0.780517</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                         accuracy  precision    recall        f1\n",
              "baseline                 0.792651   0.811139  0.792651  0.786219\n",
              "simple_dense             0.787402   0.791492  0.787402  0.784697\n",
              "lstm                     0.750656   0.751008  0.750656  0.748927\n",
              "gru                      0.767717   0.767545  0.767717  0.766793\n",
              "bidirectional            0.766404   0.766590  0.766404  0.765121\n",
              "conv1d                   0.778215   0.780752  0.778215  0.775881\n",
              "tf_hub_sentence_encoder  0.812336   0.814880  0.812336  0.810687\n",
              "tf_hub_10_percent_data   0.770341   0.775563  0.770341  0.766706\n",
              "ensemble_results         0.780840   0.780522  0.780840  0.780517"
            ]
          },
          "metadata": {},
          "execution_count": 109
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HZwqwF_swdIA"
      },
      "source": [
        "How did the stacked model go against the other models?\n",
        "\n",
        "> 🔑 **Note:** It seems many of our model's results are similar. This may mean there are some limitations to what can be learned from our data. When many of your modelling experiments return similar results, it's a good idea to revisit your data, we'll do this shortly."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UpwErZOgX_nC"
      },
      "source": [
        "## Saving and loading a trained model\n",
        "\n",
        "Although training time didn't take very long, it's good practice to save your trained models to avoid having to retrain them.\n",
        "\n",
        "Saving your models also enables you to export them for use elsewhere outside of your notebooks, such as in a web application.\n",
        "\n",
        "There are two main ways of [saving a model in TensorFlow](https://www.tensorflow.org/tutorials/keras/save_and_load#save_the_entire_model):\n",
        "1. The `HDF5` format. \n",
        "2. The `SavedModel` format (default).\n",
        "\n",
        "Let's take a look at both."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SlwjGFVyX-_T"
      },
      "source": [
        "# Save TF Hub Sentence Encoder model to HDF5 format\n",
        "model_6.save(\"model_6.h5\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cp6zvmprm9A3"
      },
      "source": [
        "If you save a model as a `HDF5`, when loading it back in, you need to let [TensorFlow know about any custom objects you've used](https://www.tensorflow.org/tutorials/keras/save_and_load#saving_custom_objects) (e.g. components which aren't built from pure TensorFlow, such as TensorFlow Hub components)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sSINZ0Q-nRb2"
      },
      "source": [
        "# Load model with custom Hub Layer (required with HDF5 format)\n",
        "loaded_model_6 = tf.keras.models.load_model(\"model_6.h5\", \n",
        "                                            custom_objects={\"KerasLayer\": hub.KerasLayer})"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G4BCJ8iXnZ4r",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4d3a7c0e-bea8-4f99-923d-943213cd72cd"
      },
      "source": [
        "# How does our loaded model perform?\n",
        "loaded_model_6.evaluate(val_sentences, val_labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 1s 14ms/step - loss: 0.4309 - accuracy: 0.8123\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.43088313937187195, 0.8123359680175781]"
            ]
          },
          "metadata": {},
          "execution_count": 112
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "02rbT4fwn0It"
      },
      "source": [
        "Calling the `save()` method on our target model and passing it a filepath allows us to save our model in the `SavedModel` format. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e3eVaNBDoMsv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a55b3bd1-ac2d-45c1-90f2-4b021c9368f9"
      },
      "source": [
        "# Save TF Hub Sentence Encoder model to SavedModel format (default)\n",
        "model_6.save(\"model_6_SavedModel_format\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:absl:Function `_wrapped_model` contains input name(s) USE_input with unsupported characters which will be renamed to use_input in the SavedModel.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO:tensorflow:Assets written to: model_6_SavedModel_format/assets\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:tensorflow:Assets written to: model_6_SavedModel_format/assets\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l-t01S-JoOqK"
      },
      "source": [
        "If you use SavedModel format (default), you can reload your model without specifying custom objects using the [`tensorflow.keras.models.load_model()`](https://www.tensorflow.org/tutorials/keras/save_and_load) function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dw3zf4fVoU5H"
      },
      "source": [
        "# Load TF Hub Sentence Encoder SavedModel\n",
        "loaded_model_6_SavedModel = tf.keras.models.load_model(\"model_6_SavedModel_format\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IqiPr6iiofi1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d4395889-dca0-4061-ba22-1723001c5c4e"
      },
      "source": [
        "# Evaluate loaded SavedModel format\n",
        "loaded_model_6_SavedModel.evaluate(val_sentences, val_labels)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 1s 14ms/step - loss: 0.4309 - accuracy: 0.8123\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.43088313937187195, 0.8123359680175781]"
            ]
          },
          "metadata": {},
          "execution_count": 115
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xzp3SHi3oQ3u"
      },
      "source": [
        "As you can see saving and loading our model with either format results in the same performance.\n",
        "\n",
        "> 🤔 **Question:** Should you used the `SavedModel` format or `HDF5` format?\n",
        "\n",
        "For most use cases, the `SavedModel` format will suffice. However, this is a TensorFlow specific standard. If you need a more general-purpose data standard, `HDF5` might be better. For more, check out the [TensorFlow documentation on saving and loading models](https://www.tensorflow.org/tutorials/keras/save_and_load)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V5a1648rG3z1"
      },
      "source": [
        "## Finding the most wrong examples\n",
        "\n",
        "We mentioned before that if many of our modelling experiments are returning similar results, despite using different kinds of models, it's a good idea to return to the data and inspect why this might be.\n",
        "\n",
        "One of the best ways to inspect your data is to sort your model's predictions and find the samples it got *most* wrong, meaning, what predictions had a high prediction probability but turned out to be wrong.\n",
        "\n",
        "Once again, visualization is your friend. Visualize, visualize, visualize.\n",
        "\n",
        "To make things visual, let's take our best performing model's prediction probabilities and classes along with the validation samples (text and ground truth labels) and combine them in a pandas DataFrame.\n",
        "\n",
        "* If our best model still isn't perfect, what examples is it getting wrong? \n",
        "* Which ones are the *most* wrong?\n",
        "* Are there some labels which are wrong? E.g. the model gets it right but the ground truth label doesn't reflect this"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gnHfX--TwMIW",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "d060505e-21c7-42b7-e664-8c517f92d425"
      },
      "source": [
        "# Create dataframe with validation sentences and best performing model predictions\n",
        "val_df = pd.DataFrame({\"text\": val_sentences,\n",
        "                       \"target\": val_labels,\n",
        "                       \"pred\": model_6_preds,\n",
        "                       \"pred_prob\": tf.squeeze(model_6_pred_probs)})\n",
        "val_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "      <th>pred</th>\n",
              "      <th>pred_prob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>DFR EP016 Monthly Meltdown - On Dnbheaven 2015...</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.144432</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>FedEx no longer to transport bioterror germs i...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.727150</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Gunmen kill four in El Salvador bus attack: Su...</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.985666</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>@camilacabello97 Internally and externally scr...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.197409</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Radiation emergency #preparedness starts with ...</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.734170</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                text  target  pred  pred_prob\n",
              "0  DFR EP016 Monthly Meltdown - On Dnbheaven 2015...       0   0.0   0.144432\n",
              "1  FedEx no longer to transport bioterror germs i...       0   1.0   0.727150\n",
              "2  Gunmen kill four in El Salvador bus attack: Su...       1   1.0   0.985666\n",
              "3  @camilacabello97 Internally and externally scr...       1   0.0   0.197409\n",
              "4  Radiation emergency #preparedness starts with ...       1   1.0   0.734170"
            ]
          },
          "metadata": {},
          "execution_count": 116
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SKJ9dTbPrIG4"
      },
      "source": [
        "Oh yeah! Now let's find our model's wrong predictions (where `target != pred`) and sort them by their prediction probability (the `pred_prob` column)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0DwBXQS1wvZx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 359
        },
        "outputId": "7d37fad0-db94-471e-fef1-3859b48c58f3"
      },
      "source": [
        "# Find the wrong predictions and sort by prediction probabilities\n",
        "most_wrong = val_df[val_df[\"target\"] != val_df[\"pred\"]].sort_values(\"pred_prob\", ascending=False)\n",
        "most_wrong[:10]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "      <th>pred</th>\n",
              "      <th>pred_prob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>? High Skies - Burning Buildings ? http://t.co...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.910481</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>759</th>\n",
              "      <td>FedEx will no longer transport bioterror patho...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.864676</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>209</th>\n",
              "      <td>Ashes 2015: AustraliaÛªs collapse at Trent Br...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.837961</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>393</th>\n",
              "      <td>@SonofLiberty357 all illuminated by the bright...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.836361</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>628</th>\n",
              "      <td>@noah_anyname That's where the concentration c...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.835225</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>@madonnamking RSPCA site multiple 7 story high...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.834875</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109</th>\n",
              "      <td>[55436] 1950 LIONEL TRAINS SMOKE LOCOMOTIVES W...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.800890</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>251</th>\n",
              "      <td>@AshGhebranious civil rights continued in the ...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.782611</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>698</th>\n",
              "      <td>åÈMGN-AFRICAå¨ pin:263789F4 åÈ Correction: Ten...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.782433</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>144</th>\n",
              "      <td>The Sound of Arson</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.771343</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                  text  target  pred  pred_prob\n",
              "31   ? High Skies - Burning Buildings ? http://t.co...       0   1.0   0.910481\n",
              "759  FedEx will no longer transport bioterror patho...       0   1.0   0.864676\n",
              "209  Ashes 2015: AustraliaÛªs collapse at Trent Br...       0   1.0   0.837961\n",
              "393  @SonofLiberty357 all illuminated by the bright...       0   1.0   0.836361\n",
              "628  @noah_anyname That's where the concentration c...       0   1.0   0.835225\n",
              "49   @madonnamking RSPCA site multiple 7 story high...       0   1.0   0.834875\n",
              "109  [55436] 1950 LIONEL TRAINS SMOKE LOCOMOTIVES W...       0   1.0   0.800890\n",
              "251  @AshGhebranious civil rights continued in the ...       0   1.0   0.782611\n",
              "698  åÈMGN-AFRICAå¨ pin:263789F4 åÈ Correction: Ten...       0   1.0   0.782433\n",
              "144                                 The Sound of Arson       0   1.0   0.771343"
            ]
          },
          "metadata": {},
          "execution_count": 117
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r3VcRHOusB2D"
      },
      "source": [
        "Finally, we can write some code to visualize the sample text, truth label, prediction class and prediction probability. Because we've sorted our samples by prediction probability, viewing samples from the head of our `most_wrong` DataFrame will show us false positives.\n",
        "\n",
        "A reminder:\n",
        "* `0` = Not a real diaster Tweet\n",
        "* `1` = Real diaster Tweet"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xLFYDEsoxRFP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "28d66210-3007-4661-e8c2-655d621f0e90"
      },
      "source": [
        "# Check the false positives (model predicted 1 when should've been 0)\n",
        "for row in most_wrong[:10].itertuples(): # loop through the top 10 rows (change the index to view different rows)\n",
        "  _, text, target, pred, prob = row\n",
        "  print(f\"Target: {target}, Pred: {int(pred)}, Prob: {prob}\")\n",
        "  print(f\"Text:\\n{text}\\n\")\n",
        "  print(\"----\\n\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Target: 0, Pred: 1, Prob: 0.9104808568954468\n",
            "Text:\n",
            "? High Skies - Burning Buildings ? http://t.co/uVq41i3Kx2 #nowplaying\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.8646755218505859\n",
            "Text:\n",
            "FedEx will no longer transport bioterror pathogens in wake of anthrax lab mishaps http://t.co/lHpgxc4b8J\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.8379608988761902\n",
            "Text:\n",
            "Ashes 2015: AustraliaÛªs collapse at Trent Bridge among worst in history: England bundled out Australia for 60 ... http://t.co/t5TrhjUAU0\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.8363614082336426\n",
            "Text:\n",
            "@SonofLiberty357 all illuminated by the brightly burning buildings all around the town!\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.8352250456809998\n",
            "Text:\n",
            "@noah_anyname That's where the concentration camps and mass murder come in. \n",
            " \n",
            "EVERY. FUCKING. TIME.\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.8348745107650757\n",
            "Text:\n",
            "@madonnamking RSPCA site multiple 7 story high rise buildings next to low density character residential in an area that floods\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.800889790058136\n",
            "Text:\n",
            "[55436] 1950 LIONEL TRAINS SMOKE LOCOMOTIVES WITH MAGNE-TRACTION INSTRUCTIONS http://t.co/xEZBs3sq0y http://t.co/C2x0QoKGlY\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.7826112508773804\n",
            "Text:\n",
            "@AshGhebranious civil rights continued in the 60s. And what about trans-generational trauma? if anything we should listen to the Americans.\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.7824334502220154\n",
            "Text:\n",
            "åÈMGN-AFRICAå¨ pin:263789F4 åÈ Correction: Tent Collapse Story: Correction: Tent Collapse story åÈ http://t.co/fDJUYvZMrv @wizkidayo\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.7713427543640137\n",
            "Text:\n",
            "The Sound of Arson\n",
            "\n",
            "----\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aXCH9J-UspWg"
      },
      "source": [
        "We can view the bottom end of our `most_wrong` DataFrame to inspect false negatives (model predicts 0, not a real diaster Tweet, when it should've predicted 1, real diaster Tweet)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6EaMchehxwLq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4c5fdb89-38b2-405a-f5d5-f816b83d3c79"
      },
      "source": [
        "# Check the most wrong false negatives (model predicted 0 when should've predict 1)\n",
        "for row in most_wrong[-10:].itertuples():\n",
        "  _, text, target, pred, prob = row\n",
        "  print(f\"Target: {target}, Pred: {int(pred)}, Prob: {prob}\")\n",
        "  print(f\"Text:\\n{text}\\n\")\n",
        "  print(\"----\\n\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Target: 1, Pred: 0, Prob: 0.06304337829351425\n",
            "Text:\n",
            "@BoyInAHorsemask its a panda trapped in a dogs body\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.06279505044221878\n",
            "Text:\n",
            "going to redo my nails and watch behind the scenes of desolation of smaug ayyy\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.06060810014605522\n",
            "Text:\n",
            "VICTORINOX SWISS ARMY DATE WOMEN'S RUBBER MOP WATCH 241487 http://t.co/yFy3nkkcoH http://t.co/KNEhVvOHVK\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.0573178268969059\n",
            "Text:\n",
            "@willienelson We need help! Horses will die!Please RT &amp; sign petition!Take a stand &amp; be a voice for them! #gilbert23 https://t.co/e8dl1lNCVu\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.04535556212067604\n",
            "Text:\n",
            "You can never escape me. Bullets don't harm me. Nothing harms me. But I know pain. I know pain. Sometimes I share it. With someone like you.\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.04145137220621109\n",
            "Text:\n",
            "I get to smoke my shit in peace\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.03926113247871399\n",
            "Text:\n",
            "@SoonerMagic_ I mean I'm a fan but I don't need a girl sounding off like a damn siren\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.0385933592915535\n",
            "Text:\n",
            "Why are you deluged with low self-image? Take the quiz: http://t.co/XsPqdOrIqj http://t.co/CQYvFR4UCy\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.03627230226993561\n",
            "Text:\n",
            "Reddit Will Now QuarantineÛ_ http://t.co/pkUAMXw6pm #onlinecommunities #reddit #amageddon #freespeech #Business http://t.co/PAWvNJ4sAP\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.032887961715459824\n",
            "Text:\n",
            "Ron &amp; Fez - Dave's High School Crush https://t.co/aN3W16c8F6 via @YouTube\n",
            "\n",
            "----\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lRKQPEAgtpJq"
      },
      "source": [
        "Do you notice anything interesting about the most wrong samples?\n",
        "\n",
        "Are the ground truth labels correct? What do you think would happen if we went back and corrected the labels which aren't?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U0W3DWgWJCWs"
      },
      "source": [
        "## Making predictions on the test dataset\n",
        "\n",
        "Alright we've seen how our model's perform on the validation set.\n",
        "\n",
        "But how about the test dataset?\n",
        "\n",
        "We don't have labels for the test dataset so we're going to have to make some predictions and inspect them for ourselves.\n",
        "\n",
        "Let's write some code to make predictions on random samples from the test dataset and visualize them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6Q9lgqoDyequ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f81e856d-8024-4471-ee0d-3c14eb128cb0"
      },
      "source": [
        "# Making predictions on the test dataset\n",
        "test_sentences = test_df[\"text\"].to_list()\n",
        "test_samples = random.sample(test_sentences, 10)\n",
        "for test_sample in test_samples:\n",
        "  pred_prob = tf.squeeze(model_6.predict([test_sample])) # has to be list\n",
        "  pred = tf.round(pred_prob)\n",
        "  print(f\"Pred: {int(pred)}, Prob: {pred_prob}\")\n",
        "  print(f\"Text:\\n{test_sample}\\n\")\n",
        "  print(\"----\\n\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pred: 1, Prob: 0.538340151309967\n",
            "Text:\n",
            "Flash Flood Watch in effect through 7:00am Thursday morning/12:00pm Thursday afternoon.\n",
            "For: Perry Wayne Cape... http://t.co/fs7vro5seS\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 1, Prob: 0.9250481128692627\n",
            "Text:\n",
            "NONSENSE &gt;&gt; famine memories  -- strong exaggeration of Ukrainian MSM\n",
            "#ukraine #russia #?????????? #sanctions https://t.co/dDOTd7W2o8\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 1, Prob: 0.8940093517303467\n",
            "Text:\n",
            "New warning for Central Hills 1' hail 60 mph winds. NOT affecting Sturgis but could later tonight. #KOTAWeather http://t.co/E8oUxVKuTE\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 0, Prob: 0.07600127905607224\n",
            "Text:\n",
            "@imaginator1dx currently reading after. as you can see after we collided is on my dresser waiting to get read http://t.co/QwrASZ6LHO\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 0, Prob: 0.026800094172358513\n",
            "Text:\n",
            "Don't ruin a good today by thinking about a bad yesterday ????\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 0, Prob: 0.21049749851226807\n",
            "Text:\n",
            "I hope I get electrocuted today at work\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 0, Prob: 0.07167388498783112\n",
            "Text:\n",
            "http://t.co/16EClWrW84 Asics GT-II Super Red 2.0 11 Ronnie Fieg Kith Red White 3M x gel grey volcano 2\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 0, Prob: 0.04366963729262352\n",
            "Text:\n",
            "I swear my eyes be bloody red but bitch I feel amazing.\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 1, Prob: 0.9714540839195251\n",
            "Text:\n",
            "Japan marks 70th anniversary of Hiroshima atomic bombing: Bells tolled in Hiroshima on Thursday as Japan marke... http://t.co/IqAIRPdIhg\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 0, Prob: 0.10844964534044266\n",
            "Text:\n",
            "@USCOURT If 90BLKs&amp;8WHTs colluded 2 take WHT F @USAgov AUTH Hostage&amp;2 make her look BLK w/Bioterrorism&amp;use her lgl/org IDis ID still hers?\n",
            "\n",
            "----\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QcvI5zgJ0Tgp"
      },
      "source": [
        "How do our model's predictions look on the test dataset?\n",
        "\n",
        "It's important to do these kind of visualization checks as often as possible to get a glance of how your model performs on unseen data and subsequently how it might perform on the real test: Tweets from the wild."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eT1jhk8xdod5"
      },
      "source": [
        "## Predicting on Tweets from the wild\n",
        "\n",
        "How about we find some Tweets and use our model to predict whether or not they're about a diaster or not?\n",
        "\n",
        "To start, let's take one of my own [Tweets on living life like an ensemble model](https://twitter.com/mrdbourke/status/1313649328351662082). "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qHmXxuPH0aUB"
      },
      "source": [
        "# Turn Tweet into string\n",
        "daniels_tweet = \"Life like an ensemble: take the best choices from others and make your own\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uPbZaGznvbEx"
      },
      "source": [
        "Now we'll write a small function to take a model and an example sentence and return a prediction."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KyH9tn9upjld"
      },
      "source": [
        "def predict_on_sentence(model, sentence):\n",
        "  \"\"\"\n",
        "  Uses model to make a prediction on sentence.\n",
        "\n",
        "  Returns the sentence, the predicted label and the prediction probability.\n",
        "  \"\"\"\n",
        "  pred_prob = model.predict([sentence])\n",
        "  pred_label = tf.squeeze(tf.round(pred_prob)).numpy()\n",
        "  print(f\"Pred: {pred_label}\", \"(real disaster)\" if pred_label > 0 else \"(not real disaster)\", f\"Prob: {pred_prob[0][0]}\")\n",
        "  print(f\"Text:\\n{sentence}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IvCG4RuUvj6d"
      },
      "source": [
        "Great! Time to test our model out."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BxONpJV8qmWP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ddd8f755-089c-4d6b-a2be-890644d10199"
      },
      "source": [
        "# Make a prediction on Tweet from the wild\n",
        "predict_on_sentence(model=model_6, # use the USE model\n",
        "                    sentence=daniels_tweet)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pred: 0.0 (not real disaster) Prob: 0.046233948320150375\n",
            "Text:\n",
            "Life like an ensemble: take the best choices from others and make your own\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tYOfNacw08Of"
      },
      "source": [
        "Woohoo! Our model predicted correctly. My Tweet wasn't about a diaster.\n",
        "\n",
        "How about we find a few Tweets about actual diasters?\n",
        "\n",
        "Such as the following two Tweets about the 2020 Beirut explosions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AqILBsTK2i9R"
      },
      "source": [
        "# Source - https://twitter.com/BeirutCityGuide/status/1290696551376007168\n",
        "beirut_tweet_1 = \"Reports that the smoke in Beirut sky contains nitric acid, which is toxic. Please share and refrain from stepping outside unless urgent. #Lebanon\"\n",
        "\n",
        "# Source - https://twitter.com/BeirutCityGuide/status/1290773498743476224\n",
        "beirut_tweet_2 = \"#Beirut declared a “devastated city”, two-week state of emergency officially declared. #Lebanon\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FvlbHDISrVmX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6be0b58b-4ac6-4948-cd1e-1d43f4ddb159"
      },
      "source": [
        "# Predict on diaster Tweet 1\n",
        "predict_on_sentence(model=model_6, \n",
        "                    sentence=beirut_tweet_1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pred: 1.0 (real disaster) Prob: 0.9625465869903564\n",
            "Text:\n",
            "Reports that the smoke in Beirut sky contains nitric acid, which is toxic. Please share and refrain from stepping outside unless urgent. #Lebanon\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5uKYx11p2zCd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6b4b7408-f56d-49ed-ff41-584dfe65cf1b"
      },
      "source": [
        "# Predict on diaster Tweet 2\n",
        "predict_on_sentence(model=model_6, \n",
        "                    sentence=beirut_tweet_2)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pred: 1.0 (real disaster) Prob: 0.9678557515144348\n",
            "Text:\n",
            "#Beirut declared a “devastated city”, two-week state of emergency officially declared. #Lebanon\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fczP1dFcwe98"
      },
      "source": [
        "Looks like our model is performing as expected, predicting both of the diaster Tweets as actual diasters.\n",
        "\n",
        "> 🔑 **Note:** The above examples are cherry-picked and are cases where you'd expect a model to function at high performance. For actual production systems, you'll want to continaully perform tests to see how your model is performing."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fp0fkK-tHPRE"
      },
      "source": [
        "## The speed/score tradeoff\n",
        "\n",
        "One of the final tests we're going to do is to find the speed/score tradeoffs between our best model and baseline model.\n",
        "\n",
        "Why is this important?\n",
        "\n",
        "Although it can be tempting to just choose the best performing model you find through experimentation, this model might not actually work in a production setting.\n",
        "\n",
        "Put it this way, imagine you're Twitter and receive 1 million Tweets per hour (this is a made up number, the actual number is much higher). And you're trying to build a diaster detection system to read Tweets and alert authorities with details about a diaster in close to real-time.\n",
        "\n",
        "Compute power isn't free so you're limited to a single compute machine for the project. On that machine, one of your models makes 10,000 predictions per second at 80% accuracy where as another one of your models (a larger model) makes 100 predictions per second at 85% accuracy.\n",
        "\n",
        "Which model do you choose?\n",
        "\n",
        "Is the second model's performance boost worth missing out on the extra capacity?\n",
        "\n",
        "Of course, there are many options you could try here, such as sending as many Tweets as possible to the first model and then sending the ones which the model is least certain of to the second model. \n",
        "\n",
        "The point here is to illustrate the best model you find through experimentation, might not be the model you end up using in production.\n",
        "\n",
        "To make this more concrete, let's write a function to take a model and a number of samples and time how long the given model takes to make predictions on those samples."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DnXp8DKOp3J6"
      },
      "source": [
        "# Calculate the time of predictions\n",
        "import time\n",
        "def pred_timer(model, samples):\n",
        "  \"\"\"\n",
        "  Times how long a model takes to make predictions on samples.\n",
        "  \n",
        "  Args:\n",
        "  ----\n",
        "  model = a trained model\n",
        "  sample = a list of samples\n",
        "\n",
        "  Returns:\n",
        "  ----\n",
        "  total_time = total elapsed time for model to make predictions on samples\n",
        "  time_per_pred = time in seconds per single sample\n",
        "  \"\"\"\n",
        "  start_time = time.perf_counter() # get start time\n",
        "  model.predict(samples) # make predictions\n",
        "  end_time = time.perf_counter() # get finish time\n",
        "  total_time = end_time-start_time # calculate how long predictions took to make\n",
        "  time_per_pred = total_time/len(val_sentences) # find prediction time per sample\n",
        "  return total_time, time_per_pred"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GxWwS73hze6Z"
      },
      "source": [
        "Looking good!\n",
        "\n",
        "Now let's use our `pred_timer()` function to evaluate the prediction times of our best performing model (`model_6`) and our baseline model (`model_0`)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JMbGMIWd5c9N",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4a151071-956a-4c97-fd95-18392a57a78b"
      },
      "source": [
        "# Calculate TF Hub Sentence Encoder prediction times\n",
        "model_6_total_pred_time, model_6_time_per_pred = pred_timer(model_6, val_sentences)\n",
        "model_6_total_pred_time, model_6_time_per_pred"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.3529780789999677, 0.0004632258254592752)"
            ]
          },
          "metadata": {},
          "execution_count": 128
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I4ej2VyT5oQs",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cfbedfa1-282e-4b04-b2c4-1a3b33771b1e"
      },
      "source": [
        "# Calculate Naive Bayes prediction times\n",
        "baseline_total_pred_time, baseline_time_per_pred = pred_timer(model_0, val_sentences)\n",
        "baseline_total_pred_time, baseline_time_per_pred"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.018752853000023606, 2.4610043307117593e-05)"
            ]
          },
          "metadata": {},
          "execution_count": 129
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nqNnKMxhz8Kl"
      },
      "source": [
        "It seems with our current hardware (in my case, I'm using a Google Colab notebook) our best performing model takes over 10x the time to make predictions as our baseline model.\n",
        "\n",
        "Is that extra prediction time worth it?\n",
        "\n",
        "Let's compare time per prediction versus our model's F1-scores."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ANKHEfRN7Nhd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 458
        },
        "outputId": "a4ca1fff-0b84-41d0-85fd-b7f258d47558"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.figure(figsize=(10, 7))\n",
        "plt.scatter(baseline_time_per_pred, baseline_results[\"f1\"], label=\"baseline\")\n",
        "plt.scatter(model_6_time_per_pred, model_6_results[\"f1\"], label=\"tf_hub_sentence_encoder\")\n",
        "plt.legend()\n",
        "plt.title(\"F1-score versus time per prediction\")\n",
        "plt.xlabel(\"Time per prediction\")\n",
        "plt.ylabel(\"F1-Score\");"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm4AAAG5CAYAAAA3e7gZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5hXZb3//+dbRDHNM+1SUDAVRRg5iccSNcPSrVZquLWvh8rM7Bylu9yZ5fen2S93GqbWVtpa4jEjraQUU9PUYaMoIorKFtAUETQIlMP7+8dnzfhhnBMyn5lZ8Hxc17pmHe51r3utNTgv73WKzESSJEnd3wZd3QBJkiS1j8FNkiSpJAxukiRJJWFwkyRJKgmDmyRJUkkY3CRJkkrC4CZJayAi/j0iftHV7ejuImJURMytmp4eEaPeQT0fiIiZHdo4qcQMblI3FBGzI2JpRCyuGrYrll0ZETMjYlVEnNzFTV2nNQ0fAJn5fzPzM13VprLKzD0y8+62ykVERsTOVevdm5kDato4qUQMblL39a+ZuVnV8EIx/1HgDOB/urBtAETEhuvjtsumI45VRPToiLZIWjsGN6lkMnNcZt4JLGurbET0iohrI2JBRCyKiIcj4l+KZVtHxNUR8UJELIyIW6vW+2xEzIqIVyNiYkNvX7EsI+ILEfE08HQx74iIeKTYxv0RUddCe34WET9qMu+3EfG1Yny7iLg5IuZHxHMR8aWqcudGxE3F/rwOnBwRIyOiPiJej4iXIuLHRdm39ZQVvZgfKsabXa9J+U2BPwDbVfd6Fu24tijTrzgep0TEnOI4nh4Re0XEtOJ4/LRJvadGxIyi7B0RsWMLx6qh7tOKc/RiRHyjavkGEXFWRDxTnN8bImLrJut+OiKeB+5qpv5RETG3uPT7SnF8TqhaPr44X7+PiCXAQW2cn02KdRZGxBPAXq0c/x7Fdp+JiH9ExJSI6BsR9xTFHy2O9yebnsuI2D0i7i6O7fSIOLJJm8dFxO1FvQ9GxPubO75SaWWmg4NDNxuA2cCH2ihzH3ByG2U+B/wOeBfQAxgObF4sux24HtgK6AkcWMw/GHgFGAZsDFwK3FNVZwJ/ArYGNgGGAi8DexfbOKlo/8bNtOeDwBwgiumtgKXAdlT+R3IK8B/ARsBOwLPA6KLsucBy4Oii7CbAA8CniuWbAfsU46OAuS0d05bWa6a9zdVzLnBtMd6vOB6XA72AD1MJ1LcC7wG2L45Nw7E9CpgF7A5sCHwHuL+FbTfUfR2wKTAYmF+1D18G/gb0Kc7TFcB1Tdb972LdTVrYtxXAj4v1DwSWAAOK5eOB14D9i+P9rjbOzwXAvcXvRV/g8epj1+T4jwUeAwYAAewJbFP1+7Vzc+eAyu/pLODfizYcDPyjSZsXACOL4/srYEJX/3t2cOjIwR43qfu6tehVWFTdG7aGlgPbUPlDuDIzp2Tm6xHxPuAjwOmZuTAzl2fmX4p1TgCuysz/ycw3gLOBfSOiX1W9/19mvpqZS4HTgCsy88FiG78E3gD2aaY991L5w/yBYvoY4IGsXAbeC+idmedl5puZ+Szwc2BM1foPZOatmbmq2PZyYOeI2DYzF2fm39bguLyT9Vry/cxclpmTqISf6zLz5cycV+zz0KLc6VSO3YzMXAH8X2BIS71uhe9l5pLMfAy4Gji+qq5vZ+bc4jydCxwTq18WPbdYd2kr9Z+TmW8U5/924LiqZb/NzL9m5ioqwbG183MccH7xezEHuKSVbX4G+E5mzsyKRzNzQSvlG+xDJWhfULThLuC2qmMC8JvMfKg4vr8ChrSjXqk0DG5S93V0Zm5ZDEe3Z4VY/WGGHYBrgDuACcXlth9GRE8qPSKvZubCZqrZDvjfhonMXEylF2P7qjJzqsZ3BL5eFTIXFfVvRxOZmcAE3vpD+29U/rg21LNdk3r+HfiXFrYL8GlgV+DJqFwGPqKlY9NB67Xkparxpc1Mb1aM7wj8pGr/XqXS41R9bJuq3uf/5a3juiPwm6q6ZgAraf14NbUwM5e0UH/T9ds6P9s109aW9AWeaaNtzdkOmFMEyertVB+/v1eN/5O3jr20TvDmXmkdkpnN/ZH6HvC9osfs98DM4ufWEbFlZi5qUv4FKn+kgcZ7vbYB5lVvqmp8DpWelvPb2czrgEkRcQGVy6sfq6rnuczcpZV1c7WJzKeB4yNiA+DjwE0RsQ2VXq93Ve1DD6B3W+s1CTFv214HaDhWv2qz5Fv6Ak8W4ztQOT8NdZ2amX9tukJV72hb7d8qIjat2u8dqFzibND0PLd2fl4s2jq9qq6WzAHe32Rb7fEC0DciNqgKbzsAT61hPVJp2eMmlUxEbBQRvaj01PSMygMIzf5bjoiDImJwEVxep3KJcFVmvkjlxvvLImKriOgZER8sVrsOOCUihkTExlQu5z2YmbNbaNLPgdMjYu+o2DQiDo+IdzdXODOnUrmH7hfAHVXB8SHgHxHxreJG9x4RMSgi9mqunmL/ToyI3sUf8YZ6VlH5Q96raEdPKveSbdyO9Zp6CdgmIrZoqQ1r6HLg7IjYo2jHFhFxbBvrnBMR7yrWOYXKfYkNdZ3fcJk1InpHxFHvoE3fK36nPgAcAdzYQrm2zs8Nxb5tFRF9gC+2ss1fAN+PiF2K35m6InBD5Zjv1MJ6D1LpRftm8Ts7CvhXKr240nrB4CaVzyQql9/2A64sxj/YQtn3AjdRCW0zgL9QuXwK8CkqQe5JKjfQfwUgM/8MnAPcTKUX5f2sfp/ZajKzHvgs8FNgIZWbx09uYx9+DXyo+NlQz0oqwWEI8BxvhbvWQtNhwPSIWAz8BBiTmUsz8zUqr0z5BZWewiXA3LbWa2bfnqQSZJ8tLg++7fLvmsjM3wAXUrl0/TqVHqePtLHaX6gc0zuBHxX30VG0eyKV3st/UHlQYe81bNLfqZyzF6hcsj692Ofm2t7W+fkelcuWz1H5Hb2mmWoa/JhK0JtE5Xfzv6g8bAKVe/V+WRzv6vvtyMw3qQS1jxTbvwz4Py21WVoXNTzZJUnqRorLnc8BPYsb7Tu6/lFUno7t09F1S6ode9wkSZJKwuAmSZJUEl4qlSRJKgl73CRJkkpivXiP27bbbpv9+vXr6mZIkiS1acqUKa9kZu/mlq0Xwa1fv37U19d3dTMkSZLaFBEtfnnES6WSJEklYXCTJEkqCYObJElSSawX97g1Z/ny5cydO5dly5Z1dVO0nuvVqxd9+vShZ8+eXd0USVI3t94Gt7lz5/Lud7+bfv36ERFd3RytpzKTBQsWMHfuXPr379/VzZEkdXPr7aXSZcuWsc022xja1KUigm222caeX0lSu6y3wQ0wtKlb8PdQktRe63VwkyRJKhODWxeaPXs2gwYNqkndd999N0cccQQAEydO5IILLqjJdiRJUudZbx9OWJ8ceeSRHHnkkV3dDEmStJbscWunW6fOY/8L7qL/Wbez/wV3cevUeR1S74oVKzjhhBPYfffdOeaYY/jnP//Jeeedx1577cWgQYM47bTTyEwALrnkEgYOHEhdXR1jxowBYMmSJZx66qmMHDmSoUOH8tvf/vZt2xg/fjxnnnkmACeffDJf+tKX2G+//dhpp5246aabGstddNFF7LXXXtTV1fHd7363Q/ZPkiR1HINbO9w6dR5n3/IY8xYtJYF5i5Zy9i2PdUh4mzlzJmeccQYzZsxg880357LLLuPMM8/k4Ycf5vHHH2fp0qXcdtttAFxwwQVMnTqVadOmcfnllwNw/vnnc/DBB/PQQw8xefJkxo4dy5IlS1rd5osvvsh9993HbbfdxllnnQXApEmTePrpp3nooYd45JFHmDJlCvfcc89a758kSeo4Brd2uOiOmSxdvnK1eUuXr+SiO2audd19+/Zl//33B+DEE0/kvvvuY/Lkyey9994MHjyYu+66i+nTpwNQV1fHCSecwLXXXsuGG1auck+aNIkLLriAIUOGMGrUKJYtW8bzzz/f6jaPPvpoNthgAwYOHMhLL73UWM+kSZMYOnQow4YN48knn+Tpp59e6/2TJEkdx3vc2uGFRUvXaP6aaPoqiIjgjDPOoL6+nr59+3Luuec2vuPr9ttv55577uF3v/sd559/Po899hiZyc0338yAAQNWq6chkDVn4403bhxvuAybmZx99tl87nOfW+t9kiRpnTLtBrjzPHhtLmzRBw75D6g7rkuaYo9bO2y35SZrNH9NPP/88zzwwAMA/PrXv+aAAw4AYNttt2Xx4sWN96CtWrWKOXPmcNBBB3HhhRfy2muvsXjxYkaPHs2ll17aGMCmTp36jtoxevRorrrqKhYvXgzAvHnzePnll9d29yRJKrdpN8DvvgSvzQGy8vN3X6rM7wL2uLXD2NEDOPuWx1a7XLpJzx6MHT2glbXaZ8CAAYwbN45TTz2VgQMH8vnPf56FCxcyaNAg3vve97LXXnsBsHLlSk488URee+01MpMvfelLbLnllpxzzjl85Stfoa6ujlWrVtG/f//Ge+LWxIc//GFmzJjBvvvuC8Bmm23Gtddey3ve85613kdJkkrrzvNgeZMrbMuXVuZ3Qa9bNPTUrMtGjBiR9fX1q82bMWMGu+++e7vruHXqPC66YyYvLFrKdltuwtjRAzh66PYd3VStp9b091GS1EnO3RJoLisFnLuoJpuMiCmZOaK5Zfa4tdPRQ7c3qEmStL7Zok9xmbSZ+V3Ae9wkSZJacsh/QM8m97T33KQyvwsY3CRJklpSdxz86yWwRV8gKj//9ZIue6rUS6WSJEmtqTuuy4JaU/a4SZIklURNg1tEHBYRMyNiVkSc1czyHSJickRMjYhpEfHRYv42xfzFEfHTJusMj4jHijoviaZvsJUkSVpH1Sy4RUQPYBzwEWAgcHxEDGxS7DvADZk5FBgDXFbMXwacA3yjmap/BnwW2KUYDuv41kuSJHU/texxGwnMysxnM/NNYAJwVJMyCWxejG8BvACQmUsy8z4qAa5RRLwP2Dwz/5aVF9D9N3B0DfehZhYtWsRll13WOD127Fj22GMPxo4d22z5k08+ufErCu3Vr18/XnnllbVq55r6z//8T/75z3926ja70t13380RRxzR1c2QJK0nahnctgeqX3wyt5hX7VzgxIiYC/we+GI76pzbRp0ARMRpEVEfEfXz589fk3Y3b9oNcPGgyov4Lh601p+6aBrcrrzySqZNm8ZFF120ti3tUutbcFtTK1as6OomSJJKrKsfTjgeGJ+ZfYCPAtdERIe0KTOvzMwRmTmid+/ea1dZDb5TdtZZZ/HMM88wZMgQDj30UBYvXszw4cO5/vrrW1znnnvuYb/99mOnnXZq7H1r2uNz5plnMn78+MbpH/7whwwePJiRI0cya9asFuu+8cYbGTRoEHvuuScf/OAHgcpntsaOHctee+1FXV0dV1xxReM2R40axTHHHMNuu+3GCSecQGZyySWX8MILL3DQQQdx0EEHATBp0iT23Xdfhg0bxrHHHtv4LdR+/frx3e9+l2HDhjF48GCefPJJABYvXswpp5zC4MGDqaur4+abb261nuZMmTKFAw88kOHDhzN69GhefPFFAEaNGsW3vvUtRo4cya677sq9997buJ/f+MY3GDRoEHV1dVx66aUA3HnnnQwdOpTBgwdz6qmn8sYbbwDwxz/+kd12241hw4Zxyy23NG53yZIlnHrqqYwcOZKhQ4fy29/+FoDx48dz5JFHcvDBB3PIIYe02G5JktqUmTUZgH2BO6qmzwbOblJmOtC3avpZ4D1V0ycDP62afh/wZNX08cAVbbVl+PDh2dQTTzzxtnkt+vEemd/d/O3Dj/dofx1NPPfcc7nHHm+tv+mmm7Za/qSTTspjjjkmV65cmdOnT8/3v//9mZk5efLkPPzwwxvLfeELX8irr746MzN33HHH/MEPfpCZmb/85S9XK9fUoEGDcu7cuZmZuXDhwszMvOKKK/L73/9+ZmYuW7Yshw8fns8++2xOnjw5N99885wzZ06uXLky99lnn7z33nsbtzl//vzMzJw/f35+4AMfyMWLF2dm5gUXXJDf+973GstdcsklmZk5bty4/PSnP52Zmd/85jfzy1/+cmO7Xn311VbraerNN9/MfffdN19++eXMzJwwYUKecsopmZl54IEH5te+9rXMzLz99tvzkEMOyczMyy67LD/xiU/k8uXLMzNzwYIFuXTp0uzTp0/OnDkzMzM/9alP5cUXX9w4/6mnnspVq1blscce23hczz777Lzmmmsaj+Euu+ySixcvzquvvjq33377XLBgQYvHf41+HyVJ6zSgPlvINLV8j9vDwC4R0R+YR+Xhg39rUuZ54BBgfETsDvQCWryumZkvRsTrEbEP8CDwf4BLa9H41bw2d83m18jRRx/NBhtswMCBA3nppZfatc7xxx/f+POrX/1qi+X2339/Tj75ZI477jg+/vGPA5VermnTpjX27r322ms8/fTTbLTRRowcOZI+fSqf+xgyZAizZ8/mgAMOWK3Ov/3tbzzxxBPsv//+ALz55puNH7EHGrczfPjwxp6rP//5z0yYMKGxzFZbbcVtt93Waj3VZs6cyeOPP86hhx4KVHrT3ve+9zW7zdmzZzdu8/TTT2fDDSv/HLbeemseffRR+vfvz6677grASSedxLhx4xg1ahT9+/dnl112AeDEE0/kyiuvbDxeEydO5Ec/+hEAy5Yt4/nnnwfg0EMPZeutt27x+EuS1B41C26ZuSIizgTuAHoAV2Xm9Ig4j0qSnAh8Hfh5RHyVyoMKJxdJk4iYTeXBhY0i4mjgw5n5BHAGMB7YBPhDMdRWN/lO2cYbb9w4XhwmNtxwQ1atWtU4f9my1Z7noPptKa29OeXyyy/nwQcf5Pbbb2f48OFMmTKFzOTSSy9l9OjRq5W9++67V2tLjx49mr13KzM59NBDue6661rdn5bWb289TcvusccePPDAA2u1zXciM7n55psZMGDAavMffPBBNt100w7dliRp/VTTe9wy8/eZuWtmvj8zzy/m/UcR2sjMJzJz/8zcMzOHZOakqnX7ZebWmblZZvYpQhuZWZ+Zg4o6z2wIejVVg++Uvfvd7+Yf//jHWjYMdtxxR5544gneeOMNFi1axJ133rna8oZ75q6//voWe6kAnnnmGfbee2/OO+88evfuzZw5cxg9ejQ/+9nPWL58OQBPPfUUS5YsabU91fu1zz778Ne//rXx3rolS5bw1FNPtbr+oYceyrhx4xqnFy5cuEb1DBgwgPnz5zcGt+XLlzN9+vQ2t3nFFVc0BrlXX32VAQMGMHv27MZtXnPNNRx44IHstttuzJ49m2eeeQZgtTA5evRoLr300sZQPXXq1Fa3K0nSmurqhxPKoQbfKdtmm23Yf//9GTRoUIuvAGmPvn37ctxxxzFo0CCOO+44hg4dutryhQsXUldXx09+8hMuvvjiFusZO3YsgwcPZtCgQey3337sueeefOYzn2HgwIEMGzaMQYMG8bnPfa7NXqrTTjuNww47jIMOOojevXszfvx4jj/+eOrq6th3330bH0JoyXe+8x0WLlzY+KDE5MmT16iejTbaiJtuuolvfetb7LnnngwZMoT777+/1W1+5jOfYYcddqCuro4999yTX//61/Tq1Yurr76aY489lsGDB7PBBhtw+umn06tXL6688koOP/xwhg0bxnve857Ges455xyWL19OXV0de+yxB+ecc06r25UkaU1FZ3RYdbURI0ZkfX39avNmzJjB7rvv3kUtklbn76MkqUFETMnMEc0ts8dNkiSpJGr5VKnegfPPP58bb7xxtXnHHnss3/72t0tRf2f62Mc+xnPPPbfavAsvvPBtD1NIkrSuWK8vle62226tPmkpdYbM5Mknn/RSqSQJ8FJps3r16sWCBQtYH4Kruq/MZMGCBfTq1aurmyJJKoH19lJpnz59mDt3Lh3yHVNpLfTq1avxZcaSJLVmvQ1uPXv2pH///l3dDEmSpHZbby+VSpIklY3BTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKoqbBLSIOi4iZETErIs5qZvkOETE5IqZGxLSI+GjVsrOL9WZGxOiq+bMj4rGIeCQi6mvZfkmSpO5kw1pVHBE9gHHAocBc4OGImJiZT1QV+w5wQ2b+LCIGAr8H+hXjY4A9gO2AP0fErpm5sljvoMx8pVZtlyRJ6o5q2eM2EpiVmc9m5pvABOCoJmUS2LwY3wJ4oRg/CpiQmW9k5nPArKI+SZKk9VYtg9v2wJyq6bnFvGrnAidGxFwqvW1fbMe6CUyKiCkRcVpLG4+I0yKiPiLq58+f/873QpIkqZvo6ocTjgfGZ2Yf4KPANRHRVpsOyMxhwEeAL0TEB5srlJlXZuaIzBzRu3fvjm21JElSF6hlcJsH9K2a7lPMq/Zp4AaAzHwA6AVs29q6mdnw82XgN3gJVZIkrSdqGdweBnaJiP4RsRGVhw0mNinzPHAIQETsTiW4zS/KjYmIjSOiP7AL8FBEbBoR7y7Kbwp8GHi8hvsgSZLUbdTsqdLMXBERZwJ3AD2AqzJzekScB9Rn5kTg68DPI+KrVO5dOzkzE5geETcATwArgC9k5sqI+BfgNxHR0PZfZ+Yfa7UPkiRJ3UlUctK6bcSIEVlf7yvfJElS9xcRUzJzRHPLuvrhBEmSJLWTwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSqKmwS0iDouImRExKyLOamb5DhExOSKmRsS0iPho1bKzi/VmRsTo9tYpSZK0rqpZcIuIHsA44CPAQOD4iBjYpNh3gBsycygwBrisWHdgMb0HcBhwWUT0aGedkiRJ66Ra9riNBGZl5rOZ+SYwATiqSZkENi/GtwBeKMaPAiZk5huZ+Rwwq6ivPXVKkiStk2oZ3LYH5lRNzy3mVTsXODEi5gK/B77YxrrtqROAiDgtIuojon7+/PnvdB8kSZK6ja5+OOF4YHxm9gE+ClwTER3Spsy8MjNHZOaI3r17d0SVkiRJXWrDGtY9D+hbNd2nmFft01TuYSMzH4iIXsC2bazbVp2SJEnrpHb1bkXErhFxZ0Q8XkzXRcR32ljtYWCXiOgfERtRedhgYpMyzwOHFHXuDvQC5hflxkTExhHRH9gFeKiddUqSJK2T2ntZ8ufA2cBygMycRiU0tSgzVwBnAncAM6g8PTo9Is6LiCOLYl8HPhsRjwLXASdnxXTgBuAJ4I/AFzJzZUt1tn93JUmSyisys+1CEQ9n5l4RMbV4dQcR8UhmDql5CzvAiBEjsr6+vqubIUmS1KaImJKZI5pb1t4et1ci4v1UXt9BRBwDvNhB7ZMkSVI7tPfhhC8AVwK7RcQ84DnghJq1SpIkSW/TZnArvlZwRmZ+KCI2BTbIzH/UvmmSJEmq1mZwy8yVEXFAMb6k9k2SJElSc9p7qXRqREwEbgQaw1tm3lKTVkmSJOlt2hvcegELgIOr5iVgcJMkSeok7QpumXlKrRsiSZKk1rX3ywl9IuI3EfFyMdwcEX1q3ThJkiS9pb3vcbuayqeltiuG3xXzJEmS1EnaG9x6Z+bVmbmiGMYDvWvYLkmSJDXR3uC2ICJOjIgexXAilYcVJEmS1EnaG9xOBY4D/k7lU1fHAD6wIEmS1Ina+1Tp/wJH1rgtkiRJakV7nyr9ZURsWTW9VURcVbtmSZIkqan2Xiqty8xFDROZuRAYWpsmSZIkqTntDW4bRMRWDRMRsTXt/+qCJEmSOkB7w9f/DzwQETcCQeXhhPNr1ipJkiS9TXsfTvjviKin8q3SBD6emU/UtGWSJElaTauXSiPiXRHRE6AIan8CNgJ264S2SZIkqUpb97j9EegHEBE7Aw8AOwFfiIgLats0SZIkVWsruG2VmU8X4ycB12XmF4GPAIfXtGWSJElaTVvBLavGD6ZyqZTMfBNYVatGSZIk6e3aejhhWkT8CJgH7AxMAqh+Ga8kSZI6R1s9bp8FXqFyn9uHM/OfxfyBwI9q2C5JkiQ10WqPW2YuBVZ7CCEihmXm/cD9tWyYJEmSVtfeLydU+0WHt0KSJElteifBLTq8FZIkSWrTOwlu3+vwVkiSJKlNaxzcMvNWgIjw6wmSJEmd6J30uDWY1GGtkCRJUptafao0Ii5paRHgu9wkSZI6UVsv4D0F+DrwRjPLju/45kiSJKklbQW3h4HHi/e2rSYizq1JiyRJkujB3woAABCGSURBVNSstoLbMcCy5hZkZv+Ob44kSZJa0tbDCZtVfeZKkiRJXait4HZrw0hE3FzjtkiSJKkVbQW36q8k7FTLhkiSJKl1bQW3bGFckiRJnaythxP2jIjXqfS8bVKMU0xnZm5e09ZJkiSpUavBLTN7dFZDJEmS1Lq1+eSVJEmSOpHBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJ1DS4RcRhETEzImZFxFnNLL84Ih4phqciYlHVsgsj4vFi+GTV/PER8VzVekNquQ+SJEndxYa1qjgiegDjgEOBucDDETExM59oKJOZX60q/0VgaDF+ODAMGAJsDNwdEX/IzNeL4mMz86ZatV2SJKk7qmWP20hgVmY+m5lvAhOAo1opfzxwXTE+ELgnM1dk5hJgGnBYDdsqSZLU7dUyuG0PzKmanlvMe5uI2BHoD9xVzHoUOCwi3hUR2wIHAX2rVjk/IqYVl1o3bqHO0yKiPiLq58+fv7b7IkmS1OW6y8MJY4CbMnMlQGZOAn4P3E+lF+4BYGVR9mxgN2AvYGvgW81VmJlXZuaIzBzRu3fvGjdfkiSp9moZ3Oaxei9Zn2Jec8bw1mVSADLz/MwckpmHAgE8Vcx/MSveAK6mcklWkiRpnVfL4PYwsEtE9I+IjaiEs4lNC0XEbsBWVHrVGub1iIhtivE6oA6YVEy/r/gZwNHA4zXcB0mSpG6jZk+VZuaKiDgTuAPoAVyVmdMj4jygPjMbQtwYYEJmZtXqPYF7K9mM14ETM3NFsexXEdGbSi/cI8DptdoHSZKk7iRWz0vrphEjRmR9fX1XN0OSJKlNETElM0c0t6y7PJwgSZKkNhjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkahrcIuKwiJgZEbMi4qxmll8cEY8Uw1MRsahq2YUR8XgxfLJqfv+IeLCo8/qI2KiW+yBJktRd1Cy4RUQPYBzwEWAgcHxEDKwuk5lfzcwhmTkEuBS4pVj3cGAYMATYG/hGRGxerHYhcHFm7gwsBD5dq32QJEnqTmrZ4zYSmJWZz2bmm8AE4KhWyh8PXFeMDwTuycwVmbkEmAYcFhEBHAzcVJT7JXB0TVovSZLUzdQyuG0PzKmanlvMe5uI2BHoD9xVzHqUSlB7V0RsCxwE9AW2ARZl5op21HlaRNRHRP38+fPXemckSZK6Wnd5OGEMcFNmrgTIzEnA74H7qfTCPQCsXJMKM/PKzByRmSN69+7d0e2VJEnqdLUMbvOo9JI16FPMa84Y3rpMCkBmnl/c/3YoEMBTwAJgy4jYsB11SpIkrVNqGdweBnYpngLdiEo4m9i0UETsBmxFpVetYV6PiNimGK8D6oBJmZnAZOCYouhJwG9ruA+SJEndxoZtF3lnMnNFRJwJ3AH0AK7KzOkRcR5Qn5kNIW4MMKEIZQ16AvdWnkXgdeDEqvvavgVMiIgfAFOB/6rVPkiSJHUnsXpeWjeNGDEi6+vru7oZkiRJbYqIKZk5orll3eXhBEmSJLXB4CZJklQSBjdJkqSSMLhJkiSVhMFNkiSpJAxukiRJJWFwkyRJKgmDmyRJUkkY3CRJkkrC4CZJklQSBjdJkqSSMLhJkiSVhMFNkiSpJAxukiRJJWFwkyRJKgmDmyRJUkkY3CRJkkrC4CZJklQSBjdJkqSSMLhJkiSVhMFNkiSpJAxukiRJJWFwkyRJKgmDmyRJUkkY3CRJkkrC4CZJklQSBjdJkqSSMLhJkiSVhMFNkiSpJAxukiRJJWFwkyRJKgmDmyRJUkkY3CRJkkrC4CZJklQSBjdJkqSSMLhJkiSVhMFNkiSpJAxukiRJJWFwkyRJKgmDmyRJUkkY3CRJkkrC4CZJklQSBjdJkqSSMLhJkiSVhMFNkiSpJDbs6gaU3a1T53HRHTN5YdFStttyE8aOHsDRQ7fv6mZJkqR1kMFtLdw6dR5n3/IYS5evBGDeoqWcfctjAIY3SZLU4bxUuhYuumNmY2hrsHT5Si66Y2YXtUiSJK3LDG5r4YVFS9doviRJ0towuK2F7bbcZI3mS5IkrQ2D21oYO3oAm/Tssdq8TXr2YOzoAV3UIkmStC7z4YS10PAAgk+VSpKkzmBwW0tHD93eoCZJkjqFl0olSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKoqbBLSIOi4iZETErIs5qZvnFEfFIMTwVEYuqlv0wIqZHxIyIuCQioph/d1Fnw3rvqeU+SJIkdRc1++RVRPQAxgGHAnOBhyNiYmY+0VAmM79aVf6LwNBifD9gf6CuWHwfcCBwdzF9QmbW16rtkiRJ3VEte9xGArMy89nMfBOYABzVSvnjgeuK8QR6ARsBGwM9gZdq2FZJkqRur5Yfmd8emFM1PRfYu7mCEbEj0B+4CyAzH4iIycCLQAA/zcwZVatcHRErgZuBH2RmNlPnacBpxeTiiJi5lvujrrMt8EpXN0IdxvO5bvF8rjs8l93Hji0tqGVwWxNjgJsycyVAROwM7A70KZb/KSI+kJn3UrlMOi8i3k0luH0K+O+mFWbmlcCVndJ61VRE1GfmiK5uhzqG53Pd4vlcd3guy6GWl0rnAX2rpvsU85ozhrcukwJ8DPhbZi7OzMXAH4B9ATJzXvHzH8CvqVySlSRJWufVMrg9DOwSEf0jYiMq4Wxi00IRsRuwFfBA1ezngQMjYsOI6EnlwYQZxfS2xXo9gSOAx2u4D5IkSd1GzYJbZq4AzgTuAGYAN2Tm9Ig4LyKOrCo6BpjQ5D61m4BngMeAR4FHM/N3VB5UuCMipgGPUOnB+3mt9kHdhpe81y2ez3WL53Pd4bksgWjmvn5JkiR1Q345QZIkqSQMbpIkSSVhcFOnaMfnzzaOiOuL5Q9GRL+qZWcX82dGxOi26oyIM4t52fAwizpOJ5/LXxXzH4+Iq4qHktSBOvl8/ldEPBoR0yLipojYrNb7t77pzPNZtfySiFhcq31SE5np4FDTAehB5WGTnah8DeNRYGCTMmcAlxfjY4Dri/GBRfmNqbyk+ZmivhbrpPLptH7AbGDbrt7/dWnognP5USov4Q4qrwz6fFcfg3Vp6ILzuXlVvT8GzurqY7AuDZ19Pov1RgDXAIu7ev/Xl8EeN3WG9nz+7Cjgl8X4TcAhERHF/AmZ+UZmPgfMKuprsc7MnJqZs2u9U+upzj6Xv88C8BBvvZRbHaOzz+frAMX6m1D5vKE6Tqeez+Kb5BcB36zxfqmKwU2dobnPn23fUpmsvErmNWCbVtZtT53qeF1yLotLpJ8C/rjWe6BqnX4+I+Jq4O/AbsClHbETatTZ5/NMYGJmvthB7Vc7GNwklcFlwD1Z+eydSiwzTwG2o/J+z092cXP0DkXEdsCxGL47ncFNnaE9nz9rLBMRGwJbAAtaWXdNPqmmjtPp5zIivgv0Br7WIXugal3ybzMr36WeAHxirfdA1TrzfA4FdgZmRcRs4F0RMaujdkQtM7ipM7Tn82cTgZOK8WOAu4r7miYCY4onofoDu1C516ldn1RTh+vUcxkRnwFGA8dn5qoa79v6qNPOZ1TsDI33uB0JPFnj/VvfdNr5zMzbM/O9mdkvM/sB/8zMnWu+h/KpUofOGag8HfgUlaeTvl3MOw84shjvBdxI5YbYh4Cdqtb9drHeTOAjrdVZzP8SlfswVgAvAL/o6v1fl4ZOPpcrinmPFMN/dPX+r2tDZ51PKh0Ff6XyKcPHgV9R9ZSpQ7nOZzPb9anSThr85JUkSVJJeKlUkiSpJAxukiRJJWFwkyRJKgmDmyRJUkkY3CRJkkrC4CapS0TENhHxSDH8PSLmFeOLI+Kyrm5fZ4qIfhHxeDE+IiIuaaP8vzeZvr+W7ZPUffg6EEldLiLOpfIeqB91dVuaExEbZuW7jjVZLyL6Abdl5qB21rs4Mzdb0/ZIKj973CR1KxExKiJuK8bPjYhfRsS9EfG/EfHxiPhhRDwWEX8sPj5PRAyPiL9ExJSIuCMi3tdMveMj4vKIqI+IpyLiiGJ+j4i4KCIejohpEfG5qnbcGxETgSeaqW9xRFwcEdMj4s6I6F3Mvzsi/jMi6oEvt9S2Yv6jEfEo8IUW9n+ziLi62N9pEfGJiLgA2KTonfxVQ1uKn1Hsy+PFOp+sqvPuiLgpIp6MiF8VXy+QVDIGN0nd3fuBg6l8IulaYHJmDgaWAocX4e1S4JjMHA5cBZzfQl39gJHA4cDlEdEL+DTwWmbuBewFfLb45A/AMODLmblrM3VtCtRn5h7AX4DvVi3bKDNHAJe00rargS9m5p6t7Ps5RdsGZ2Ydlc8TnQUszcwhmXlCk/IfB4YAewIfAi6qCrFDga8AA4GdgP1b2a6kbmrDrm6AJLXhD5m5PCIeA3oAfyzmP0YliA0ABgF/KjqRegAvtlDXDVn55unTEfEssBvwYaAuIo4pymxB5TuNbwIPZeZzLdS1Cri+GL8WuKVqWcP8ZtsWEVsCW2bmPUW5a4CPNLOND1H5NiQAmbmwhbY0OAC4LisfcX8pIv5CJYy+XuzLXICIeITKsbuvjfokdTMGN0nd3RsAmbkqIpbnWzfmrqLy37AApmfmvu2oq+lNvVms/8XMvKN6QUSMApasQTur625Yr9m2FcGts71RNb4S//svlZKXSiWV3Uygd0TsCxARPSNijxbKHhsRG0TE+6lcLpwJ3AF8vup+uV0jYtN2bHcDoKGX7t9ovveq2bZl5iJgUUQcUJRresmzwZ9Y/f63rYrR5Q3tbeJe4JPFfXu9gQ9S+ZC4pHWEwU1SqWXmm1QC1IXFjf6PAPu1UPx5KkHmD8DpmbkM+AWVhw/+p3glxxW0rzdqCTCyWOdg4Lw1bNspwLjismVLDwr8ANiqeNjgUeCgYv6VwLSGhxOq/AaYBjwK3AV8MzP/3o59kVQSvg5E0nohIsZTeeXGTR1Un6/kkNTp7HGTJEkqCXvcJEmSSsIeN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqif8HsZN/cPkoiigAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 720x504 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QlHdTqTl0aOq"
      },
      "source": [
        "![](https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/images/08-ideal-performance-speed-of-pred-tradeoff-highlighted.png)\n",
        "*Ideal position for speed and performance tradeoff model (fast predictions with great results).*\n",
        "\n",
        "Of course, the ideal position for each of these dots is to be in the top left of the plot (low time per prediction, high F1-score). \n",
        "\n",
        "In our case, there's a clear tradeoff for time per prediction and performance. Our best performing model takes an order of magnitude longer per prediction but only results in a few F1-score point increase.\n",
        "\n",
        "This kind of tradeoff is something you'll need to keep in mind when incorporating machine learning models into your own applications."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DJWGI6GpH4Gl"
      },
      "source": [
        "##Exercises\n",
        "\n",
        "1. Rebuild, compile and train `model_1`, `model_2` and `model_5` using the [Keras Sequential API](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential) instead of the Functional API.\n",
        "2. Retrain the baseline model with 10% of the training data. How does perform compared to the Universal Sentence Encoder model with 10% of the training data?\n",
        "3. Try fine-tuning the TF Hub Universal Sentence Encoder model by setting `training=True` when instantiating it as a Keras layer.\n",
        "\n",
        "```\n",
        "We can use this encoding layer in place of our text_vectorizer and embedding layer\n",
        "\n",
        "sentence_encoder_layer = hub.KerasLayer(\"https://tfhub.dev/google/universal-sentence-encoder/4\",\n",
        "                                        input_shape=[],\n",
        "                                        dtype=tf.string,\n",
        "                                        trainable=True) # turn training on to fine-tune the TensorFlow Hub model\n",
        "```\n",
        "4. Retrain the best model you've got so far on the whole training set (no validation split). Then use this trained model to make predictions on the test dataset and format the predictions into the same format as the `sample_submission.csv` file from Kaggle (see the Files tab in Colab for what the `sample_submission.csv` file looks like). Once you've done this, [make a submission to the Kaggle competition](https://www.kaggle.com/c/nlp-getting-started/data), how did your model perform?\n",
        "5. Combine the ensemble predictions using the majority vote (mode), how does this perform compare to averaging the prediction probabilities of each model?\n",
        "6. Make a confusion matrix with the best performing model's predictions on the validation set and the validation ground truth labels."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BarVJji8H6M4"
      },
      "source": [
        "##Extra-curriculum \n",
        "\n",
        "To practice what you've learned, a good idea would be to spend an hour on 3 of the following (3-hours total, you could through them all if you want) and then write a blog post about what you've learned.\n",
        "\n",
        "* For an overview of the different problems within NLP and how to solve them read through: \n",
        " * [A Simple Introduction to Natural Language Processing](https://becominghuman.ai/a-simple-introduction-to-natural-language-processing-ea66a1747b32)\n",
        " * [How to solve 90% of NLP problems: a step-by-step guide](https://blog.insightdatascience.com/how-to-solve-90-of-nlp-problems-a-step-by-step-guide-fda605278e4e)\n",
        "* Go through [MIT's Recurrent Neural Networks lecture](https://youtu.be/SEnXr6v2ifU). This will be one of the greatest additions to what's happening behind the RNN model's you've been building.\n",
        "* Read through the [word embeddings page on the TensorFlow website](https://www.tensorflow.org/tutorials/text/word_embeddings). Embeddings are such a large part of NLP. We've covered them throughout this notebook but extra practice would be well worth it. A good exercise would be to write out all the code in the guide in a new notebook. \n",
        "* For more on RNN's in TensorFlow, read and reproduce [the TensorFlow RNN guide](https://www.tensorflow.org/guide/keras/rnn). We've covered many of the concepts in this guide, but it's worth writing the code again for yourself.\n",
        "* Text data doesn't always come in a nice package like the data we've downloaded. So if you're after more on preparing different text sources for being with your TensorFlow deep learning models, it's worth checking out the following:\n",
        " * [TensorFlow text loading tutorial](https://www.tensorflow.org/tutorials/load_data/text).\n",
        "  * [Reading text files with Python](https://realpython.com/read-write-files-python/) by Real Python.\n",
        "* This notebook has focused on writing NLP code. For a mathematically rich overview of how NLP with Deep Learning happens, read [Standford's Natural Language Processing with Deep Learning lecture notes Part 1](https://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes01-wordvecs1.pdf).  \n",
        "  * For an even deeper dive, you could even do the whole [CS224n](http://web.stanford.edu/class/cs224n/) (Natural Language Processing with Deep Learning) course. \n",
        "* Great blog posts to read:\n",
        "  * Andrei Karpathy's [The Unreasonable Effectiveness of RNNs](https://karpathy.github.io/2015/05/21/rnn-effectiveness/) dives into generating Shakespeare text with RNNs.\n",
        "  * [Text Classification with NLP: Tf-Idf vs Word2Vec vs BERT](https://towardsdatascience.com/text-classification-with-nlp-tf-idf-vs-word2vec-vs-bert-41ff868d1794) by Mauro Di Pietro. An overview of different techniques for turning text into numbers and then classifying it.\n",
        "  * [What are word embeddings?](https://machinelearningmastery.com/what-are-word-embeddings/) by Machine Learning Mastery.\n",
        "* Other topics worth looking into:\n",
        "  * [Attention mechanisms](https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/). These are a foundational component of the transformer architecture and also often add improvments to deep NLP models.\n",
        "  * [Transformer architectures](http://jalammar.github.io/illustrated-transformer/). This model architecture has recently taken the NLP world by storm, achieving state of the art on many benchmarks. However, it does take a little more processing to get off the ground, the [HuggingFace Models (formerly HuggingFace Transformers) library](https://huggingface.co/models/) is probably your best quick start.\n",
        "    * And now [HuggingFace even have their own course](https://huggingface.co/course/chapter1) on how their library works! I haven't done it but anything HuggingFace makes is world-class.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CLzfxgXkzEdr"
      },
      "source": [
        "> 📖 **Resource:** See the full set of course materials on GitHub: https://github.com/mrdbourke/tensorflow-deep-learning"
      ]
    }
  ]
}