20_newsgroups_automl.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "20-newsgroups-automl.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "2tr1DfOiz9cb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# 20 Newsgroups data import script for *Google Cloud AutoML Natural Language*\n",
        "\n",
        "This notebook downloads the [20 newsgroups dataset](https://scikit-learn.org/0.19/datasets/twenty_newsgroups.html) using scikit-learn. This dataset contains about 18000 posts from 20 newsgroups, and is useful for text classification. The script transforms the data into a pandas dataframe and finally into a CSV file readable by [Google Cloud AutoML Natural Language](https://cloud.google.com/natural-language/automl)."
      ]
    },
    {
      "metadata": {
        "id": "K65WZ6bMz9cc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Imports"
      ]
    },
    {
      "metadata": {
        "id": "OZDimb-5z9cd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import csv\n",
        "\n",
        "from sklearn.datasets import fetch_20newsgroups"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "zYxeG10oz9cg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Fetch data"
      ]
    },
    {
      "metadata": {
        "id": "mV7b2hHfz9ch",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 257
        },
        "outputId": "d3c53e19-eb23-4f2c-a9e4-b607ba0de044"
      },
      "cell_type": "code",
      "source": [
        "newsgroups = fetch_20newsgroups(subset='all')\n",
        "\n",
        "df = pd.DataFrame(newsgroups.data, columns=['text'])\n",
        "df['categories'] = [newsgroups.target_names[index] for index in newsgroups.target]\n",
        "df.head()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading 20news dataset. This may take a few minutes.\n",
            "Downloading dataset from https://ndownloader.figshare.com/files/5975967 (14 MB)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>categories</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>From: Mamatha Devineni Ratnam &lt;mr47+@andrew.cm...</td>\n",
              "      <td>rec.sport.hockey</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>From: mblawson@midway.ecn.uoknor.edu (Matthew ...</td>\n",
              "      <td>comp.sys.ibm.pc.hardware</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>From: hilmi-er@dsv.su.se (Hilmi Eren)\\nSubject...</td>\n",
              "      <td>talk.politics.mideast</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>From: guyd@austin.ibm.com (Guy Dawson)\\nSubjec...</td>\n",
              "      <td>comp.sys.ibm.pc.hardware</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>From: Alexander Samuel McDiarmid &lt;am2o+@andrew...</td>\n",
              "      <td>comp.sys.mac.hardware</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                text                categories\n",
              "0  From: Mamatha Devineni Ratnam <mr47+@andrew.cm...          rec.sport.hockey\n",
              "1  From: mblawson@midway.ecn.uoknor.edu (Matthew ...  comp.sys.ibm.pc.hardware\n",
              "2  From: hilmi-er@dsv.su.se (Hilmi Eren)\\nSubject...     talk.politics.mideast\n",
              "3  From: guyd@austin.ibm.com (Guy Dawson)\\nSubjec...  comp.sys.ibm.pc.hardware\n",
              "4  From: Alexander Samuel McDiarmid <am2o+@andrew...     comp.sys.mac.hardware"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "metadata": {
        "id": "RMJqpjZwz9cl",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Clean data"
      ]
    },
    {
      "metadata": {
        "id": "K6yd6XSLz9cl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 202
        },
        "outputId": "d6328a82-551a-44f2-b286-cdfa923ef2f8"
      },
      "cell_type": "code",
      "source": [
        "# Convert multiple whitespace characters into a space\n",
        "df['text'] = df['text'].str.replace('\\s+',' ')\n",
        "\n",
        "# Trim leading and tailing whitespace\n",
        "df['text'] = df['text'].str.strip()\n",
        "\n",
        "# Truncate all fields to the maximum field length of 128kB\n",
        "df['text'] = df['text'].str.slice(0,131072)\n",
        "\n",
        "# Remove any rows with empty fields\n",
        "df = df.replace('', np.NaN).dropna()\n",
        "\n",
        "# Drop duplicates\n",
        "df = df.drop_duplicates(subset='text')\n",
        "\n",
        "# Limit rows to maximum of 100,000\n",
        "df = df.sample(min(100000, len(df)))\n",
        "\n",
        "df.head()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>categories</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>16550</th>\n",
              "      <td>From: will@futon.webo.dg.com (Will Taber) Subj...</td>\n",
              "      <td>soc.religion.christian</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13121</th>\n",
              "      <td>From: zowie@daedalus.stanford.edu (Craig \"Powd...</td>\n",
              "      <td>sci.space</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15519</th>\n",
              "      <td>From: boyle@cactus.org (Craig Boyle) Subject: ...</td>\n",
              "      <td>rec.autos</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10389</th>\n",
              "      <td>From: erics@netcom.com (Eric Smith) Subject: R...</td>\n",
              "      <td>rec.sport.baseball</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15554</th>\n",
              "      <td>Organization: University of Illinois at Chicag...</td>\n",
              "      <td>talk.politics.guns</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                    text  \\\n",
              "16550  From: will@futon.webo.dg.com (Will Taber) Subj...   \n",
              "13121  From: zowie@daedalus.stanford.edu (Craig \"Powd...   \n",
              "15519  From: boyle@cactus.org (Craig Boyle) Subject: ...   \n",
              "10389  From: erics@netcom.com (Eric Smith) Subject: R...   \n",
              "15554  Organization: University of Illinois at Chicag...   \n",
              "\n",
              "                   categories  \n",
              "16550  soc.religion.christian  \n",
              "13121               sci.space  \n",
              "15519               rec.autos  \n",
              "10389      rec.sport.baseball  \n",
              "15554      talk.politics.guns  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "metadata": {
        "id": "exFdr6wVz9co",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Export to CSV"
      ]
    },
    {
      "metadata": {
        "id": "pYUzEq-oz9cp",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "csv_str = df.to_csv(index=False, header=False)\n",
        "\n",
        "with open(\"20-newsgroups-dataset.csv\", \"w\") as text_file:\n",
        "    print(csv_str, file=text_file)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1kJX0DKHz9cr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "You're all set! Download `20-newsgroups-dataset.csv` and import it into [Google Cloud AutoML Natural Language](https://cloud.google.com/natural-language/automl).\n",
        "\n",
        "If you are using [Google Colab](https://colab.research.google.com), you will find the file in the left navbar:\n",
        "\n",
        "\n",
        "*   From the menu, select **View > Table of Contents**\n",
        "*   Navigate to the **Files** tab\n",
        "*   Find the file in `/content` directory.\n",
        "\n",
        "\n"
      ]
    }
  ]
}