{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Univariate Feature Selection\n\nThis notebook is an example of using univariate feature selection\nto improve classification accuracy on a noisy dataset.\n\nIn this example, some noisy (non informative) features are added to\nthe iris dataset. Support vector machine (SVM) is used to classify the\ndataset both before and after applying univariate feature selection.\nFor each feature, we plot the p-values for the univariate feature selection\nand the corresponding weights of SVMs. With this, we will compare model\naccuracy and examine the impact of univariate feature selection on model\nweights.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Authors: The scikit-learn developers\n# SPDX-License-Identifier: BSD-3-Clause"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Generate sample data\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import numpy as np\n\nfrom sklearn.datasets import load_iris\nfrom sklearn.model_selection import train_test_split\n\n# The iris dataset\nX, y = load_iris(return_X_y=True)\n\n# Some noisy data not correlated\nE = np.random.RandomState(42).uniform(0, 0.1, size=(X.shape[0], 20))\n\n# Add the noisy data to the informative features\nX = np.hstack((X, E))\n\n# Split dataset to select feature and evaluate the classifier\nX_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Univariate feature selection\n\nUnivariate feature selection with F-test for feature scoring.\nWe use the default selection function to select\nthe four most significant features.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.feature_selection import SelectKBest, f_classif\n\nselector = SelectKBest(f_classif, k=4)\nselector.fit(X_train, y_train)\nscores = -np.log10(selector.pvalues_)\nscores /= scores.max()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n\nX_indices = np.arange(X.shape[-1])\nplt.figure(1)\nplt.clf()\nplt.bar(X_indices - 0.05, scores, width=0.2)\nplt.title(\"Feature univariate score\")\nplt.xlabel(\"Feature number\")\nplt.ylabel(r\"Univariate score ($-Log(p_{value})$)\")\nplt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In the total set of features, only the 4 of the original features are significant.\nWe can see that they have the highest score with univariate feature\nselection.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Compare with SVMs\n\nWithout univariate feature selection\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.svm import LinearSVC\n\nclf = make_pipeline(MinMaxScaler(), LinearSVC())\nclf.fit(X_train, y_train)\nprint(\n    \"Classification accuracy without selecting features: {:.3f}\".format(\n        clf.score(X_test, y_test)\n    )\n)\n\nsvm_weights = np.abs(clf[-1].coef_).sum(axis=0)\nsvm_weights /= svm_weights.sum()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "After univariate feature selection\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "clf_selected = make_pipeline(SelectKBest(f_classif, k=4), MinMaxScaler(), LinearSVC())\nclf_selected.fit(X_train, y_train)\nprint(\n    \"Classification accuracy after univariate feature selection: {:.3f}\".format(\n        clf_selected.score(X_test, y_test)\n    )\n)\n\nsvm_weights_selected = np.abs(clf_selected[-1].coef_).sum(axis=0)\nsvm_weights_selected /= svm_weights_selected.sum()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "plt.bar(\n    X_indices - 0.45, scores, width=0.2, label=r\"Univariate score ($-Log(p_{value})$)\"\n)\n\nplt.bar(X_indices - 0.25, svm_weights, width=0.2, label=\"SVM weight\")\n\nplt.bar(\n    X_indices[selector.get_support()] - 0.05,\n    svm_weights_selected,\n    width=0.2,\n    label=\"SVM weights after selection\",\n)\n\nplt.title(\"Comparing feature selection\")\nplt.xlabel(\"Feature number\")\nplt.yticks(())\nplt.axis(\"tight\")\nplt.legend(loc=\"upper right\")\nplt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Without univariate feature selection, the SVM assigns a large weight\nto the first 4 original significant features, but also selects many of the\nnon-informative features. Applying univariate feature selection before\nthe SVM increases the SVM weight attributed to the significant features,\nand will thus improve classification.\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.14"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}