{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Lasso model selection via information criteria\n\nThis example reproduces the example of Fig. 2 of [ZHT2007]_. A\n:class:`~sklearn.linear_model.LassoLarsIC` estimator is fit on a\ndiabetes dataset and the AIC and the BIC criteria are used to select\nthe best model.\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>It is important to note that the optimization to find `alpha` with\n    :class:`~sklearn.linear_model.LassoLarsIC` relies on the AIC or BIC\n    criteria that are computed in-sample, thus on the training set directly.\n    This approach differs from the cross-validation procedure. For a comparison\n    of the two approaches, you can refer to the following example:\n    `sphx_glr_auto_examples_linear_model_plot_lasso_model_selection.py`.</p></div>\n\n.. rubric:: References\n\n.. [ZHT2007] :arxiv:`Zou, Hui, Trevor Hastie, and Robert Tibshirani.\n    \"On the degrees of freedom of the lasso.\"\n    The Annals of Statistics 35.5 (2007): 2173-2192.\n    <0712.0881>`\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Authors: The scikit-learn developers\n# SPDX-License-Identifier: BSD-3-Clause"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We will use the diabetes dataset.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.datasets import load_diabetes\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nn_samples = X.shape[0]\nX.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Scikit-learn provides an estimator called\n:class:`~sklearn.linear_model.LassoLarsIC` that uses either Akaike's\ninformation criterion (AIC) or the Bayesian information criterion (BIC) to\nselect the best model. Before fitting\nthis model, we will scale the dataset.\n\nIn the following, we are going to fit two models to compare the values\nreported by AIC and BIC.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.linear_model import LassoLarsIC\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nlasso_lars_ic = make_pipeline(StandardScaler(), LassoLarsIC(criterion=\"aic\")).fit(X, y)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To be in line with the definition in [ZHT2007]_, we need to rescale the\nAIC and the BIC. Indeed, Zou et al. are ignoring some constant terms\ncompared to the original definition of AIC derived from the maximum\nlog-likelihood of a linear model. You can refer to\n`mathematical detail section for the User Guide <lasso_lars_ic>`.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def zou_et_al_criterion_rescaling(criterion, n_samples, noise_variance):\n    \"\"\"Rescale the information criterion to follow the definition of Zou et al.\"\"\"\n    return criterion - n_samples * np.log(2 * np.pi * noise_variance) - n_samples"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import numpy as np\n\naic_criterion = zou_et_al_criterion_rescaling(\n    lasso_lars_ic[-1].criterion_,\n    n_samples,\n    lasso_lars_ic[-1].noise_variance_,\n)\n\nindex_alpha_path_aic = np.flatnonzero(\n    lasso_lars_ic[-1].alphas_ == lasso_lars_ic[-1].alpha_\n)[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "lasso_lars_ic.set_params(lassolarsic__criterion=\"bic\").fit(X, y)\n\nbic_criterion = zou_et_al_criterion_rescaling(\n    lasso_lars_ic[-1].criterion_,\n    n_samples,\n    lasso_lars_ic[-1].noise_variance_,\n)\n\nindex_alpha_path_bic = np.flatnonzero(\n    lasso_lars_ic[-1].alphas_ == lasso_lars_ic[-1].alpha_\n)[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now that we collected the AIC and BIC, we can as well check that the minima\nof both criteria happen at the same alpha. Then, we can simplify the\nfollowing plot.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "index_alpha_path_aic == index_alpha_path_bic"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Finally, we can plot the AIC and BIC criterion and the subsequent selected\nregularization parameter.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n\nplt.plot(aic_criterion, color=\"tab:blue\", marker=\"o\", label=\"AIC criterion\")\nplt.plot(bic_criterion, color=\"tab:orange\", marker=\"o\", label=\"BIC criterion\")\nplt.vlines(\n    index_alpha_path_bic,\n    aic_criterion.min(),\n    aic_criterion.max(),\n    color=\"black\",\n    linestyle=\"--\",\n    label=\"Selected alpha\",\n)\nplt.legend()\nplt.ylabel(\"Information criterion\")\nplt.xlabel(\"Lasso model sequence\")\n_ = plt.title(\"Lasso model selection via AIC and BIC\")"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.14"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}