udacity-project-231120.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Introduction\" data-toc-modified-id=\"Introduction-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Introduction</a></span></li><li><span><a href=\"#Create-an-Experiment-in-Azure-ML-workspace\" data-toc-modified-id=\"Create-an-Experiment-in-Azure-ML-workspace-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Create an Experiment in Azure ML workspace</a></span></li><li><span><a href=\"#HyperDrive-Pipeline\" data-toc-modified-id=\"HyperDrive-Pipeline-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>HyperDrive Pipeline</a></span><ul class=\"toc-item\"><li><span><a href=\"#Create-Resources-for-Training-Experiments\" data-toc-modified-id=\"Create-Resources-for-Training-Experiments-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Create Resources for Training Experiments</a></span></li><li><span><a href=\"#Hyperparameter-Tunning\" data-toc-modified-id=\"Hyperparameter-Tunning-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>Hyperparameter Tunning</a></span><ul class=\"toc-item\"><li><span><a href=\"#Parameter-sampler\" data-toc-modified-id=\"Parameter-sampler-3.2.1\"><span class=\"toc-item-num\">3.2.1&nbsp;&nbsp;</span>Parameter sampler</a></span></li><li><span><a href=\"#Early-Termination-Policy\" data-toc-modified-id=\"Early-Termination-Policy-3.2.2\"><span class=\"toc-item-num\">3.2.2&nbsp;&nbsp;</span>Early Termination Policy</a></span></li><li><span><a href=\"#Create-a-SKLearn-Estimator\" data-toc-modified-id=\"Create-a-SKLearn-Estimator-3.2.3\"><span class=\"toc-item-num\">3.2.3&nbsp;&nbsp;</span>Create a SKLearn Estimator</a></span></li><li><span><a href=\"#Create-a-HyperDriveConfig\" data-toc-modified-id=\"Create-a-HyperDriveConfig-3.2.4\"><span class=\"toc-item-num\">3.2.4&nbsp;&nbsp;</span>Create a <a href=\"https://docs.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.hyperdrive.hyperdriveconfig?view=azure-ml-py\" target=\"_blank\">HyperDriveConfig</a></a></span></li></ul></li></ul></li><li><span><a href=\"#AutoML-Run\" data-toc-modified-id=\"AutoML-Run-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>AutoML Run</a></span><ul class=\"toc-item\"><li><span><a href=\"#Create-Dataset\" data-toc-modified-id=\"Create-Dataset-4.1\"><span class=\"toc-item-num\">4.1&nbsp;&nbsp;</span>Create Dataset</a></span></li><li><span><a href=\"#Inspect-Dataset\" data-toc-modified-id=\"Inspect-Dataset-4.2\"><span class=\"toc-item-num\">4.2&nbsp;&nbsp;</span>Inspect Dataset</a></span></li><li><span><a href=\"#Clean-and-Split-Dataset\" data-toc-modified-id=\"Clean-and-Split-Dataset-4.3\"><span class=\"toc-item-num\">4.3&nbsp;&nbsp;</span>Clean and Split Dataset</a></span></li><li><span><a href=\"#Configure-Experiment\" data-toc-modified-id=\"Configure-Experiment-4.4\"><span class=\"toc-item-num\">4.4&nbsp;&nbsp;</span>Configure Experiment</a></span></li><li><span><a href=\"#Submitting-Training-Experiment\" data-toc-modified-id=\"Submitting-Training-Experiment-4.5\"><span class=\"toc-item-num\">4.5&nbsp;&nbsp;</span>Submitting Training Experiment</a></span></li><li><span><a href=\"#Monitor-using-Widget\" data-toc-modified-id=\"Monitor-using-Widget-4.6\"><span class=\"toc-item-num\">4.6&nbsp;&nbsp;</span>Monitor using <code>Widget</code></a></span></li><li><span><a href=\"#Retrieve-and-Save-Best-Model\" data-toc-modified-id=\"Retrieve-and-Save-Best-Model-4.7\"><span class=\"toc-item-num\">4.7&nbsp;&nbsp;</span>Retrieve and Save Best Model</a></span></li></ul></li><li><span><a href=\"#Cleaning-Up-Cluster\" data-toc-modified-id=\"Cleaning-Up-Cluster-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Cleaning Up Cluster</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "In this notebook we use a subset of the [UCI Bank Marketing dataset](https://archive.ics.uci.edu/ml/datasets/Bank+Marketing), first introduced in the paper [[Moro et al., 2014](https://repositorio.iscte-iul.pt/bitstream/10071/9499/5/dss_v3.pdf)] to solve a classification problem using both HyperDrive and Azure AutoML. The classification goal is to predict if the client will subscribe to a term deposit with the bank.\n",
    "\n",
    "First, we use Scikit-learn Logistic Regression model optimizing hyperparameters using HyperDrive. Next, we use the same dataset and Azure AutoML to build and optimize another model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create an Experiment in Azure ML workspace\n",
    "\n",
    "For this project we will be using an Azure Machine Learning Notebook VM, therefore we can skip setting up the environment.\n",
    "\n",
    "To start we need to initialize our workspace and create a Azule ML experiment. It is also to remember that accessing the Azure ML workspace requires authentication with Azure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Workspace, Experiment\n",
    "\n",
    "# Initialize a workspace object for an existing Azure Machine Learning Workspace\n",
    "ws = Workspace.get(\"quick-starts-ws-127549\")\n",
    "\n",
    "# Create a experiment\n",
    "exp = Experiment(workspace=ws, name=\"udacity-project\")\n",
    "\n",
    "run = exp.start_logging()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Workspace name</th>\n",
       "      <td>quick-starts-ws-127549</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Azure region</th>\n",
       "      <td>southcentralus</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Subscription id</th>\n",
       "      <td>55e71b9d-a209-42c0-8818-ca9cc885909c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Resource group</th>\n",
       "      <td>aml-quickstarts-127549</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Experiment Name</th>\n",
       "      <td>udacity-project</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     \n",
       "Workspace name                 quick-starts-ws-127549\n",
       "Azure region                           southcentralus\n",
       "Subscription id  55e71b9d-a209-42c0-8818-ca9cc885909c\n",
       "Resource group                 aml-quickstarts-127549\n",
       "Experiment Name                       udacity-project"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "dic_data = {'Workspace name': ws.name,\n",
    "            'Azure region': ws.location,\n",
    "            'Subscription id': ws.subscription_id,\n",
    "            'Resource group': ws.resource_group,\n",
    "            'Experiment Name': exp.name}\n",
    "\n",
    "df_data = pd.DataFrame.from_dict(data = dic_data, orient='index')\n",
    "\n",
    "df_data.rename(columns={0:''}, inplace = True)\n",
    "df_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HyperDrive Pipeline\n",
    "\n",
    "## Create Resources for Training Experiments\n",
    "\n",
    "Now that we have initialized our workspace and created our experiment, it is time to define our resources.\n",
    "\n",
    "In this section you will create default compute clusters for use by the notebook and any other necessary operations we need.\n",
    "\n",
    "In order to create a cluster we need to specify a compute configuration that defines the `type of machine` to be used and the `scalability behaviors`. Also, it is necessary to define the name of the cluster which must be unique within the workspace. This name is used to address the cluster later.\n",
    "\n",
    "For this project we use a CPU cluster with following parameters:\n",
    "\n",
    "* `type of the machine`:\n",
    "\n",
    "    * `vm_size`: Defines the size of the virtual machine. We use here \"STANDARD_D2_V2\" (more details [here](https://docs.microsoft.com/en-us/azure/cloud-services/cloud-services-sizes-specs#dv2-series))\n",
    "\n",
    "* `Scalability behaviors`:\n",
    "\n",
    "    * `min_nodes`: Sets minimun size of the cluster. Setting the minimum to 0 the cluster will shut down all nodes while not in use. If you use another value you are able to have faster start-up times, but you will also be billed when the cluster is not in use.\n",
    "\n",
    "    * `max_nodes`: Sets the maximun size of the cluster. Larger number allows for more concurrency and a greater distributed processing of scale-out jobs.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating\n",
      "Succeeded\n",
      "AmlCompute wait for completion finished\n",
      "\n",
      "Minimum number of nodes requested have been provisioned\n"
     ]
    }
   ],
   "source": [
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "\n",
    "# Define CPU cluster name\n",
    "cpu_cluster_name = \"cpu-cluster\"\n",
    "\n",
    "\n",
    "# Verify that cluster does not exist already\n",
    "try:\n",
    "    cpu_cluster = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
    "    print(\"Found existing cpu-cluster\")\n",
    "except ComputeTargetException:\n",
    "    \n",
    "    # Specify the configuration for the new cluster\n",
    "    compute_config = AmlCompute.provisioning_configuration(vm_size=\"STANDARD_D2_V2\",\n",
    "                                                           min_nodes=0, # when innactive\n",
    "                                                           max_nodes=4) # when busy\n",
    "\n",
    "    # Create the cluster with the specified name and configuration\n",
    "    cpu_cluster = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
    "    \n",
    "    # Wait for the cluster to complete, show the output log\n",
    "    cpu_cluster.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "gather": {
     "logged": 1598275788675
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cpu-c231120 ComputeInstance Succeeded\n",
      "cpu-cluster AmlCompute Succeeded\n"
     ]
    }
   ],
   "source": [
    "# Check details about compute_targets (i.e. cpu_cluster)\n",
    "\n",
    "compute_targets = ws.compute_targets\n",
    "for name, ct in compute_targets.items():\n",
    "    print(name, ct.type, ct.provisioning_state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameter Tunning\n",
    "\n",
    "### Parameter sampler\n",
    "\n",
    "In this example using HyperDrive we use [`random sampling`](https://docs.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.hyperdrive.randomparametersampling?view=azure-ml-py) to try different configuration sets of hyperparameters to maximize the chosen primary metric, accuracy. The function `choice` specify a discrete set of options to sample from.\n",
    "\n",
    "The hyperparameters and metric used are defined in the script `train.py`.\n",
    "\n",
    "### Early Termination Policy\n",
    "\n",
    "This saves us from continuing to explore hyperparameters that don't show promise of helping reach our target metric.\n",
    "\n",
    "An early termination policy help us improving computational efficiency by terminating poorly performing runs.\n",
    "\n",
    "The `early termination policy` we used [`Bandit Policy`]( https://docs.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.hyperdrive.banditpolicy?preserve-view=true&view=azure-ml-py#&preserve-view=truedefinition ). This policy is based on `slack factor/slack amount` and `evaluation interval`. Bandit terminates runs where the primary metric is not within the specified slack factor/slack amount compared to the best performing run.\n",
    "\n",
    "This allows more aggressive savings than Median Stopping policy if we apply a smaller allowable slack.\n",
    "\n",
    "Parameter `slack_factor` which is the slack allowed with respect to the best performing training run, need to be defined while `evaluation_interval` and `delay_interval` are optional.\n",
    "\n",
    "`evaluation_interval` says when the policy is applied. If the `evaluation_interval` is not defined the default value is one, i.e., policy is applied every time the training script reports the primary metric.\n",
    "\n",
    "Specifying `delay_interval` avoids premature termination of training runs by allowing all configurations to run for a minimum number of intervals. If specified, the policy applies every multiple of evaluation_interval that is greater than or equal to delay_evaluation.\n",
    "\n",
    "For example, in our example, by applying the Bandit policy with `slack_factor = 0.1`, `evaluation_interval=2`, `delay_evaluation=5` the early termination policy is applied at every other time interval when metrics are reported, starting at evaluation interval 5. Any run whose primary metric falls outside of the top 10% range, Azure ML terminate the job.\n",
    "\n",
    "### Create a SKLearn Estimator\n",
    "\n",
    "[SKLearn Class](https://docs.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.sklearn.sklearn?view=azure-ml-py) creates an estimator for training in Scikit-learn experiments.\n",
    "\n",
    "### Create a [HyperDriveConfig](https://docs.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.hyperdrive.hyperdriveconfig?view=azure-ml-py)\n",
    "\n",
    "Now we are ready to configure a run configuration object. \n",
    "\n",
    "As parameters we inform `parameter sampler`, `early termination policy`, and `estimator` that we just configured. We also specify the primary metric `Accuracy` that's recorded in your training runs and we tell the service that we want to maximize this value.  \n",
    "\n",
    "Moreover, we set `maximal concurrent job` to 4, which is the same as the number of nodes in our computer cluster.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.widgets import RunDetails\n",
    "from azureml.train.sklearn import SKLearn\n",
    "from azureml.train.hyperdrive.run import PrimaryMetricGoal\n",
    "from azureml.train.hyperdrive.policy import BanditPolicy\n",
    "from azureml.train.hyperdrive.sampling import RandomParameterSampling\n",
    "from azureml.train.hyperdrive.runconfig import HyperDriveConfig\n",
    "from azureml.train.hyperdrive.parameter_expressions import uniform, choice\n",
    "import os\n",
    "\n",
    "# Specify parameter sampler\n",
    "\n",
    "ps = RandomParameterSampling({\n",
    "    '--C': choice(0.01, 0.1, 0.2, 0.5, 0.7, 1.0),\n",
    "    '--max_iter': choice(range(10,110,10))\n",
    "    }\n",
    ")\n",
    "\n",
    "# Specify a Policy\n",
    "policy = BanditPolicy(slack_factor = 0.1, # specifies the allowable slack as a ratio\n",
    "                      evaluation_interval=2, # frequency for applying the policy\n",
    "                      delay_evaluation=5) # delays the first policy evaluation for a specified number of intervals\n",
    "\n",
    "if \"training\" not in os.listdir():\n",
    "    os.mkdir(\"./training\")\n",
    "\n",
    "# Create a SKLearn estimator for use with train.py\n",
    "\n",
    "est = SKLearn( \n",
    "    source_directory='./', # directory containing experiment configuration files (train.py)\n",
    "    compute_target=cpu_cluster, # compute target where training will happen\n",
    "    vm_size=\"STANDARD_D2_V2\", # VM size of the compute target\n",
    "    vm_priority='lowpriority', # VM priority of the compute target (default value is 'dedicated')\n",
    "    entry_script='train.py'\n",
    ")\n",
    "\n",
    "# Create a HyperDriveConfig using the estimator, hyperparameter sampler, and policy.\n",
    "\n",
    "hyperdrive_config = HyperDriveConfig(estimator=est,\n",
    "                                hyperparameter_sampling=ps,\n",
    "                                policy=policy,\n",
    "                                primary_metric_name='Accuracy',\n",
    "                                primary_metric_goal=PrimaryMetricGoal.MAXIMIZE,\n",
    "                                max_total_runs=4,\n",
    "                                max_concurrent_runs=4\n",
    "                                    )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING - If 'script' has been provided here and a script file name has been specified in 'run_config', 'script' provided in ScriptRunConfig initialization will take precedence.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "259a3a3c5b5248a4ac3d9b73a00cfce0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_HyperDriveWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO'…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/experiments/udacity-project/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7?wsid=/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourcegroups/aml-quickstarts-127549/workspaces/quick-starts-ws-127549\", \"run_id\": \"HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7\", \"run_properties\": {\"run_id\": \"HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7\", \"created_utc\": \"2020-11-23T14:39:18.745043Z\", \"properties\": {\"primary_metric_config\": \"{\\\"name\\\": \\\"Accuracy\\\", \\\"goal\\\": \\\"maximize\\\"}\", \"resume_from\": \"null\", \"runTemplate\": \"HyperDrive\", \"azureml.runsource\": \"hyperdrive\", \"platform\": \"AML\", \"ContentSnapshotId\": \"26c3792e-5253-43f8-a916-e70710c8f45d\", \"score\": \"0.9108750632271118\", \"best_child_run_id\": \"HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3\", \"best_metric_status\": \"Succeeded\"}, \"tags\": {\"_aml_system_max_concurrent_jobs\": \"4\", \"max_concurrent_jobs\": \"4\", \"_aml_system_max_total_jobs\": \"4\", \"max_total_jobs\": \"4\", \"_aml_system_max_duration_minutes\": \"10080\", \"max_duration_minutes\": \"10080\", \"_aml_system_policy_config\": \"{\\\"name\\\": \\\"BANDIT\\\", \\\"properties\\\": {\\\"evaluation_interval\\\": 2, \\\"delay_evaluation\\\": 5, \\\"slack_factor\\\": 0.1}}\", \"policy_config\": \"{\\\"name\\\": \\\"BANDIT\\\", \\\"properties\\\": {\\\"evaluation_interval\\\": 2, \\\"delay_evaluation\\\": 5, \\\"slack_factor\\\": 0.1}}\", \"_aml_system_generator_config\": \"{\\\"name\\\": \\\"RANDOM\\\", \\\"parameter_space\\\": {\\\"--C\\\": [\\\"choice\\\", [[0.01, 0.1, 0.2, 0.5, 0.7, 1.0]]], \\\"--max_iter\\\": [\\\"choice\\\", [[10, 20, 30, 40, 50, 60, 70, 80, 90, 100]]]}}\", \"generator_config\": \"{\\\"name\\\": \\\"RANDOM\\\", \\\"parameter_space\\\": {\\\"--C\\\": [\\\"choice\\\", [[0.01, 0.1, 0.2, 0.5, 0.7, 1.0]]], \\\"--max_iter\\\": [\\\"choice\\\", [[10, 20, 30, 40, 50, 60, 70, 80, 90, 100]]]}}\", \"_aml_system_primary_metric_config\": \"{\\\"name\\\": \\\"Accuracy\\\", \\\"goal\\\": \\\"maximize\\\"}\", \"primary_metric_config\": \"{\\\"name\\\": \\\"Accuracy\\\", \\\"goal\\\": \\\"maximize\\\"}\", \"_aml_system_platform_config\": \"{\\\"ServiceAddress\\\": \\\"https://southcentralus.experiments.azureml.net\\\", \\\"ServiceArmScope\\\": \\\"subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/udacity-project\\\", \\\"SubscriptionId\\\": \\\"55e71b9d-a209-42c0-8818-ca9cc885909c\\\", \\\"ResourceGroupName\\\": \\\"aml-quickstarts-127549\\\", \\\"WorkspaceName\\\": \\\"quick-starts-ws-127549\\\", \\\"ExperimentName\\\": \\\"udacity-project\\\", \\\"Definition\\\": {\\\"Overrides\\\": {\\\"script\\\": \\\"train.py\\\", \\\"arguments\\\": [], \\\"target\\\": \\\"cpu-cluster\\\", \\\"framework\\\": \\\"Python\\\", \\\"communicator\\\": \\\"None\\\", \\\"maxRunDurationSeconds\\\": null, \\\"nodeCount\\\": 1, \\\"environment\\\": {\\\"name\\\": null, \\\"version\\\": null, \\\"environmentVariables\\\": {\\\"EXAMPLE_ENV_VAR\\\": \\\"EXAMPLE_VALUE\\\"}, \\\"python\\\": {\\\"userManagedDependencies\\\": true, \\\"interpreterPath\\\": \\\"python\\\", \\\"condaDependenciesFile\\\": null, \\\"baseCondaEnvironment\\\": null, \\\"condaDependencies\\\": {\\\"name\\\": \\\"project_environment\\\", \\\"dependencies\\\": [\\\"python=3.6.2\\\", {\\\"pip\\\": [\\\"azureml-defaults\\\"]}], \\\"channels\\\": [\\\"anaconda\\\", \\\"conda-forge\\\"]}}, \\\"docker\\\": {\\\"enabled\\\": true, \\\"baseImage\\\": \\\"sklearn:0.20.3-cpu\\\", \\\"baseDockerfile\\\": null, \\\"sharedVolumes\\\": true, \\\"shmSize\\\": \\\"2g\\\", \\\"arguments\\\": [], \\\"baseImageRegistry\\\": {\\\"address\\\": \\\"viennaprivate.azurecr.io\\\", \\\"username\\\": null, \\\"password\\\": null, \\\"registryIdentity\\\": null}, \\\"platform\\\": {\\\"os\\\": \\\"Linux\\\", \\\"architecture\\\": \\\"amd64\\\"}}, \\\"spark\\\": {\\\"repositories\\\": [], \\\"packages\\\": [], \\\"precachePackages\\\": false}, \\\"databricks\\\": {\\\"mavenLibraries\\\": [], \\\"pypiLibraries\\\": [], \\\"rcranLibraries\\\": [], \\\"jarLibraries\\\": [], \\\"eggLibraries\\\": []}, \\\"r\\\": null, \\\"inferencingStackVersion\\\": null}, \\\"history\\\": {\\\"outputCollection\\\": true, \\\"snapshotProject\\\": true, \\\"directoriesToWatch\\\": [\\\"logs\\\"]}, \\\"spark\\\": {\\\"configuration\\\": {\\\"spark.app.name\\\": \\\"Azure ML Experiment\\\", \\\"spark.yarn.maxAppAttempts\\\": 1}}, \\\"hdi\\\": {\\\"yarnDeployMode\\\": \\\"cluster\\\"}, \\\"tensorflow\\\": {\\\"workerCount\\\": 1, \\\"parameterServerCount\\\": 1}, \\\"mpi\\\": {\\\"processCountPerNode\\\": 1, \\\"nodeCount\\\": 1}, \\\"paralleltask\\\": {\\\"maxRetriesPerWorker\\\": 0, \\\"workerCountPerNode\\\": 1, \\\"terminalExitCodes\\\": null}, \\\"dataReferences\\\": {}, \\\"data\\\": {}, \\\"outputData\\\": {}, \\\"sourceDirectoryDataStore\\\": null, \\\"amlcompute\\\": {\\\"vmSize\\\": \\\"STANDARD_D2_V2\\\", \\\"vmPriority\\\": \\\"lowpriority\\\", \\\"retainCluster\\\": false, \\\"name\\\": null, \\\"clusterMaxNodeCount\\\": 1}}, \\\"TargetDetails\\\": null, \\\"SnapshotId\\\": \\\"26c3792e-5253-43f8-a916-e70710c8f45d\\\", \\\"TelemetryValues\\\": {\\\"amlClientType\\\": \\\"azureml-sdk-train\\\", \\\"amlClientModule\\\": \\\"[Scrubbed]\\\", \\\"amlClientFunction\\\": \\\"[Scrubbed]\\\", \\\"tenantId\\\": \\\"660b3398-b80e-49d2-bc5b-ac1dc93b5254\\\", \\\"amlClientRequestId\\\": \\\"99696872-a89b-49ea-9aa6-27c82d7d0270\\\", \\\"amlClientSessionId\\\": \\\"ccd6bdde-162f-45c5-b22a-f8563f4497be\\\", \\\"subscriptionId\\\": \\\"55e71b9d-a209-42c0-8818-ca9cc885909c\\\", \\\"estimator\\\": \\\"SKLearn\\\", \\\"samplingMethod\\\": \\\"RANDOM\\\", \\\"terminationPolicy\\\": \\\"Bandit\\\", \\\"primaryMetricGoal\\\": \\\"maximize\\\", \\\"maxTotalRuns\\\": 4, \\\"maxConcurrentRuns\\\": 4, \\\"maxDurationMinutes\\\": 10080, \\\"vmSize\\\": \\\"STANDARD_D2_V2\\\"}}}\", \"platform_config\": \"{\\\"ServiceAddress\\\": \\\"https://southcentralus.experiments.azureml.net\\\", \\\"ServiceArmScope\\\": \\\"subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/udacity-project\\\", \\\"SubscriptionId\\\": \\\"55e71b9d-a209-42c0-8818-ca9cc885909c\\\", \\\"ResourceGroupName\\\": \\\"aml-quickstarts-127549\\\", \\\"WorkspaceName\\\": \\\"quick-starts-ws-127549\\\", \\\"ExperimentName\\\": \\\"udacity-project\\\", \\\"Definition\\\": {\\\"Overrides\\\": {\\\"script\\\": \\\"train.py\\\", \\\"arguments\\\": [], \\\"target\\\": \\\"cpu-cluster\\\", \\\"framework\\\": \\\"Python\\\", \\\"communicator\\\": \\\"None\\\", \\\"maxRunDurationSeconds\\\": null, \\\"nodeCount\\\": 1, \\\"environment\\\": {\\\"name\\\": null, \\\"version\\\": null, \\\"environmentVariables\\\": {\\\"EXAMPLE_ENV_VAR\\\": \\\"EXAMPLE_VALUE\\\"}, \\\"python\\\": {\\\"userManagedDependencies\\\": true, \\\"interpreterPath\\\": \\\"python\\\", \\\"condaDependenciesFile\\\": null, \\\"baseCondaEnvironment\\\": null, \\\"condaDependencies\\\": {\\\"name\\\": \\\"project_environment\\\", \\\"dependencies\\\": [\\\"python=3.6.2\\\", {\\\"pip\\\": [\\\"azureml-defaults\\\"]}], \\\"channels\\\": [\\\"anaconda\\\", \\\"conda-forge\\\"]}}, \\\"docker\\\": {\\\"enabled\\\": true, \\\"baseImage\\\": \\\"sklearn:0.20.3-cpu\\\", \\\"baseDockerfile\\\": null, \\\"sharedVolumes\\\": true, \\\"shmSize\\\": \\\"2g\\\", \\\"arguments\\\": [], \\\"baseImageRegistry\\\": {\\\"address\\\": \\\"viennaprivate.azurecr.io\\\", \\\"username\\\": null, \\\"password\\\": null, \\\"registryIdentity\\\": null}, \\\"platform\\\": {\\\"os\\\": \\\"Linux\\\", \\\"architecture\\\": \\\"amd64\\\"}}, \\\"spark\\\": {\\\"repositories\\\": [], \\\"packages\\\": [], \\\"precachePackages\\\": false}, \\\"databricks\\\": {\\\"mavenLibraries\\\": [], \\\"pypiLibraries\\\": [], \\\"rcranLibraries\\\": [], \\\"jarLibraries\\\": [], \\\"eggLibraries\\\": []}, \\\"r\\\": null, \\\"inferencingStackVersion\\\": null}, \\\"history\\\": {\\\"outputCollection\\\": true, \\\"snapshotProject\\\": true, \\\"directoriesToWatch\\\": [\\\"logs\\\"]}, \\\"spark\\\": {\\\"configuration\\\": {\\\"spark.app.name\\\": \\\"Azure ML Experiment\\\", \\\"spark.yarn.maxAppAttempts\\\": 1}}, \\\"hdi\\\": {\\\"yarnDeployMode\\\": \\\"cluster\\\"}, \\\"tensorflow\\\": {\\\"workerCount\\\": 1, \\\"parameterServerCount\\\": 1}, \\\"mpi\\\": {\\\"processCountPerNode\\\": 1, \\\"nodeCount\\\": 1}, \\\"paralleltask\\\": {\\\"maxRetriesPerWorker\\\": 0, \\\"workerCountPerNode\\\": 1, \\\"terminalExitCodes\\\": null}, \\\"dataReferences\\\": {}, \\\"data\\\": {}, \\\"outputData\\\": {}, \\\"sourceDirectoryDataStore\\\": null, \\\"amlcompute\\\": {\\\"vmSize\\\": \\\"STANDARD_D2_V2\\\", \\\"vmPriority\\\": \\\"lowpriority\\\", \\\"retainCluster\\\": false, \\\"name\\\": null, \\\"clusterMaxNodeCount\\\": 1}}, \\\"TargetDetails\\\": null, \\\"SnapshotId\\\": \\\"26c3792e-5253-43f8-a916-e70710c8f45d\\\", \\\"TelemetryValues\\\": {\\\"amlClientType\\\": \\\"azureml-sdk-train\\\", \\\"amlClientModule\\\": \\\"[Scrubbed]\\\", \\\"amlClientFunction\\\": \\\"[Scrubbed]\\\", \\\"tenantId\\\": \\\"660b3398-b80e-49d2-bc5b-ac1dc93b5254\\\", \\\"amlClientRequestId\\\": \\\"99696872-a89b-49ea-9aa6-27c82d7d0270\\\", \\\"amlClientSessionId\\\": \\\"ccd6bdde-162f-45c5-b22a-f8563f4497be\\\", \\\"subscriptionId\\\": \\\"55e71b9d-a209-42c0-8818-ca9cc885909c\\\", \\\"estimator\\\": \\\"SKLearn\\\", \\\"samplingMethod\\\": \\\"RANDOM\\\", \\\"terminationPolicy\\\": \\\"Bandit\\\", \\\"primaryMetricGoal\\\": \\\"maximize\\\", \\\"maxTotalRuns\\\": 4, \\\"maxConcurrentRuns\\\": 4, \\\"maxDurationMinutes\\\": 10080, \\\"vmSize\\\": \\\"STANDARD_D2_V2\\\"}}}\", \"_aml_system_resume_child_runs\": \"null\", \"resume_child_runs\": \"null\", \"_aml_system_all_jobs_generated\": \"true\", \"all_jobs_generated\": \"true\", \"_aml_system_cancellation_requested\": \"false\", \"cancellation_requested\": \"false\", \"_aml_system_progress_metadata_evaluation_timestamp\": \"\\\"2020-11-23T14:39:19.520495\\\"\", \"progress_metadata_evaluation_timestamp\": \"\\\"2020-11-23T14:39:19.520495\\\"\", \"_aml_system_progress_metadata_digest\": \"\\\"008ff2669a4f568a246289a6f964a07fce955d1fd4d00c2a4cb0f0973d3de029\\\"\", \"progress_metadata_digest\": \"\\\"008ff2669a4f568a246289a6f964a07fce955d1fd4d00c2a4cb0f0973d3de029\\\"\", \"_aml_system_progress_metadata_active_timestamp\": \"\\\"2020-11-23T14:39:19.520495\\\"\", \"progress_metadata_active_timestamp\": \"\\\"2020-11-23T14:39:19.520495\\\"\", \"_aml_system_HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_0\": \"{\\\"--C\\\": 0.7, \\\"--max_iter\\\": 70}\", \"HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_0\": \"{\\\"--C\\\": 0.7, \\\"--max_iter\\\": 70}\", \"_aml_system_HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_1\": \"{\\\"--C\\\": 0.01, \\\"--max_iter\\\": 30}\", \"HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_1\": \"{\\\"--C\\\": 0.01, \\\"--max_iter\\\": 30}\", \"_aml_system_HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_2\": \"{\\\"--C\\\": 0.5, \\\"--max_iter\\\": 20}\", \"HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_2\": \"{\\\"--C\\\": 0.5, \\\"--max_iter\\\": 20}\", \"_aml_system_HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3\": \"{\\\"--C\\\": 0.7, \\\"--max_iter\\\": 100}\", \"HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3\": \"{\\\"--C\\\": 0.7, \\\"--max_iter\\\": 100}\", \"_aml_system_environment_preparation_status\": \"PREPARED\", \"environment_preparation_status\": \"PREPARED\", \"_aml_system_prepare_run_id\": \"HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_preparation\", \"prepare_run_id\": \"HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_preparation\", \"_aml_system_final_best_metric_update_retry_count\": \"1\", \"final_best_metric_update_retry_count\": \"1\"}, \"end_time_utc\": \"2020-11-23T14:46:49.925747Z\", \"status\": \"Completed\", \"log_files\": {\"azureml-logs/hyperdrive.txt\": \"https://mlstrg127549.blob.core.windows.net/azureml/ExperimentRun/dcid.HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7/azureml-logs/hyperdrive.txt?sv=2019-02-02&sr=b&sig=blevWefqcBtQBeCQ6RdPWPn1%2BYmCIXXb1gmrvYSuohc%3D&st=2020-11-23T15%3A38%3A00Z&se=2020-11-23T23%3A48%3A00Z&sp=r\"}, \"log_groups\": [[\"azureml-logs/hyperdrive.txt\"]], \"run_duration\": \"0:07:31\", \"hyper_parameters\": {\"--C\": [\"choice\", [[0.01, 0.1, 0.2, 0.5, 0.7, 1.0]]], \"--max_iter\": [\"choice\", [[10, 20, 30, 40, 50, 60, 70, 80, 90, 100]]]}}, \"child_runs\": [{\"run_id\": \"HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3\", \"run_number\": 4, \"metric\": 0.91087506, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-11-23T14:43:38.40974Z\", \"end_time\": \"2020-11-23T14:45:41.693604Z\", \"created_time\": \"2020-11-23T14:39:52.884488Z\", \"created_time_dt\": \"2020-11-23T14:39:52.884488Z\", \"duration\": \"0:05:48\", \"hyperdrive_id\": \"e4a8f3f1-8636-4161-ae5b-58a20a6eadf7\", \"arguments\": null, \"param_--C\": 0.7, \"param_--max_iter\": 100, \"best_metric\": 0.91087506}, {\"run_id\": \"HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_0\", \"run_number\": 5, \"metric\": 0.91087506, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-11-23T14:43:49.935322Z\", \"end_time\": \"2020-11-23T14:45:52.215Z\", \"created_time\": \"2020-11-23T14:39:52.935924Z\", \"created_time_dt\": \"2020-11-23T14:39:52.935924Z\", \"duration\": \"0:05:59\", \"hyperdrive_id\": \"e4a8f3f1-8636-4161-ae5b-58a20a6eadf7\", \"arguments\": null, \"param_--C\": 0.7, \"param_--max_iter\": 70, \"best_metric\": 0.91087506}, {\"run_id\": \"HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_1\", \"run_number\": 6, \"metric\": 0.91047041, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-11-23T14:43:42.19395Z\", \"end_time\": \"2020-11-23T14:46:01.713529Z\", \"created_time\": \"2020-11-23T14:39:53.588563Z\", \"created_time_dt\": \"2020-11-23T14:39:53.588563Z\", \"duration\": \"0:06:08\", \"hyperdrive_id\": \"e4a8f3f1-8636-4161-ae5b-58a20a6eadf7\", \"arguments\": null, \"param_--C\": 0.01, \"param_--max_iter\": 30, \"best_metric\": 0.91087506}, {\"run_id\": \"HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_2\", \"run_number\": 7, \"metric\": 0.9107739, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-11-23T14:43:37.249627Z\", \"end_time\": \"2020-11-23T14:45:46.040085Z\", \"created_time\": \"2020-11-23T14:39:53.690883Z\", \"created_time_dt\": \"2020-11-23T14:39:53.690883Z\", \"duration\": \"0:05:52\", \"hyperdrive_id\": \"e4a8f3f1-8636-4161-ae5b-58a20a6eadf7\", \"arguments\": null, \"param_--C\": 0.5, \"param_--max_iter\": 20, \"best_metric\": 0.91087506}], \"children_metrics\": {\"categories\": [0], \"series\": {\"Regularization Strength:\": [{\"categories\": [4, 5, 6, 7], \"mode\": \"markers\", \"name\": \"Regularization Strength:\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.7, 0.7, 0.01, 0.5]}, {\"categories\": [4, 5, 6, 7], \"mode\": \"lines\", \"name\": \"Regularization Strength:_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.7, 0.7, 0.7, 0.7]}], \"Max iterations:\": [{\"categories\": [4, 5, 6, 7], \"mode\": \"markers\", \"name\": \"Max iterations:\", \"stepped\": false, \"type\": \"scatter\", \"data\": [100, 70, 30, 20]}, {\"categories\": [4, 5, 6, 7], \"mode\": \"lines\", \"name\": \"Max iterations:_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [100, 100, 100, 100]}], \"Accuracy\": [{\"categories\": [4, 5, 6, 7], \"mode\": \"markers\", \"name\": \"Accuracy\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9108750632271118, 0.9108750632271118, 0.9104704097116844, 0.910773899848255]}, {\"categories\": [4, 5, 6, 7], \"mode\": \"lines\", \"name\": \"Accuracy_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9108750632271118, 0.9108750632271118, 0.9108750632271118, 0.9108750632271118]}]}, \"metricName\": null, \"primaryMetricName\": \"Accuracy\", \"showLegend\": false}, \"run_metrics\": [{\"name\": \"best_child_by_primary_metric\", \"run_id\": \"HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7\", \"categories\": [0], \"series\": [{\"data\": [{\"metric_name\": [\"Accuracy\", \"Accuracy\"], \"timestamp\": [\"2020-11-23 14:46:04.699605+00:00\", \"2020-11-23 14:46:04.699605+00:00\"], \"run_id\": [\"HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3\", \"HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3\"], \"metric_value\": [0.9108750632271118, 0.9108750632271118], \"final\": [false, true]}]}]}], \"run_logs\": \"[2020-11-23T14:39:19.649662][GENERATOR][INFO]Trying to sample '4' jobs from the hyperparameter space\\r\\n[2020-11-23T14:39:19.930344][GENERATOR][INFO]Successfully sampled '4' jobs, they will soon be submitted to the execution target.\\r\\n[2020-11-23T14:39:19.138318][API][INFO]Experiment created\\r\\n[2020-11-23T14:39:20.4789276Z][SCHEDULER][INFO]The execution environment is being prepared. Please be patient as it can take a few minutes.\\r\\n[2020-11-23T14:39:49.883482][GENERATOR][INFO]Max number of jobs '4' reached for experiment.\\r\\n[2020-11-23T14:39:50.185697][GENERATOR][INFO]All jobs generated.\\r\\n[2020-11-23T14:39:52.1597596Z][SCHEDULER][INFO]Scheduling job, id='HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_0'\\r\\n[2020-11-23T14:39:52.1607394Z][SCHEDULER][INFO]Scheduling job, id='HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_1'\\r\\n[2020-11-23T14:39:52.2264007Z][SCHEDULER][INFO]Scheduling job, id='HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_2'\\r\\n[2020-11-23T14:39:52.1590698Z][SCHEDULER][INFO]The execution environment was successfully prepared.\\r\\n[2020-11-23T14:39:52.1847630Z][SCHEDULER][INFO]Scheduling job, id='HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3'\\r\\n[2020-11-23T14:39:53.0180865Z][SCHEDULER][INFO]Successfully scheduled a job. Id='HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3'\\r\\n[2020-11-23T14:39:53.1746666Z][SCHEDULER][INFO]Successfully scheduled a job. Id='HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_0'\\r\\n[2020-11-23T14:39:53.8085239Z][SCHEDULER][INFO]Successfully scheduled a job. Id='HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_1'\\r\\n[2020-11-23T14:39:53.8755342Z][SCHEDULER][INFO]Successfully scheduled a job. Id='HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_2'\\r\\n[2020-11-23T14:44:05.191087][ENFORCER][INFO]Jobs [https://southcentralus.experiments.azureml.net/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/**SCRUBBED**/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_0, https://southcentralus.experiments.azureml.net/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/**SCRUBBED**/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_1, https://southcentralus.experiments.azureml.net/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/**SCRUBBED**/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_2, https://southcentralus.experiments.azureml.net/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/**SCRUBBED**/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3] do not contain any metrics with the primary metric name at this moment, policy cannot be applied.\\r\\n[2020-11-23T14:44:35.762119][ENFORCER][INFO]Jobs [https://southcentralus.experiments.azureml.net/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/**SCRUBBED**/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_0, https://southcentralus.experiments.azureml.net/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/**SCRUBBED**/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_1, https://southcentralus.experiments.azureml.net/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/**SCRUBBED**/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_2, https://southcentralus.experiments.azureml.net/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/**SCRUBBED**/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3] do not contain any metrics with the primary metric name at this moment, policy cannot be applied.\\r\\n[2020-11-23T14:45:05.759796][ENFORCER][INFO]Jobs [https://southcentralus.experiments.azureml.net/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/**SCRUBBED**/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_0, https://southcentralus.experiments.azureml.net/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/**SCRUBBED**/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_1, https://southcentralus.experiments.azureml.net/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/**SCRUBBED**/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_2, https://southcentralus.experiments.azureml.net/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/**SCRUBBED**/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3] do not contain any metrics with the primary metric name at this moment, policy cannot be applied.\\r\\n[2020-11-23T14:45:36.061742][ENFORCER][INFO]Jobs [https://southcentralus.experiments.azureml.net/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/**SCRUBBED**/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_0, https://southcentralus.experiments.azureml.net/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/**SCRUBBED**/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_1, https://southcentralus.experiments.azureml.net/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/**SCRUBBED**/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_2, https://southcentralus.experiments.azureml.net/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourceGroups/aml-quickstarts-127549/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-127549/experiments/**SCRUBBED**/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3] do not contain any metrics with the primary metric name at this moment, policy cannot be applied.\\r\\n[2020-11-23T14:46:50.633251][CONTROLLER][INFO]Experiment was 'ExperimentStatus.RUNNING', is 'ExperimentStatus.FINISHED'.\\n\\nRun is completed.\", \"graph\": {}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"INFO\", \"sdk_version\": \"1.18.0\"}, \"loading\": false}"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RunId: HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7\n",
      "Web View: https://ml.azure.com/experiments/udacity-project/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7?wsid=/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourcegroups/aml-quickstarts-127549/workspaces/quick-starts-ws-127549\n",
      "\n",
      "Streaming azureml-logs/hyperdrive.txt\n",
      "=====================================\n",
      "\n",
      "\"<START>[2020-11-23T14:39:19.649662][GENERATOR][INFO]Trying to sample '4' jobs from the hyperparameter space<END>\\n\"\"<START>[2020-11-23T14:39:19.930344][GENERATOR][INFO]Successfully sampled '4' jobs, they will soon be submitted to the execution target.<END>\\n\"\"<START>[2020-11-23T14:39:19.138318][API][INFO]Experiment created<END>\\n\"<START>[2020-11-23T14:39:20.4789276Z][SCHEDULER][INFO]The execution environment is being prepared. Please be patient as it can take a few minutes.<END>\n",
      "\n",
      "Execution Summary\n",
      "=================\n",
      "RunId: HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7\n",
      "Web View: https://ml.azure.com/experiments/udacity-project/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7?wsid=/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourcegroups/aml-quickstarts-127549/workspaces/quick-starts-ws-127549\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'runId': 'HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7',\n",
       " 'target': 'cpu-cluster',\n",
       " 'status': 'Completed',\n",
       " 'startTimeUtc': '2020-11-23T14:39:18.803266Z',\n",
       " 'endTimeUtc': '2020-11-23T14:46:49.925747Z',\n",
       " 'properties': {'primary_metric_config': '{\"name\": \"Accuracy\", \"goal\": \"maximize\"}',\n",
       "  'resume_from': 'null',\n",
       "  'runTemplate': 'HyperDrive',\n",
       "  'azureml.runsource': 'hyperdrive',\n",
       "  'platform': 'AML',\n",
       "  'ContentSnapshotId': '26c3792e-5253-43f8-a916-e70710c8f45d',\n",
       "  'score': '0.9108750632271118',\n",
       "  'best_child_run_id': 'HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3',\n",
       "  'best_metric_status': 'Succeeded'},\n",
       " 'inputDatasets': [],\n",
       " 'outputDatasets': [],\n",
       " 'logFiles': {'azureml-logs/hyperdrive.txt': 'https://mlstrg127549.blob.core.windows.net/azureml/ExperimentRun/dcid.HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7/azureml-logs/hyperdrive.txt?sv=2019-02-02&sr=b&sig=kBCS7oPWfkz1lAJMl23SHVuC%2Fl4o8SzOXMBQroDogNY%3D&st=2020-11-23T14%3A37%3A10Z&se=2020-11-23T22%3A47%3A10Z&sp=r'}}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Submit hyperdrive run to the experiment \n",
    "\n",
    "hyperdrive_run = exp.submit(config = hyperdrive_config)\n",
    "\n",
    "# Show run details with the Jupyter widget\n",
    "\n",
    "RunDetails(hyperdrive_run).show()\n",
    "hyperdrive_run.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Run Id:  HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3\n",
      "Accuracy: 0.9108750632271118\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table style=\"width:100%\"><tr><th>Experiment</th><th>Id</th><th>Type</th><th>Status</th><th>Details Page</th><th>Docs Page</th></tr><tr><td>udacity-project</td><td>HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3</td><td>azureml.scriptrun</td><td>Completed</td><td><a href=\"https://ml.azure.com/experiments/udacity-project/runs/HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3?wsid=/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourcegroups/aml-quickstarts-127549/workspaces/quick-starts-ws-127549\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.run.Run?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
      ],
      "text/plain": [
       "Run(Experiment: udacity-project,\n",
       "Id: HD_e4a8f3f1-8636-4161-ae5b-58a20a6eadf7_3,\n",
       "Type: azureml.scriptrun,\n",
       "Status: Completed)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import joblib\n",
    "\n",
    "# Get your best run and save the model from that run.\n",
    "\n",
    "best_run = hyperdrive_run.get_best_run_by_primary_metric()\n",
    "best_run_metrics = best_run.get_metrics()\n",
    "\n",
    "print('Best Run Id: ', best_run.id)\n",
    "print('Accuracy:', best_run_metrics['Accuracy'])\n",
    "\n",
    "best_run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Regularization Strength:': 0.7,\n",
       " 'Max iterations:': 100,\n",
       " 'Accuracy': 0.9108750632271118}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check metrics details\n",
    "\n",
    "best_run_metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['azureml-logs/55_azureml-execution-tvmps_dad1ad448bab9b03d12706e0d3dfd042b705522863d87ca5d29363f64da9d37b_d.txt',\n",
       " 'azureml-logs/65_job_prep-tvmps_dad1ad448bab9b03d12706e0d3dfd042b705522863d87ca5d29363f64da9d37b_d.txt',\n",
       " 'azureml-logs/70_driver_log.txt',\n",
       " 'azureml-logs/75_job_post-tvmps_dad1ad448bab9b03d12706e0d3dfd042b705522863d87ca5d29363f64da9d37b_d.txt',\n",
       " 'azureml-logs/process_info.json',\n",
       " 'azureml-logs/process_status.json',\n",
       " 'logs/azureml/102_azureml.log',\n",
       " 'logs/azureml/job_prep_azureml.log',\n",
       " 'logs/azureml/job_release_azureml.log',\n",
       " 'outputs/model.joblib']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get name of files of best_run\n",
    "best_run.get_file_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the model, i.e., output file of best_run\n",
    "model = best_run.register_model(model_name='model_hd', model_path='outputs/model.joblib')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AutoML Run\n",
    "\n",
    "Now we use the same dataset to obtain a model by running AutoML."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.data.dataset_factory import TabularDatasetFactory\n",
    "\n",
    "# Create TabularDataset using TabularDatasetFactory\n",
    "\n",
    "ds = TabularDatasetFactory.from_delimited_files(path = \"https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/bankmarketing_train.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a dataframe with ds data\n",
    "\n",
    "ds_df = ds.to_pandas_dataframe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>job</th>\n",
       "      <th>marital</th>\n",
       "      <th>education</th>\n",
       "      <th>default</th>\n",
       "      <th>housing</th>\n",
       "      <th>loan</th>\n",
       "      <th>contact</th>\n",
       "      <th>month</th>\n",
       "      <th>day_of_week</th>\n",
       "      <th>...</th>\n",
       "      <th>campaign</th>\n",
       "      <th>pdays</th>\n",
       "      <th>previous</th>\n",
       "      <th>poutcome</th>\n",
       "      <th>emp.var.rate</th>\n",
       "      <th>cons.price.idx</th>\n",
       "      <th>cons.conf.idx</th>\n",
       "      <th>euribor3m</th>\n",
       "      <th>nr.employed</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>57</td>\n",
       "      <td>technician</td>\n",
       "      <td>married</td>\n",
       "      <td>high.school</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>cellular</td>\n",
       "      <td>may</td>\n",
       "      <td>mon</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>1</td>\n",
       "      <td>failure</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>92.893</td>\n",
       "      <td>-46.2</td>\n",
       "      <td>1.299</td>\n",
       "      <td>5099.1</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>55</td>\n",
       "      <td>unknown</td>\n",
       "      <td>married</td>\n",
       "      <td>unknown</td>\n",
       "      <td>unknown</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>telephone</td>\n",
       "      <td>may</td>\n",
       "      <td>thu</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.1</td>\n",
       "      <td>93.994</td>\n",
       "      <td>-36.4</td>\n",
       "      <td>4.860</td>\n",
       "      <td>5191.0</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>33</td>\n",
       "      <td>blue-collar</td>\n",
       "      <td>married</td>\n",
       "      <td>basic.9y</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>may</td>\n",
       "      <td>fri</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>1</td>\n",
       "      <td>failure</td>\n",
       "      <td>-1.8</td>\n",
       "      <td>92.893</td>\n",
       "      <td>-46.2</td>\n",
       "      <td>1.313</td>\n",
       "      <td>5099.1</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>36</td>\n",
       "      <td>admin.</td>\n",
       "      <td>married</td>\n",
       "      <td>high.school</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>telephone</td>\n",
       "      <td>jun</td>\n",
       "      <td>fri</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.4</td>\n",
       "      <td>94.465</td>\n",
       "      <td>-41.8</td>\n",
       "      <td>4.967</td>\n",
       "      <td>5228.1</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>27</td>\n",
       "      <td>housemaid</td>\n",
       "      <td>married</td>\n",
       "      <td>high.school</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "      <td>cellular</td>\n",
       "      <td>jul</td>\n",
       "      <td>fri</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>999</td>\n",
       "      <td>0</td>\n",
       "      <td>nonexistent</td>\n",
       "      <td>1.4</td>\n",
       "      <td>93.918</td>\n",
       "      <td>-42.7</td>\n",
       "      <td>4.963</td>\n",
       "      <td>5228.1</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   age          job  marital    education  default housing loan    contact  \\\n",
       "0   57   technician  married  high.school       no      no  yes   cellular   \n",
       "1   55      unknown  married      unknown  unknown     yes   no  telephone   \n",
       "2   33  blue-collar  married     basic.9y       no      no   no   cellular   \n",
       "3   36       admin.  married  high.school       no      no   no  telephone   \n",
       "4   27    housemaid  married  high.school       no     yes   no   cellular   \n",
       "\n",
       "  month day_of_week  ...  campaign  pdays  previous     poutcome emp.var.rate  \\\n",
       "0   may         mon  ...         1    999         1      failure         -1.8   \n",
       "1   may         thu  ...         2    999         0  nonexistent          1.1   \n",
       "2   may         fri  ...         1    999         1      failure         -1.8   \n",
       "3   jun         fri  ...         4    999         0  nonexistent          1.4   \n",
       "4   jul         fri  ...         2    999         0  nonexistent          1.4   \n",
       "\n",
       "   cons.price.idx  cons.conf.idx  euribor3m  nr.employed   y  \n",
       "0          92.893          -46.2      1.299       5099.1  no  \n",
       "1          93.994          -36.4      4.860       5191.0  no  \n",
       "2          92.893          -46.2      1.313       5099.1  no  \n",
       "3          94.465          -41.8      4.967       5228.1  no  \n",
       "4          93.918          -42.7      4.963       5228.1  no  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 32950 entries, 0 to 32949\n",
      "Data columns (total 21 columns):\n",
      "age               32950 non-null int64\n",
      "job               32950 non-null object\n",
      "marital           32950 non-null object\n",
      "education         32950 non-null object\n",
      "default           32950 non-null object\n",
      "housing           32950 non-null object\n",
      "loan              32950 non-null object\n",
      "contact           32950 non-null object\n",
      "month             32950 non-null object\n",
      "day_of_week       32950 non-null object\n",
      "duration          32950 non-null int64\n",
      "campaign          32950 non-null int64\n",
      "pdays             32950 non-null int64\n",
      "previous          32950 non-null int64\n",
      "poutcome          32950 non-null object\n",
      "emp.var.rate      32950 non-null float64\n",
      "cons.price.idx    32950 non-null float64\n",
      "cons.conf.idx     32950 non-null float64\n",
      "euribor3m         32950 non-null float64\n",
      "nr.employed       32950 non-null float64\n",
      "y                 32950 non-null object\n",
      "dtypes: float64(5), int64(5), object(11)\n",
      "memory usage: 5.3+ MB\n"
     ]
    }
   ],
   "source": [
    "ds_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "no     0.887951\n",
       "yes    0.112049\n",
       "Name: y, dtype: float64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds_df.y.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset contains 32950 entries and 21 columns where one of the column is our target (`y`). This dataset is a bit imbalanced, therefore I'm applying stratify when spliting the data so we can have the same percentage of each class in train and test dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean and Split Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from train import clean_data\n",
    "\n",
    "# Use the clean_data function to clean your data.\n",
    "x, y = clean_data(ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# split data into train and test sets - \n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.30, random_state=123, stratify = y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new dataframe with only training data\n",
    "\n",
    "df_train = pd.concat([x_train,y_train], axis=1)\n",
    "df_train.reset_index(drop=True, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>marital</th>\n",
       "      <th>default</th>\n",
       "      <th>housing</th>\n",
       "      <th>loan</th>\n",
       "      <th>month</th>\n",
       "      <th>day_of_week</th>\n",
       "      <th>duration</th>\n",
       "      <th>campaign</th>\n",
       "      <th>pdays</th>\n",
       "      <th>...</th>\n",
       "      <th>contact_telephone</th>\n",
       "      <th>education_basic.4y</th>\n",
       "      <th>education_basic.6y</th>\n",
       "      <th>education_basic.9y</th>\n",
       "      <th>education_high.school</th>\n",
       "      <th>education_illiterate</th>\n",
       "      <th>education_professional.course</th>\n",
       "      <th>education_university.degree</th>\n",
       "      <th>education_unknown</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>31</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>161</td>\n",
       "      <td>5</td>\n",
       "      <td>999</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>48</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>3</td>\n",
       "      <td>1061</td>\n",
       "      <td>4</td>\n",
       "      <td>999</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>134</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>36</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>347</td>\n",
       "      <td>1</td>\n",
       "      <td>999</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>12</td>\n",
       "      <td>10</td>\n",
       "      <td>999</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 40 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  marital  default  housing  loan  month  day_of_week  duration  \\\n",
       "0   31        1        0        1     0      5            2       161   \n",
       "1   48        1        0        0     0     11            3      1061   \n",
       "2   32        1        0        1     1      5            5       134   \n",
       "3   36        0        0        1     1      5            2       347   \n",
       "4   31        0        0        1     0      5            5        12   \n",
       "\n",
       "   campaign  pdays  ...  contact_telephone  education_basic.4y  \\\n",
       "0         5    999  ...                  0                   0   \n",
       "1         4    999  ...                  0                   0   \n",
       "2         1    999  ...                  0                   0   \n",
       "3         1    999  ...                  0                   0   \n",
       "4        10    999  ...                  0                   0   \n",
       "\n",
       "   education_basic.6y  education_basic.9y  education_high.school  \\\n",
       "0                   0                   1                      0   \n",
       "1                   0                   0                      1   \n",
       "2                   0                   0                      0   \n",
       "3                   0                   0                      0   \n",
       "4                   0                   1                      0   \n",
       "\n",
       "   education_illiterate  education_professional.course  \\\n",
       "0                     0                              0   \n",
       "1                     0                              0   \n",
       "2                     0                              1   \n",
       "3                     0                              0   \n",
       "4                     0                              0   \n",
       "\n",
       "   education_university.degree  education_unknown  y  \n",
       "0                            0                  0  0  \n",
       "1                            0                  0  1  \n",
       "2                            0                  0  0  \n",
       "3                            1                  0  0  \n",
       "4                            0                  0  0  \n",
       "\n",
       "[5 rows x 40 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 23065 entries, 0 to 23064\n",
      "Data columns (total 40 columns):\n",
      "age                              23065 non-null int64\n",
      "marital                          23065 non-null int64\n",
      "default                          23065 non-null int64\n",
      "housing                          23065 non-null int64\n",
      "loan                             23065 non-null int64\n",
      "month                            23065 non-null int64\n",
      "day_of_week                      23065 non-null int64\n",
      "duration                         23065 non-null int64\n",
      "campaign                         23065 non-null int64\n",
      "pdays                            23065 non-null int64\n",
      "previous                         23065 non-null int64\n",
      "poutcome                         23065 non-null int64\n",
      "emp.var.rate                     23065 non-null float64\n",
      "cons.price.idx                   23065 non-null float64\n",
      "cons.conf.idx                    23065 non-null float64\n",
      "euribor3m                        23065 non-null float64\n",
      "nr.employed                      23065 non-null float64\n",
      "job_admin.                       23065 non-null uint8\n",
      "job_blue-collar                  23065 non-null uint8\n",
      "job_entrepreneur                 23065 non-null uint8\n",
      "job_housemaid                    23065 non-null uint8\n",
      "job_management                   23065 non-null uint8\n",
      "job_retired                      23065 non-null uint8\n",
      "job_self-employed                23065 non-null uint8\n",
      "job_services                     23065 non-null uint8\n",
      "job_student                      23065 non-null uint8\n",
      "job_technician                   23065 non-null uint8\n",
      "job_unemployed                   23065 non-null uint8\n",
      "job_unknown                      23065 non-null uint8\n",
      "contact_cellular                 23065 non-null uint8\n",
      "contact_telephone                23065 non-null uint8\n",
      "education_basic.4y               23065 non-null uint8\n",
      "education_basic.6y               23065 non-null uint8\n",
      "education_basic.9y               23065 non-null uint8\n",
      "education_high.school            23065 non-null uint8\n",
      "education_illiterate             23065 non-null uint8\n",
      "education_professional.course    23065 non-null uint8\n",
      "education_university.degree      23065 non-null uint8\n",
      "education_unknown                23065 non-null uint8\n",
      "y                                23065 non-null int64\n",
      "dtypes: float64(5), int64(13), uint8(22)\n",
      "memory usage: 3.7 MB\n"
     ]
    }
   ],
   "source": [
    "df_train.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure Experiment "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "gather": {
     "logged": 1598275665403
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "from azureml.train.automl import AutoMLConfig\n",
    "\n",
    "# Set parameters for AutoMLConfig\n",
    "# NOTE: DO NOT CHANGE THE experiment_timeout_minutes PARAMETER OR YOUR INSTANCE WILL TIME OUT.\n",
    "# If you wish to run the experiment longer, you will need to run this notebook in your own\n",
    "# Azure tenant, which will incur personal costs.\n",
    "\n",
    "automl_config = AutoMLConfig(\n",
    "    experiment_timeout_minutes=30,\n",
    "    task=\"classification\",\n",
    "    primary_metric=\"accuracy\",\n",
    "    training_data=df_train,\n",
    "    label_column_name='y',\n",
    "    n_cross_validations=5)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submitting Training Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on local machine\n",
      "Parent Run ID: AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\n",
      "\n",
      "Current status: DatasetEvaluation. Gathering dataset statistics.\n",
      "Current status: FeaturesGeneration. Generating features for the dataset.\n",
      "Current status: DatasetFeaturization. Beginning to fit featurizers and featurize the dataset.\n",
      "Current status: DatasetFeaturizationCompleted. Completed fit featurizers and featurizing the dataset.\n",
      "Current status: DatasetBalancing. Performing class balancing sweeping\n",
      "Current status: DatasetCrossValidationSplit. Generating individually featurized CV splits.\n",
      "\n",
      "****************************************************************************************************\n",
      "DATA GUARDRAILS: \n",
      "\n",
      "TYPE:         Class balancing detection\n",
      "STATUS:       ALERTED\n",
      "DESCRIPTION:  To decrease model bias, please cancel the current run and fix balancing problem.\n",
      "              Learn more about imbalanced data: https://aka.ms/AutomatedMLImbalancedData\n",
      "DETAILS:      Imbalanced data can lead to a falsely perceived positive effect of a model's accuracy because the input data has bias towards one class.\n",
      "+---------------------------------+---------------------------------+--------------------------------------+\n",
      "|Size of the smallest class       |Name/Label of the smallest class |Number of samples in the training data|\n",
      "+=================================+=================================+======================================+\n",
      "|2584                             |1                                |23065                                 |\n",
      "+---------------------------------+---------------------------------+--------------------------------------+\n",
      "\n",
      "****************************************************************************************************\n",
      "\n",
      "TYPE:         Missing feature values imputation\n",
      "STATUS:       PASSED\n",
      "DESCRIPTION:  No feature missing values were detected in the training data.\n",
      "              Learn more about missing value imputation: https://aka.ms/AutomatedMLFeaturization\n",
      "\n",
      "****************************************************************************************************\n",
      "\n",
      "TYPE:         High cardinality feature detection\n",
      "STATUS:       PASSED\n",
      "DESCRIPTION:  Your inputs were analyzed, and no high cardinality features were detected.\n",
      "              Learn more about high cardinality feature handling: https://aka.ms/AutomatedMLFeaturization\n",
      "\n",
      "****************************************************************************************************\n",
      "Current status: ModelSelection. Beginning model selection.\n",
      "\n",
      "****************************************************************************************************\n",
      "ITERATION: The iteration being evaluated.\n",
      "PIPELINE: A summary description of the pipeline being evaluated.\n",
      "DURATION: Time taken for the current iteration.\n",
      "METRIC: The result of computing score on the fitted pipeline.\n",
      "BEST: The best observed score thus far.\n",
      "****************************************************************************************************\n",
      "\n",
      " ITERATION   PIPELINE                                       DURATION      METRIC      BEST\n",
      "         0   MaxAbsScaler LightGBM                          0:00:40       0.9142    0.9142\n",
      "         1   MaxAbsScaler XGBoostClassifier                 0:00:38       0.9146    0.9146\n",
      "         2   MaxAbsScaler RandomForest                      0:00:31       0.8921    0.9146\n",
      "         3   MaxAbsScaler RandomForest                      0:00:32       0.8880    0.9146\n",
      "         4   MaxAbsScaler SGD                               0:00:26       0.8704    0.9146\n",
      "         5   MaxAbsScaler SGD                               0:00:24       0.9061    0.9146\n",
      "         6   MaxAbsScaler ExtremeRandomTrees                0:00:26       0.8996    0.9146\n",
      "         7   MaxAbsScaler ExtremeRandomTrees                0:00:41       0.9008    0.9146\n",
      "         8   MaxAbsScaler ExtremeRandomTrees                0:00:30       0.9002    0.9146\n",
      "         9   MaxAbsScaler ExtremeRandomTrees                0:00:32       0.7605    0.9146\n",
      "        10   MaxAbsScaler SGD                               0:00:25       0.9031    0.9146\n",
      "        11   MaxAbsScaler SGD                               0:00:31       0.9038    0.9146\n",
      "        12   MaxAbsScaler RandomForest                      0:00:27       0.8892    0.9146\n",
      "        13   StandardScalerWrapper ExtremeRandomTrees       0:00:33       0.8880    0.9146\n",
      "        14   MaxAbsScaler RandomForest                      0:00:27       0.7944    0.9146\n",
      "        15   MaxAbsScaler SGD                               0:00:27       0.8632    0.9146\n",
      "        16   MaxAbsScaler RandomForest                      0:00:32       0.8880    0.9146\n",
      "        17   MaxAbsScaler ExtremeRandomTrees                0:00:33       0.8964    0.9146\n",
      "        18   SparseNormalizer ExtremeRandomTrees            0:00:40       0.7270    0.9146\n",
      "        19   MaxAbsScaler SGD                               0:00:39       0.9067    0.9146\n",
      "        20   MaxAbsScaler ExtremeRandomTrees                0:00:29       0.7553    0.9146\n",
      "        21   MaxAbsScaler RandomForest                      0:00:28       0.7898    0.9146\n",
      "        22   MaxAbsScaler LightGBM                          0:00:33       0.9127    0.9146\n",
      "        23   MaxAbsScaler ExtremeRandomTrees                0:00:44       0.8880    0.9146\n",
      "        24   MaxAbsScaler ExtremeRandomTrees                0:01:07       0.8900    0.9146\n",
      "        25   TruncatedSVDWrapper XGBoostClassifier          0:00:30       0.8880    0.9146\n",
      "        26   MaxAbsScaler ExtremeRandomTrees                0:00:35       0.9002    0.9146\n",
      "        27   SparseNormalizer XGBoostClassifier             0:00:45       0.9131    0.9146\n",
      "        28   MaxAbsScaler ExtremeRandomTrees                0:00:38       0.8880    0.9146\n",
      "        29   MaxAbsScaler LightGBM                          0:00:33       0.9054    0.9146\n",
      "        30   TruncatedSVDWrapper XGBoostClassifier          0:00:35       0.8880    0.9146\n",
      "        31   StandardScalerWrapper ExtremeRandomTrees       0:00:48       0.8997    0.9146\n",
      "        32   SparseNormalizer LightGBM                      0:00:34       0.9023    0.9146\n",
      "        33   StandardScalerWrapper XGBoostClassifier        0:00:42       0.9130    0.9146\n",
      "        34   SparseNormalizer XGBoostClassifier             0:01:17       0.9127    0.9146\n",
      "        35   TruncatedSVDWrapper XGBoostClassifier          0:00:31       0.8880    0.9146\n",
      "        36   MaxAbsScaler LightGBM                          0:00:30       0.9099    0.9146\n",
      "        37   MaxAbsScaler ExtremeRandomTrees                0:00:52       0.9002    0.9146\n",
      "        38   StandardScalerWrapper LightGBM                 0:00:27       0.9042    0.9146\n",
      "        39   StandardScalerWrapper LightGBM                 0:00:31       0.9063    0.9146\n",
      "        40   StandardScalerWrapper XGBoostClassifier        0:00:25       0.8929    0.9146\n",
      "        41   SparseNormalizer XGBoostClassifier             0:00:37       0.9116    0.9146\n",
      "        42   SparseNormalizer XGBoostClassifier             0:00:27       0.9122    0.9146\n",
      "        43   SparseNormalizer XGBoostClassifier             0:01:04       0.9119    0.9146\n",
      "        44   MaxAbsScaler LightGBM                          0:00:27       0.9067    0.9146\n",
      "        45   StandardScalerWrapper XGBoostClassifier        0:00:33       0.8880    0.9146\n",
      "        46   SparseNormalizer XGBoostClassifier             0:00:40       0.9130    0.9146\n",
      "        47   VotingEnsemble                                 0:01:12       0.9174    0.9174\n",
      "        48   StackEnsemble                                  0:01:24       0.9138    0.9174\n",
      "Stopping criteria reached at iteration 49. Ending experiment.\n",
      "****************************************************************************************************\n",
      "Current status: BestRunExplainModel. Best run model explanations started\n",
      "Current status: ModelExplanationDataSetSetup. Model explanations data setup completed\n",
      "Current status: PickSurrogateModel. Choosing LightGBM as the surrogate model for explanations\n",
      "Current status: EngineeredFeatureExplanations. Computation of engineered features started\n",
      "Current status: EngineeredFeatureExplanations. Computation of engineered features completed\n",
      "Current status: RawFeaturesExplanations. Computation of raw features started\n",
      "Current status: RawFeaturesExplanations. Computation of raw features completed\n",
      "Current status: BestRunExplainModel. Best run model explanations completed\n",
      "****************************************************************************************************\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table style=\"width:100%\"><tr><th>Experiment</th><th>Id</th><th>Type</th><th>Status</th><th>Details Page</th><th>Docs Page</th></tr><tr><td>automl-experiment</td><td>AutoML_770c77d2-9751-454d-8d86-98b0c225ada9</td><td>automl</td><td>Completed</td><td><a href=\"https://ml.azure.com/experiments/automl-experiment/runs/AutoML_770c77d2-9751-454d-8d86-98b0c225ada9?wsid=/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourcegroups/aml-quickstarts-127549/workspaces/quick-starts-ws-127549\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/overview/azure/ml/intro?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
      ],
      "text/plain": [
       "Run(Experiment: automl-experiment,\n",
       "Id: AutoML_770c77d2-9751-454d-8d86-98b0c225ada9,\n",
       "Type: automl,\n",
       "Status: Completed)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Submit your automl run\n",
    "\n",
    "experiment_name = 'automl-experiment'\n",
    "\n",
    "experiment = Experiment(ws, experiment_name)\n",
    "automl_run = experiment.submit(config=automl_config, show_output=True)\n",
    "automl_run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monitor using `Widget`\n",
    "\n",
    "Once more we make use of `widget`. This time to explore the results obtained by using AutoML."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5394a452441b4feabc7e569f6c0f38a9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_AutoMLWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', 's…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/experiments/automl-experiment/runs/AutoML_770c77d2-9751-454d-8d86-98b0c225ada9?wsid=/subscriptions/55e71b9d-a209-42c0-8818-ca9cc885909c/resourcegroups/aml-quickstarts-127549/workspaces/quick-starts-ws-127549\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"run_properties\": {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"created_utc\": \"2020-11-23T14:53:42.96724Z\", \"properties\": {\"num_iterations\": \"1000\", \"training_type\": \"TrainFull\", \"acquisition_function\": \"EI\", \"primary_metric\": \"accuracy\", \"train_split\": \"0\", \"acquisition_parameter\": \"0\", \"num_cross_validation\": \"5\", \"target\": \"local\", \"AMLSettingsJsonString\": \"{\\\"path\\\":null,\\\"name\\\":\\\"automl-experiment\\\",\\\"subscription_id\\\":\\\"55e71b9d-a209-42c0-8818-ca9cc885909c\\\",\\\"resource_group\\\":\\\"aml-quickstarts-127549\\\",\\\"workspace_name\\\":\\\"quick-starts-ws-127549\\\",\\\"region\\\":\\\"southcentralus\\\",\\\"compute_target\\\":\\\"local\\\",\\\"spark_service\\\":null,\\\"azure_service\\\":\\\"Microsoft.AzureNotebookVM\\\",\\\"_local_managed_run_id\\\":null,\\\"many_models\\\":false,\\\"pipeline_fetch_max_batch_size\\\":1,\\\"iterations\\\":1000,\\\"primary_metric\\\":\\\"accuracy\\\",\\\"task_type\\\":\\\"classification\\\",\\\"data_script\\\":null,\\\"validation_size\\\":0.0,\\\"n_cross_validations\\\":5,\\\"y_min\\\":null,\\\"y_max\\\":null,\\\"num_classes\\\":2,\\\"featurization\\\":\\\"auto\\\",\\\"_ignore_package_version_incompatibilities\\\":false,\\\"is_timeseries\\\":false,\\\"max_cores_per_iteration\\\":1,\\\"max_concurrent_iterations\\\":1,\\\"iteration_timeout_minutes\\\":null,\\\"mem_in_mb\\\":null,\\\"enforce_time_on_windows\\\":false,\\\"experiment_timeout_minutes\\\":30,\\\"experiment_exit_score\\\":null,\\\"whitelist_models\\\":null,\\\"blacklist_algos\\\":[\\\"TensorFlowLinearClassifier\\\",\\\"TensorFlowDNN\\\"],\\\"supported_models\\\":[\\\"LinearSVM\\\",\\\"SGD\\\",\\\"BernoulliNaiveBayes\\\",\\\"GradientBoosting\\\",\\\"KNN\\\",\\\"SVM\\\",\\\"AveragedPerceptronClassifier\\\",\\\"TensorFlowLinearClassifier\\\",\\\"MultinomialNaiveBayes\\\",\\\"RandomForest\\\",\\\"LogisticRegression\\\",\\\"DecisionTree\\\",\\\"XGBoostClassifier\\\",\\\"TensorFlowDNN\\\",\\\"ExtremeRandomTrees\\\",\\\"LightGBM\\\"],\\\"auto_blacklist\\\":true,\\\"blacklist_samples_reached\\\":false,\\\"exclude_nan_labels\\\":true,\\\"verbosity\\\":20,\\\"_debug_log\\\":\\\"automl.log\\\",\\\"show_warnings\\\":false,\\\"model_explainability\\\":true,\\\"service_url\\\":null,\\\"sdk_url\\\":null,\\\"sdk_packages\\\":null,\\\"enable_onnx_compatible_models\\\":false,\\\"enable_split_onnx_featurizer_estimator_models\\\":false,\\\"vm_type\\\":null,\\\"telemetry_verbosity\\\":20,\\\"send_telemetry\\\":true,\\\"enable_dnn\\\":false,\\\"scenario\\\":\\\"SDK-1.13.0\\\",\\\"environment_label\\\":null,\\\"force_text_dnn\\\":false,\\\"enable_feature_sweeping\\\":true,\\\"enable_early_stopping\\\":false,\\\"early_stopping_n_iters\\\":10,\\\"metrics\\\":null,\\\"enable_ensembling\\\":true,\\\"enable_stack_ensembling\\\":true,\\\"ensemble_iterations\\\":15,\\\"enable_tf\\\":false,\\\"enable_subsampling\\\":null,\\\"subsample_seed\\\":null,\\\"enable_nimbusml\\\":false,\\\"enable_streaming\\\":false,\\\"force_streaming\\\":false,\\\"track_child_runs\\\":true,\\\"allowed_private_models\\\":[],\\\"label_column_name\\\":\\\"y\\\",\\\"weight_column_name\\\":null,\\\"cv_split_column_names\\\":null,\\\"enable_local_managed\\\":false,\\\"cost_mode\\\":1,\\\"lag_length\\\":0,\\\"metric_operation\\\":\\\"maximize\\\",\\\"preprocess\\\":true}\", \"DataPrepJsonString\": null, \"EnableSubsampling\": null, \"runTemplate\": \"AutoML\", \"azureml.runsource\": \"automl\", \"display_task_type\": \"classification\", \"dependencies_versions\": \"{\\\"azureml-widgets\\\": \\\"1.18.0\\\", \\\"azureml-train\\\": \\\"1.18.0\\\", \\\"azureml-train-restclients-hyperdrive\\\": \\\"1.18.0\\\", \\\"azureml-train-core\\\": \\\"1.18.0\\\", \\\"azureml-train-automl\\\": \\\"1.18.0\\\", \\\"azureml-train-automl-runtime\\\": \\\"1.18.0\\\", \\\"azureml-train-automl-client\\\": \\\"1.18.0\\\", \\\"azureml-tensorboard\\\": \\\"1.18.0\\\", \\\"azureml-telemetry\\\": \\\"1.18.0\\\", \\\"azureml-sdk\\\": \\\"1.18.0\\\", \\\"azureml-samples\\\": \\\"0+unknown\\\", \\\"azureml-pipeline\\\": \\\"1.18.0\\\", \\\"azureml-pipeline-steps\\\": \\\"1.18.0\\\", \\\"azureml-pipeline-core\\\": \\\"1.18.0\\\", \\\"azureml-opendatasets\\\": \\\"1.18.0\\\", \\\"azureml-model-management-sdk\\\": \\\"1.0.1b6.post1\\\", \\\"azureml-mlflow\\\": \\\"1.18.0\\\", \\\"azureml-interpret\\\": \\\"1.18.0\\\", \\\"azureml-explain-model\\\": \\\"1.18.0\\\", \\\"azureml-defaults\\\": \\\"1.18.0\\\", \\\"azureml-dataset-runtime\\\": \\\"1.18.0\\\", \\\"azureml-dataprep\\\": \\\"2.4.2\\\", \\\"azureml-dataprep-rslex\\\": \\\"1.2.2\\\", \\\"azureml-dataprep-native\\\": \\\"24.0.0\\\", \\\"azureml-datadrift\\\": \\\"1.18.0\\\", \\\"azureml-core\\\": \\\"1.18.0\\\", \\\"azureml-contrib-services\\\": \\\"1.18.0\\\", \\\"azureml-contrib-server\\\": \\\"1.18.0\\\", \\\"azureml-contrib-reinforcementlearning\\\": \\\"1.18.0\\\", \\\"azureml-contrib-pipeline-steps\\\": \\\"1.18.0\\\", \\\"azureml-contrib-notebook\\\": \\\"1.18.0\\\", \\\"azureml-contrib-interpret\\\": \\\"1.18.0\\\", \\\"azureml-contrib-gbdt\\\": \\\"1.18.0\\\", \\\"azureml-contrib-fairness\\\": \\\"1.18.0\\\", \\\"azureml-contrib-dataset\\\": \\\"1.18.0\\\", \\\"azureml-cli-common\\\": \\\"1.18.0\\\", \\\"azureml-automl-runtime\\\": \\\"1.18.0\\\", \\\"azureml-automl-core\\\": \\\"1.18.0\\\", \\\"azureml-accel-models\\\": \\\"1.18.0\\\"}\", \"_aml_system_scenario_identification\": \"Local.Parent\", \"ClientSdkVersion\": \"1.18.0\", \"ClientType\": \"SDK\", \"environment_cpu_name\": \"AzureML-AutoML\", \"environment_cpu_version\": \"44\", \"environment_gpu_name\": \"AzureML-AutoML-GPU\", \"environment_gpu_version\": \"32\", \"root_attribution\": \"automl\", \"attribution\": \"AutoML\", \"Orchestrator\": \"AutoML\", \"_azureml.ComputeTargetType\": \"local\", \"ProblemInfoJsonString\": \"{\\\"dataset_num_categorical\\\": 0, \\\"is_sparse\\\": true, \\\"subsampling\\\": false, \\\"dataset_classes\\\": 2, \\\"dataset_features\\\": 120, \\\"dataset_samples\\\": 23065, \\\"single_frequency_class_detected\\\": false}\", \"feature_skus\": \"automatedml_sdk_guardrails\"}, \"tags\": {\"model_explain_run\": \"best_run\", \"best_score\": \"0.9174073271190115\", \"best_pipeline\": \"VotingEnsemble\", \"automl_best_child_run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_47\", \"model_explain_best_run_child_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_47\"}, \"end_time_utc\": \"2020-11-23T15:26:42.769357Z\", \"status\": \"Completed\", \"log_files\": {}, \"log_groups\": [], \"run_duration\": \"0:32:59\"}, \"child_runs\": [{\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_0\", \"run_number\": 2, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T14:54:23.012358Z\", \"end_time\": \"2020-11-23T14:55:03.270894Z\", \"created_time\": \"2020-11-23T14:54:22.900059Z\", \"created_time_dt\": \"2020-11-23T14:54:22.900059Z\", \"duration\": \"0:00:40\", \"iteration\": \"0\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, LightGBM\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.91415565, \"best_metric\": 0.91415565}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_1\", \"run_number\": 3, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T14:55:03.700Z\", \"end_time\": \"2020-11-23T14:55:41.927571Z\", \"created_time\": \"2020-11-23T14:55:03.583949Z\", \"created_time_dt\": \"2020-11-23T14:55:03.583949Z\", \"duration\": \"0:00:38\", \"iteration\": \"1\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, XGBoostClassifier\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.91463256, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_2\", \"run_number\": 4, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T14:55:42.594262Z\", \"end_time\": \"2020-11-23T14:56:13.518607Z\", \"created_time\": \"2020-11-23T14:55:42.486203Z\", \"created_time_dt\": \"2020-11-23T14:55:42.486203Z\", \"duration\": \"0:00:31\", \"iteration\": \"2\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, RandomForest\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.89213093, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_3\", \"run_number\": 5, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T14:56:16.191675Z\", \"end_time\": \"2020-11-23T14:56:48.424937Z\", \"created_time\": \"2020-11-23T14:56:15.724196Z\", \"created_time_dt\": \"2020-11-23T14:56:15.724196Z\", \"duration\": \"0:00:32\", \"iteration\": \"3\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, RandomForest\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.88796878, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_4\", \"run_number\": 6, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T14:56:50.965292Z\", \"end_time\": \"2020-11-23T14:57:17.464965Z\", \"created_time\": \"2020-11-23T14:56:50.554281Z\", \"created_time_dt\": \"2020-11-23T14:56:50.554281Z\", \"duration\": \"0:00:26\", \"iteration\": \"4\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, SGD\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.87036636, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_5\", \"run_number\": 7, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T14:57:18.846587Z\", \"end_time\": \"2020-11-23T14:57:43.520875Z\", \"created_time\": \"2020-11-23T14:57:18.734517Z\", \"created_time_dt\": \"2020-11-23T14:57:18.734517Z\", \"duration\": \"0:00:24\", \"iteration\": \"5\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, SGD\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.90609148, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_6\", \"run_number\": 8, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T14:57:45.041425Z\", \"end_time\": \"2020-11-23T14:58:10.970241Z\", \"created_time\": \"2020-11-23T14:57:44.915936Z\", \"created_time_dt\": \"2020-11-23T14:57:44.915936Z\", \"duration\": \"0:00:26\", \"iteration\": \"6\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, ExtremeRandomTrees\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.89958812, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_7\", \"run_number\": 9, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T14:58:13.761785Z\", \"end_time\": \"2020-11-23T14:58:54.359632Z\", \"created_time\": \"2020-11-23T14:58:13.28344Z\", \"created_time_dt\": \"2020-11-23T14:58:13.28344Z\", \"duration\": \"0:00:41\", \"iteration\": \"7\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, ExtremeRandomTrees\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.90080208, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_8\", \"run_number\": 10, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T14:58:57.22936Z\", \"end_time\": \"2020-11-23T14:59:27.381997Z\", \"created_time\": \"2020-11-23T14:58:56.796156Z\", \"created_time_dt\": \"2020-11-23T14:58:56.796156Z\", \"duration\": \"0:00:30\", \"iteration\": \"8\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, ExtremeRandomTrees\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.90023846, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_9\", \"run_number\": 11, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T14:59:28.964311Z\", \"end_time\": \"2020-11-23T15:00:01.046263Z\", \"created_time\": \"2020-11-23T14:59:28.857213Z\", \"created_time_dt\": \"2020-11-23T14:59:28.857213Z\", \"duration\": \"0:00:32\", \"iteration\": \"9\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, ExtremeRandomTrees\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.76054628, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_10\", \"run_number\": 12, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:00:02.60792Z\", \"end_time\": \"2020-11-23T15:00:27.857202Z\", \"created_time\": \"2020-11-23T15:00:02.473747Z\", \"created_time_dt\": \"2020-11-23T15:00:02.473747Z\", \"duration\": \"0:00:25\", \"iteration\": \"10\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, SGD\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.90314329, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_11\", \"run_number\": 13, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:00:29.098764Z\", \"end_time\": \"2020-11-23T15:01:00.440424Z\", \"created_time\": \"2020-11-23T15:00:28.973511Z\", \"created_time_dt\": \"2020-11-23T15:00:28.973511Z\", \"duration\": \"0:00:31\", \"iteration\": \"11\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, SGD\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.90379363, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_12\", \"run_number\": 14, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:01:01.803758Z\", \"end_time\": \"2020-11-23T15:01:28.835202Z\", \"created_time\": \"2020-11-23T15:01:01.692335Z\", \"created_time_dt\": \"2020-11-23T15:01:01.692335Z\", \"duration\": \"0:00:27\", \"iteration\": \"12\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, RandomForest\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.88918274, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_13\", \"run_number\": 15, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:01:30.247773Z\", \"end_time\": \"2020-11-23T15:02:03.821737Z\", \"created_time\": \"2020-11-23T15:01:30.12696Z\", \"created_time_dt\": \"2020-11-23T15:01:30.12696Z\", \"duration\": \"0:00:33\", \"iteration\": \"13\", \"goal\": \"accuracy_max\", \"run_name\": \"StandardScalerWrapper, ExtremeRandomTrees\", \"run_properties\": \"<azureml.automl.runtime.shared.model_wrappers.StandardScalerWrapper object at 0x7f987e352470\", \"primary_metric\": 0.88796878, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_14\", \"run_number\": 16, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:02:05.169703Z\", \"end_time\": \"2020-11-23T15:02:32.679966Z\", \"created_time\": \"2020-11-23T15:02:05.045193Z\", \"created_time_dt\": \"2020-11-23T15:02:05.045193Z\", \"duration\": \"0:00:27\", \"iteration\": \"14\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, RandomForest\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.79436375, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_15\", \"run_number\": 17, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:02:34.131955Z\", \"end_time\": \"2020-11-23T15:03:01.436297Z\", \"created_time\": \"2020-11-23T15:02:34.020209Z\", \"created_time_dt\": \"2020-11-23T15:02:34.020209Z\", \"duration\": \"0:00:27\", \"iteration\": \"15\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, SGD\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.86321266, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_16\", \"run_number\": 18, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:03:02.976569Z\", \"end_time\": \"2020-11-23T15:03:35.685091Z\", \"created_time\": \"2020-11-23T15:03:02.864547Z\", \"created_time_dt\": \"2020-11-23T15:03:02.864547Z\", \"duration\": \"0:00:32\", \"iteration\": \"16\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, RandomForest\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.88796878, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_17\", \"run_number\": 19, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:03:41.701361Z\", \"end_time\": \"2020-11-23T15:04:15.487422Z\", \"created_time\": \"2020-11-23T15:03:41.57532Z\", \"created_time_dt\": \"2020-11-23T15:03:41.57532Z\", \"duration\": \"0:00:33\", \"iteration\": \"17\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, ExtremeRandomTrees\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.8963798, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_18\", \"run_number\": 20, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:04:17.049145Z\", \"end_time\": \"2020-11-23T15:04:57.117917Z\", \"created_time\": \"2020-11-23T15:04:16.941832Z\", \"created_time_dt\": \"2020-11-23T15:04:16.941832Z\", \"duration\": \"0:00:40\", \"iteration\": \"18\", \"goal\": \"accuracy_max\", \"run_name\": \"SparseNormalizer, ExtremeRandomTrees\", \"run_properties\": \"<azureml.automl.runtime.shared.model_wrappers.SparseNormalizer object at 0x7f987e199978\", \"primary_metric\": 0.72698894, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_19\", \"run_number\": 21, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:05:00.943101Z\", \"end_time\": \"2020-11-23T15:05:39.834421Z\", \"created_time\": \"2020-11-23T15:05:00.503714Z\", \"created_time_dt\": \"2020-11-23T15:05:00.503714Z\", \"duration\": \"0:00:39\", \"iteration\": \"19\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, SGD\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.90669846, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_20\", \"run_number\": 22, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:05:41.77106Z\", \"end_time\": \"2020-11-23T15:06:11.58266Z\", \"created_time\": \"2020-11-23T15:05:41.438558Z\", \"created_time_dt\": \"2020-11-23T15:05:41.438558Z\", \"duration\": \"0:00:30\", \"iteration\": \"20\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, ExtremeRandomTrees\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.75525688, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_21\", \"run_number\": 23, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:06:13.175091Z\", \"end_time\": \"2020-11-23T15:06:41.678169Z\", \"created_time\": \"2020-11-23T15:06:13.036585Z\", \"created_time_dt\": \"2020-11-23T15:06:13.036585Z\", \"duration\": \"0:00:28\", \"iteration\": \"21\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, RandomForest\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.78976805, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_22\", \"run_number\": 24, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:06:43.676766Z\", \"end_time\": \"2020-11-23T15:07:17.542922Z\", \"created_time\": \"2020-11-23T15:06:43.546159Z\", \"created_time_dt\": \"2020-11-23T15:06:43.546159Z\", \"duration\": \"0:00:33\", \"iteration\": \"22\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, LightGBM\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.91268155, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_23\", \"run_number\": 25, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:07:18.799935Z\", \"end_time\": \"2020-11-23T15:08:03.180953Z\", \"created_time\": \"2020-11-23T15:07:18.665009Z\", \"created_time_dt\": \"2020-11-23T15:07:18.665009Z\", \"duration\": \"0:00:44\", \"iteration\": \"23\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, ExtremeRandomTrees\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.88796878, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_24\", \"run_number\": 26, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:08:04.438923Z\", \"end_time\": \"2020-11-23T15:09:12.159258Z\", \"created_time\": \"2020-11-23T15:08:04.332121Z\", \"created_time_dt\": \"2020-11-23T15:08:04.332121Z\", \"duration\": \"0:01:07\", \"iteration\": \"24\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, ExtremeRandomTrees\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.89004986, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_25\", \"run_number\": 27, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:09:15.981228Z\", \"end_time\": \"2020-11-23T15:09:46.069009Z\", \"created_time\": \"2020-11-23T15:09:15.857552Z\", \"created_time_dt\": \"2020-11-23T15:09:15.857552Z\", \"duration\": \"0:00:30\", \"iteration\": \"25\", \"goal\": \"accuracy_max\", \"run_name\": \"TruncatedSVDWrapper, XGBoostClassifier\", \"run_properties\": \"n_components=0.05947368421052632, random_state=None\", \"primary_metric\": 0.88796878, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_26\", \"run_number\": 28, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:09:47.52377Z\", \"end_time\": \"2020-11-23T15:10:22.709128Z\", \"created_time\": \"2020-11-23T15:09:47.403959Z\", \"created_time_dt\": \"2020-11-23T15:09:47.403959Z\", \"duration\": \"0:00:35\", \"iteration\": \"26\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, ExtremeRandomTrees\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.9001951, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_27\", \"run_number\": 29, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:10:24.118893Z\", \"end_time\": \"2020-11-23T15:11:09.242199Z\", \"created_time\": \"2020-11-23T15:10:23.988255Z\", \"created_time_dt\": \"2020-11-23T15:10:23.988255Z\", \"duration\": \"0:00:45\", \"iteration\": \"27\", \"goal\": \"accuracy_max\", \"run_name\": \"SparseNormalizer, XGBoostClassifier\", \"run_properties\": \"<azureml.automl.runtime.shared.model_wrappers.SparseNormalizer object at 0x7f97a615c6a0\", \"primary_metric\": 0.91307175, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_28\", \"run_number\": 30, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:11:10.651452Z\", \"end_time\": \"2020-11-23T15:11:48.911894Z\", \"created_time\": \"2020-11-23T15:11:10.524458Z\", \"created_time_dt\": \"2020-11-23T15:11:10.524458Z\", \"duration\": \"0:00:38\", \"iteration\": \"28\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, ExtremeRandomTrees\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.88796878, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_29\", \"run_number\": 31, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:11:51.369615Z\", \"end_time\": \"2020-11-23T15:12:24.461498Z\", \"created_time\": \"2020-11-23T15:11:50.915709Z\", \"created_time_dt\": \"2020-11-23T15:11:50.915709Z\", \"duration\": \"0:00:33\", \"iteration\": \"29\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, LightGBM\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.90544114, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_30\", \"run_number\": 32, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:12:27.407596Z\", \"end_time\": \"2020-11-23T15:13:02.47808Z\", \"created_time\": \"2020-11-23T15:12:27.084203Z\", \"created_time_dt\": \"2020-11-23T15:12:27.084203Z\", \"duration\": \"0:00:35\", \"iteration\": \"30\", \"goal\": \"accuracy_max\", \"run_name\": \"TruncatedSVDWrapper, XGBoostClassifier\", \"run_properties\": \"n_components=0.5047368421052632, random_state=None\", \"primary_metric\": 0.88796878, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_31\", \"run_number\": 33, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:13:03.819318Z\", \"end_time\": \"2020-11-23T15:13:52.40314Z\", \"created_time\": \"2020-11-23T15:13:03.696232Z\", \"created_time_dt\": \"2020-11-23T15:13:03.696232Z\", \"duration\": \"0:00:48\", \"iteration\": \"31\", \"goal\": \"accuracy_max\", \"run_name\": \"StandardScalerWrapper, ExtremeRandomTrees\", \"run_properties\": \"<azureml.automl.runtime.shared.model_wrappers.StandardScalerWrapper object at 0x7f979f9827f0\", \"primary_metric\": 0.89971819, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_32\", \"run_number\": 34, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:13:53.811944Z\", \"end_time\": \"2020-11-23T15:14:27.818245Z\", \"created_time\": \"2020-11-23T15:13:53.699788Z\", \"created_time_dt\": \"2020-11-23T15:13:53.699788Z\", \"duration\": \"0:00:34\", \"iteration\": \"32\", \"goal\": \"accuracy_max\", \"run_name\": \"SparseNormalizer, LightGBM\", \"run_properties\": \"<azureml.automl.runtime.shared.model_wrappers.SparseNormalizer object at 0x7f979f8d8eb8\", \"primary_metric\": 0.90231953, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_33\", \"run_number\": 35, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:14:29.584449Z\", \"end_time\": \"2020-11-23T15:15:11.89329Z\", \"created_time\": \"2020-11-23T15:14:29.050011Z\", \"created_time_dt\": \"2020-11-23T15:14:29.050011Z\", \"duration\": \"0:00:42\", \"iteration\": \"33\", \"goal\": \"accuracy_max\", \"run_name\": \"StandardScalerWrapper, XGBoostClassifier\", \"run_properties\": \"<azureml.automl.runtime.shared.model_wrappers.StandardScalerWrapper object at 0x7f979fbee390\", \"primary_metric\": 0.9130284, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_34\", \"run_number\": 36, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:15:15.939943Z\", \"end_time\": \"2020-11-23T15:16:32.246896Z\", \"created_time\": \"2020-11-23T15:15:15.512454Z\", \"created_time_dt\": \"2020-11-23T15:15:15.512454Z\", \"duration\": \"0:01:16\", \"iteration\": \"34\", \"goal\": \"accuracy_max\", \"run_name\": \"SparseNormalizer, XGBoostClassifier\", \"run_properties\": \"<azureml.automl.runtime.shared.model_wrappers.SparseNormalizer object at 0x7f978fc1b208\", \"primary_metric\": 0.91272491, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_35\", \"run_number\": 37, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:16:36.059877Z\", \"end_time\": \"2020-11-23T15:17:07.074806Z\", \"created_time\": \"2020-11-23T15:16:35.55897Z\", \"created_time_dt\": \"2020-11-23T15:16:35.55897Z\", \"duration\": \"0:00:31\", \"iteration\": \"35\", \"goal\": \"accuracy_max\", \"run_name\": \"TruncatedSVDWrapper, XGBoostClassifier\", \"run_properties\": \"n_components=0.10894736842105263, random_state=None\", \"primary_metric\": 0.88796878, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_36\", \"run_number\": 38, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:17:08.415759Z\", \"end_time\": \"2020-11-23T15:17:38.76474Z\", \"created_time\": \"2020-11-23T15:17:08.263715Z\", \"created_time_dt\": \"2020-11-23T15:17:08.263715Z\", \"duration\": \"0:00:30\", \"iteration\": \"36\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, LightGBM\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.90986343, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_37\", \"run_number\": 39, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:17:40.20059Z\", \"end_time\": \"2020-11-23T15:18:32.934757Z\", \"created_time\": \"2020-11-23T15:17:40.056529Z\", \"created_time_dt\": \"2020-11-23T15:17:40.056529Z\", \"duration\": \"0:00:52\", \"iteration\": \"37\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, ExtremeRandomTrees\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.90023846, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_38\", \"run_number\": 40, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:18:34.305681Z\", \"end_time\": \"2020-11-23T15:19:02.05171Z\", \"created_time\": \"2020-11-23T15:18:34.157531Z\", \"created_time_dt\": \"2020-11-23T15:18:34.157531Z\", \"duration\": \"0:00:27\", \"iteration\": \"38\", \"goal\": \"accuracy_max\", \"run_name\": \"StandardScalerWrapper, LightGBM\", \"run_properties\": \"<azureml.automl.runtime.shared.model_wrappers.StandardScalerWrapper object at 0x7f979fb92cc0\", \"primary_metric\": 0.90422718, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_39\", \"run_number\": 41, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:19:03.377097Z\", \"end_time\": \"2020-11-23T15:19:35.156946Z\", \"created_time\": \"2020-11-23T15:19:03.251527Z\", \"created_time_dt\": \"2020-11-23T15:19:03.251527Z\", \"duration\": \"0:00:31\", \"iteration\": \"39\", \"goal\": \"accuracy_max\", \"run_name\": \"StandardScalerWrapper, LightGBM\", \"run_properties\": \"<azureml.automl.runtime.shared.model_wrappers.StandardScalerWrapper object at 0x7f978ac4a470\", \"primary_metric\": 0.90630826, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_40\", \"run_number\": 42, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:19:37.163784Z\", \"end_time\": \"2020-11-23T15:20:02.147806Z\", \"created_time\": \"2020-11-23T15:19:37.038132Z\", \"created_time_dt\": \"2020-11-23T15:19:37.038132Z\", \"duration\": \"0:00:25\", \"iteration\": \"40\", \"goal\": \"accuracy_max\", \"run_name\": \"StandardScalerWrapper, XGBoostClassifier\", \"run_properties\": \"<azureml.automl.runtime.shared.model_wrappers.StandardScalerWrapper object at 0x7f97a5569978\", \"primary_metric\": 0.89286798, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_41\", \"run_number\": 43, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:20:03.441807Z\", \"end_time\": \"2020-11-23T15:20:40.592109Z\", \"created_time\": \"2020-11-23T15:20:03.333377Z\", \"created_time_dt\": \"2020-11-23T15:20:03.333377Z\", \"duration\": \"0:00:37\", \"iteration\": \"41\", \"goal\": \"accuracy_max\", \"run_name\": \"SparseNormalizer, XGBoostClassifier\", \"run_properties\": \"<azureml.automl.runtime.shared.model_wrappers.SparseNormalizer object at 0x7f979f9444a8\", \"primary_metric\": 0.91159766, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_42\", \"run_number\": 44, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:20:41.901961Z\", \"end_time\": \"2020-11-23T15:21:09.765921Z\", \"created_time\": \"2020-11-23T15:20:41.777623Z\", \"created_time_dt\": \"2020-11-23T15:20:41.777623Z\", \"duration\": \"0:00:27\", \"iteration\": \"42\", \"goal\": \"accuracy_max\", \"run_name\": \"SparseNormalizer, XGBoostClassifier\", \"run_properties\": \"<azureml.automl.runtime.shared.model_wrappers.SparseNormalizer object at 0x7f979faf02b0\", \"primary_metric\": 0.91224799, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_43\", \"run_number\": 45, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:21:11.142894Z\", \"end_time\": \"2020-11-23T15:22:15.441173Z\", \"created_time\": \"2020-11-23T15:21:11.028948Z\", \"created_time_dt\": \"2020-11-23T15:21:11.028948Z\", \"duration\": \"0:01:04\", \"iteration\": \"43\", \"goal\": \"accuracy_max\", \"run_name\": \"SparseNormalizer, XGBoostClassifier\", \"run_properties\": \"<azureml.automl.runtime.shared.model_wrappers.SparseNormalizer object at 0x7f97a5430f98\", \"primary_metric\": 0.91185779, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_44\", \"run_number\": 46, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:22:16.859915Z\", \"end_time\": \"2020-11-23T15:22:44.244842Z\", \"created_time\": \"2020-11-23T15:22:16.743975Z\", \"created_time_dt\": \"2020-11-23T15:22:16.743975Z\", \"duration\": \"0:00:27\", \"iteration\": \"44\", \"goal\": \"accuracy_max\", \"run_name\": \"MaxAbsScaler, LightGBM\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.90665511, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_45\", \"run_number\": 47, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:22:47.057699Z\", \"end_time\": \"2020-11-23T15:23:20.704375Z\", \"created_time\": \"2020-11-23T15:22:46.739986Z\", \"created_time_dt\": \"2020-11-23T15:22:46.739986Z\", \"duration\": \"0:00:33\", \"iteration\": \"45\", \"goal\": \"accuracy_max\", \"run_name\": \"StandardScalerWrapper, XGBoostClassifier\", \"run_properties\": \"<azureml.automl.runtime.shared.model_wrappers.StandardScalerWrapper object at 0x7f97a54d1518\", \"primary_metric\": 0.88796878, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_46\", \"run_number\": 48, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:23:22.602574Z\", \"end_time\": \"2020-11-23T15:24:03.071432Z\", \"created_time\": \"2020-11-23T15:23:22.286037Z\", \"created_time_dt\": \"2020-11-23T15:23:22.286037Z\", \"duration\": \"0:00:40\", \"iteration\": \"46\", \"goal\": \"accuracy_max\", \"run_name\": \"SparseNormalizer, XGBoostClassifier\", \"run_properties\": \"<azureml.automl.runtime.shared.model_wrappers.SparseNormalizer object at 0x7f97a538d0b8\", \"primary_metric\": 0.91298504, \"best_metric\": 0.91463256}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_47\", \"run_number\": 49, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:24:03.398206Z\", \"end_time\": \"2020-11-23T15:25:15.061231Z\", \"created_time\": \"2020-11-23T15:24:03.287671Z\", \"created_time_dt\": \"2020-11-23T15:24:03.287671Z\", \"duration\": \"0:01:11\", \"iteration\": \"47\", \"goal\": \"accuracy_max\", \"run_name\": \"VotingEnsemble\", \"run_properties\": \"classification_labels=None,\\n                              estimators=[('1',\\n                                           Pipeline(memory=None,\\n                                                    steps=[('maxabsscaler',\\n                                                            MaxAbsScaler(copy=True\", \"primary_metric\": 0.91740733, \"best_metric\": 0.91740733}, {\"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9_48\", \"run_number\": 50, \"metric\": null, \"status\": \"Completed\", \"run_type\": null, \"training_percent\": \"100\", \"start_time\": \"2020-11-23T15:25:15.897821Z\", \"end_time\": \"2020-11-23T15:26:39.927973Z\", \"created_time\": \"2020-11-23T15:25:15.707792Z\", \"created_time_dt\": \"2020-11-23T15:25:15.707792Z\", \"duration\": \"0:01:24\", \"iteration\": \"48\", \"goal\": \"accuracy_max\", \"run_name\": \"StackEnsemble\", \"run_properties\": \"base_learners=[('1',\\n                                        Pipeline(memory=None,\\n                                                 steps=[('maxabsscaler',\\n                                                         MaxAbsScaler(copy=True\", \"primary_metric\": 0.9138088, \"best_metric\": 0.91740733}], \"children_metrics\": {\"categories\": [0], \"series\": {\"weighted_accuracy\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"weighted_accuracy\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9552359731579247, 0.9597511583713523, 0.9825089510077051, 0.984324629486036, 0.8773741911432944, 0.9651609031033344, 0.97705084029714, 0.9785626807635568, 0.9783153012276964, 0.7672138136074275, 0.9742231808214292, 0.9744443910607388, 0.9839962119008595, 0.984324629486036, 0.8041820529232622, 0.8637967986302131, 0.984324629486036, 0.9789840634133462, 0.7251279217499099, 0.9669146561979464, 0.7609996997378256, 0.7929702457537615, 0.9612869352551942, 0.984324629486036, 0.9839397162623496, 0.984324629486036, 0.9776799945754743, 0.9549155709257017, 0.984324629486036, 0.9674942812621691, 0.984324629486036, 0.9762281782647658, 0.9764661103555732, 0.9645687710590762, 0.954657302152102, 0.984324629486036, 0.9551784181685591, 0.9779395338343484, 0.9689657083000969, 0.9597639836765917, 0.9839605951065182, 0.9669304257022293, 0.9555557413748733, 0.9535303267231038, 0.9596039306829216, 0.984324629486036, 0.9554496995194258, 0.9560281405495372, 0.9569934894788601]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"weighted_accuracy_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9552359731579247, 0.9597511583713523, 0.9825089510077051, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036, 0.984324629486036]}], \"log_loss\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"log_loss\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.17671757509192162, 0.1784088720061951, 0.23862728319414733, 0.26297151758822446, 0.33502587733194955, 0.2208939465954432, 0.22850830591085197, 0.24914320853291616, 0.2537437390156425, 0.5257471452203797, 0.24387249252616688, 0.24071072619338435, 0.26313945109921416, 0.2571043180400028, 0.5927014902830525, 0.8198394627973901, 0.27924016572008475, 0.23903663705004616, 0.513326445557406, 0.3124240444309948, 0.5729372288140455, 0.5061765887820953, 0.18595177769837293, 0.25168599639950207, 0.2740578157617983, 0.41035752563302585, 0.23513990437742094, 0.20429872444406688, 0.2907100970590228, 0.204383309197748, 0.3507520406533812, 0.22723286467393136, 0.21666009399566746, 0.20874327478542773, 0.18245301254506638, 0.35075378652340083, 0.18348681869046982, 0.23170147485468418, 0.20396197158914786, 0.1911698855972346, 0.3619051440422057, 0.1863980799083853, 0.3186492747468012, 0.18593452460274093, 0.19330141843913543, 0.3507526603354876, 0.20491888377665504, 0.19807490901665323, 0.19076004572116412]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"log_loss_min\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162, 0.17671757509192162]}], \"recall_score_macro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"recall_score_macro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.7487416205914255, 0.7329903856224893, 0.5283884158579879, 0.5, 0.8418116928552435, 0.6677294151617159, 0.5875383505710482, 0.587651440566883, 0.5858682969603205, 0.7337131163937511, 0.6171073846825752, 0.6193436480701361, 0.5072676070941237, 0.5, 0.7549601053868316, 0.860904277378179, 0.5, 0.564068970590051, 0.7342896555146216, 0.6644125819326914, 0.7320341367311299, 0.7764482575843101, 0.716919645521084, 0.5, 0.5121256881652927, 0.5, 0.5881838170712937, 0.7445636517681156, 0.5, 0.655672340204044, 0.5, 0.5916304100625689, 0.6037690265563833, 0.7054697777118799, 0.7438364023697923, 0.5, 0.7274845797843545, 0.5872736843928131, 0.643518829300537, 0.6910627491570928, 0.5260916477509336, 0.6887891208205587, 0.7378717434836586, 0.7440544883171755, 0.6934687467900875, 0.5, 0.7419528760104626, 0.7618471411716752, 0.7399583353105958]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"recall_score_macro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.7487416205914255, 0.7487416205914255, 0.7487416205914255, 0.7487416205914255, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179]}], \"AUC_weighted\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"AUC_weighted\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9463784776106465, 0.9455889379724427, 0.9069555622409655, 0.8923917564814072, 0.9289399453250102, 0.9303576632861063, 0.9119849408531167, 0.8940005077269187, 0.8875096957920862, 0.8290714487974851, 0.9237890853903299, 0.924214707495994, 0.9011783240442817, 0.8797667800945648, 0.8428554094724696, 0.9321485793375233, 0.8931643574686399, 0.9028972510266591, 0.8578390736931374, 0.9252946902579969, 0.8083888459265272, 0.8776880346609296, 0.9407081912076857, 0.8954327728826922, 0.8496793640059057, 0.5, 0.9091799497370803, 0.9431706036793928, 0.8369906317258735, 0.9323991422191854, 0.5, 0.9164029151470838, 0.9259682014450098, 0.9435188862412955, 0.9431259155564484, 0.5, 0.9412852744634117, 0.9161867856295306, 0.9312765757322172, 0.9366227572766134, 0.8794266535321636, 0.9424784620809463, 0.9375427516202974, 0.9414189673628217, 0.9340932554166631, 0.5, 0.9423266245514641, 0.9463987033748344, 0.9446242455967647]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"AUC_weighted_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463987033748344, 0.9463987033748344]}], \"recall_score_micro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"recall_score_micro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9141556470843268, 0.9146325601560807, 0.8921309343160633, 0.887968783871667, 0.8703663559505745, 0.906091480598309, 0.89958812052894, 0.9008020810752221, 0.9002384565358769, 0.7605462822458271, 0.9031432907001952, 0.903793626707132, 0.8891827444179492, 0.887968783871667, 0.7943637546065467, 0.8632126598742683, 0.887968783871667, 0.8963797962280513, 0.7269889442878822, 0.9066984608714502, 0.7552568827227402, 0.7897680468241924, 0.9126815521352698, 0.887968783871667, 0.8900498590938651, 0.887968783871667, 0.9001951008020811, 0.9130717537394322, 0.887968783871667, 0.9054411445913721, 0.887968783871667, 0.8997181877303273, 0.902319531758075, 0.9130283980056362, 0.9127249078690657, 0.887968783871667, 0.9098634294385433, 0.9002384565358769, 0.90422718404509, 0.9063082592672881, 0.8928679817905918, 0.911597658790375, 0.9122479947973119, 0.9118577931931497, 0.9066551051376545, 0.887968783871667, 0.9129850422718404, 0.9174073271190115, 0.9138088012139605]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"recall_score_micro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9141556470843268, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9174073271190115, 0.9174073271190115]}], \"f1_score_micro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"f1_score_micro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9141556470843268, 0.9146325601560807, 0.8921309343160633, 0.887968783871667, 0.8703663559505745, 0.906091480598309, 0.89958812052894, 0.9008020810752221, 0.9002384565358769, 0.7605462822458271, 0.9031432907001952, 0.903793626707132, 0.8891827444179492, 0.887968783871667, 0.7943637546065467, 0.8632126598742683, 0.887968783871667, 0.8963797962280513, 0.7269889442878821, 0.9066984608714502, 0.7552568827227402, 0.7897680468241924, 0.9126815521352698, 0.887968783871667, 0.8900498590938651, 0.887968783871667, 0.9001951008020811, 0.9130717537394322, 0.887968783871667, 0.9054411445913721, 0.887968783871667, 0.8997181877303273, 0.902319531758075, 0.9130283980056362, 0.9127249078690658, 0.887968783871667, 0.9098634294385433, 0.9002384565358769, 0.90422718404509, 0.9063082592672881, 0.8928679817905918, 0.911597658790375, 0.9122479947973119, 0.91185779319315, 0.9066551051376545, 0.887968783871667, 0.9129850422718404, 0.9174073271190115, 0.9138088012139605]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"f1_score_micro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9141556470843268, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9174073271190115, 0.9174073271190115]}], \"average_precision_score_micro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"average_precision_score_micro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9811539776866389, 0.9810910763844991, 0.969951043857671, 0.966742104304589, 0.9158502426234987, 0.9701427868342172, 0.9707852940986038, 0.9680861726436175, 0.9667513223971934, 0.794549148353682, 0.9736165592479635, 0.973718159302377, 0.9684271789574275, 0.9635797004501606, 0.8078495031019607, 0.9108998887896009, 0.9667454493629013, 0.9697273445534966, 0.8537059342133471, 0.9720022933291748, 0.8154542257852692, 0.8610153249846324, 0.9796972833042916, 0.968103420149031, 0.9578557234087265, 0.8445080301388611, 0.9715385486478507, 0.9803034200815401, 0.9541993203282543, 0.9764953370279544, 0.8445080301388611, 0.9730196713052249, 0.9749267710767044, 0.980271181845582, 0.9801463893070054, 0.8445080301388611, 0.9795741269731894, 0.9723693471946058, 0.9763678993199554, 0.9780991527630937, 0.9587473950934184, 0.9798949928918657, 0.9782220637210285, 0.9796714256766512, 0.9775154871272651, 0.8445080301388611, 0.9799578835921681, 0.9812870923189342, 0.9807445543685039]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"average_precision_score_micro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9811539776866389, 0.9812870923189342, 0.9812870923189342]}], \"AUC_macro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"AUC_macro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9463784776106465, 0.9455889379724429, 0.9069555622409657, 0.8923917564814072, 0.9289399453250102, 0.9303576632861063, 0.9119849408531167, 0.8940005077269187, 0.8875096957920864, 0.8290714487974851, 0.9237890853903299, 0.9242147074959941, 0.9011783240442817, 0.8797667800945648, 0.8428554094724696, 0.9321485793375233, 0.8931643574686399, 0.9028972510266591, 0.8578390736931374, 0.9252946902579969, 0.8083888459265272, 0.8776880346609296, 0.9407081912076857, 0.8954327728826922, 0.8496793640059055, 0.5, 0.9091799497370803, 0.9431706036793928, 0.8369906317258735, 0.9323991422191854, 0.5, 0.9164029151470838, 0.9259682014450096, 0.9435188862412955, 0.9431259155564484, 0.5, 0.9412852744634119, 0.9161867856295306, 0.9312765757322172, 0.9366227572766134, 0.8794266535321636, 0.9424784620809465, 0.9375427516202974, 0.9414189673628217, 0.9340932554166631, 0.5, 0.9423266245514643, 0.9463987033748342, 0.9446242455967647]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"AUC_macro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463784776106465, 0.9463987033748342, 0.9463987033748342]}], \"precision_score_weighted\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"precision_score_weighted\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9086728239116434, 0.9070457764686666, 0.8821075787124167, 0.7884924220746945, 0.9159101798015999, 0.8935091607706079, 0.8824796467311679, 0.8858050393397994, 0.884508750328069, 0.87728967189483, 0.8871759609852132, 0.8882685595298657, 0.8074562320041505, 0.7884924220746945, 0.884464550491832, 0.9199344482163232, 0.7884924220746945, 0.878596731352526, 0.8787082775862617, 0.8931692383418642, 0.8768287209176691, 0.8910461588546303, 0.9036034834006775, 0.7884924220746945, 0.8827048037032869, 0.7884924220746945, 0.883762665042676, 0.9072526169911462, 0.7884924220746945, 0.8909131363857652, 0.7884924220746945, 0.8822633522845515, 0.886711633198388, 0.9028853907935283, 0.9069045037625537, 0.7884924220746945, 0.9025215418113792, 0.8841879891607165, 0.8885836134699066, 0.8947914996432031, 0.8692965781871136, 0.9002361562833524, 0.9057261795932042, 0.9063802064941962, 0.8954243245218425, 0.7884924220746945, 0.9068579515436669, 0.9128267541883142, 0.9073117236273109]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"precision_score_weighted_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9086728239116434, 0.9086728239116434, 0.9086728239116434, 0.9086728239116434, 0.9159101798015999, 0.9159101798015999, 0.9159101798015999, 0.9159101798015999, 0.9159101798015999, 0.9159101798015999, 0.9159101798015999, 0.9159101798015999, 0.9159101798015999, 0.9159101798015999, 0.9159101798015999, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232, 0.9199344482163232]}], \"accuracy\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"accuracy\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9141556470843268, 0.9146325601560807, 0.8921309343160633, 0.887968783871667, 0.8703663559505745, 0.906091480598309, 0.89958812052894, 0.9008020810752221, 0.9002384565358769, 0.7605462822458271, 0.9031432907001952, 0.903793626707132, 0.8891827444179492, 0.887968783871667, 0.7943637546065467, 0.8632126598742683, 0.887968783871667, 0.8963797962280513, 0.7269889442878822, 0.9066984608714502, 0.7552568827227402, 0.7897680468241924, 0.9126815521352698, 0.887968783871667, 0.8900498590938651, 0.887968783871667, 0.9001951008020811, 0.9130717537394322, 0.887968783871667, 0.9054411445913721, 0.887968783871667, 0.8997181877303273, 0.902319531758075, 0.9130283980056362, 0.9127249078690657, 0.887968783871667, 0.9098634294385433, 0.9002384565358769, 0.90422718404509, 0.9063082592672881, 0.8928679817905918, 0.911597658790375, 0.9122479947973119, 0.9118577931931497, 0.9066551051376545, 0.887968783871667, 0.9129850422718404, 0.9174073271190115, 0.9138088012139605]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"accuracy_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9141556470843268, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9174073271190115, 0.9174073271190115]}], \"precision_score_macro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"precision_score_macro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.7910997858673995, 0.797587650054594, 0.841246444855142, 0.4439843919358335, 0.7175404934259426, 0.7881561911006005, 0.8012699293025477, 0.8161241152207535, 0.8115140316730466, 0.615888736355298, 0.8003053588404587, 0.8034887966277466, 0.5216537274068982, 0.4439843919358335, 0.6377103234462502, 0.7127102785570584, 0.4439843919358335, 0.8008234754622612, 0.6057936899427198, 0.7903710552395651, 0.6130982403802626, 0.6402766907617407, 0.7952318586024805, 0.4439843919358335, 0.8554946376074962, 0.4439843919358335, 0.8064583586594013, 0.7881165435162293, 0.4439843919358335, 0.7871825592217844, 0.4439843919358335, 0.7970834640948954, 0.8080659610930822, 0.8016693055588375, 0.7870814661768794, 0.4439843919358335, 0.7809034121659196, 0.8090888512159033, 0.7860996497982943, 0.7760632358131723, 0.7859037921263621, 0.8032241917434682, 0.7868649394145738, 0.7842544147114554, 0.776917765128178, 0.4439843919358335, 0.7885735972900412, 0.7990786988388037, 0.7924976857839775]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"precision_score_macro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.7910997858673995, 0.797587650054594, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.841246444855142, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962, 0.8554946376074962]}], \"f1_score_weighted\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"f1_score_weighted\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.910791818485906, 0.9094298451951757, 0.8489712285881039, 0.8352782642361705, 0.8849876144867836, 0.8932058381309049, 0.8727141043915525, 0.8734802662153512, 0.8726558609681951, 0.7997121164766563, 0.8821834803845864, 0.8831954386293477, 0.8389449998176598, 0.8352782642361705, 0.8250182354619862, 0.8808040601355287, 0.8352782642361705, 0.8634420584933838, 0.7750255371736495, 0.8938307409360846, 0.7957728374963846, 0.8227997301600098, 0.9060143962367537, 0.8352782642361705, 0.8414279437026206, 0.8352782642361705, 0.8732614741887653, 0.9095059877403866, 0.8352782642361705, 0.8915417800298748, 0.8352782642361705, 0.873872245828521, 0.878616620138064, 0.904813456478759, 0.9091510589232046, 0.8352782642361705, 0.905134307112748, 0.8730130894448628, 0.8885036103036323, 0.897780285102385, 0.8484725734095917, 0.9013914016140184, 0.9081366946771505, 0.9085211409371803, 0.898385586870344, 0.8352782642361705, 0.9091610605435643, 0.9146564968169095, 0.9095916558416057]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"f1_score_weighted_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.910791818485906, 0.9146564968169095, 0.9146564968169095]}], \"average_precision_score_macro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"average_precision_score_macro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.8212248114842253, 0.8240969517339398, 0.7522316516556835, 0.7255889653624705, 0.7835707338822605, 0.7814973426525932, 0.749711355777011, 0.7508196065869456, 0.7436866893478198, 0.6973635323226162, 0.7822746698221563, 0.782360979968627, 0.7408836740239717, 0.7044342999441197, 0.6932227616928672, 0.7806189134213322, 0.7363797145277524, 0.7403167889036507, 0.7044741165920071, 0.7857474474505073, 0.7003973981428133, 0.7227620862354645, 0.8157593369138775, 0.7304616972037631, 0.7232184710555306, 0.5, 0.7624819389096169, 0.8155553970080153, 0.7133152868792008, 0.787157010620572, 0.5, 0.7686408320040852, 0.7808796678173417, 0.8217293927558224, 0.8110468578555684, 0.5, 0.8019214024904844, 0.7580040739036406, 0.7860639315500558, 0.7921931615693658, 0.7233526585371626, 0.8123263533626606, 0.8130553379221255, 0.8086844075082246, 0.7871598559725783, 0.5, 0.813899672492193, 0.8253415965296659, 0.8212792537535043]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"average_precision_score_macro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.8212248114842253, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8240969517339398, 0.8253415965296659, 0.8253415965296659]}], \"average_precision_score_weighted\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"average_precision_score_weighted\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9546017424957325, 0.9551260850581442, 0.9343532792494527, 0.9263826370846406, 0.9440936888264545, 0.943828515651145, 0.9343509456263714, 0.932776222770103, 0.9304011651606034, 0.9103962839088681, 0.9427903454504725, 0.9429463868983753, 0.9307979783232663, 0.9200709224397439, 0.9117945815895337, 0.9436221638322191, 0.9287787712519997, 0.9316631947920909, 0.9180349283015408, 0.9430241011823393, 0.9054187515969673, 0.9244140323141448, 0.9526791480170281, 0.9286907712333943, 0.9201178750530492, 0.8010472764060548, 0.9373479656206622, 0.9528764135744183, 0.9152478984435442, 0.9452555046165884, 0.8010472764060548, 0.9395957057237914, 0.943110293767554, 0.9541827337672579, 0.9519315200864185, 0.8010472764060548, 0.9497335077769838, 0.9371007478024493, 0.9449353420644186, 0.9470194843702424, 0.9211316586664552, 0.9521355394914564, 0.9509356249423735, 0.9512194488037906, 0.9456144452824471, 0.8010472764060548, 0.9522955204852608, 0.9554410206467688, 0.9542930723597269]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"average_precision_score_weighted_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9546017424957325, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9551260850581442, 0.9554410206467688, 0.9554410206467688]}], \"recall_score_weighted\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"recall_score_weighted\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9141556470843268, 0.9146325601560807, 0.8921309343160633, 0.887968783871667, 0.8703663559505745, 0.906091480598309, 0.89958812052894, 0.9008020810752221, 0.9002384565358769, 0.7605462822458271, 0.9031432907001952, 0.903793626707132, 0.8891827444179492, 0.887968783871667, 0.7943637546065467, 0.8632126598742683, 0.887968783871667, 0.8963797962280513, 0.7269889442878822, 0.9066984608714502, 0.7552568827227402, 0.7897680468241924, 0.9126815521352698, 0.887968783871667, 0.8900498590938651, 0.887968783871667, 0.9001951008020811, 0.9130717537394322, 0.887968783871667, 0.9054411445913721, 0.887968783871667, 0.8997181877303273, 0.902319531758075, 0.9130283980056362, 0.9127249078690657, 0.887968783871667, 0.9098634294385433, 0.9002384565358769, 0.90422718404509, 0.9063082592672881, 0.8928679817905918, 0.911597658790375, 0.9122479947973119, 0.9118577931931497, 0.9066551051376545, 0.887968783871667, 0.9129850422718404, 0.9174073271190115, 0.9138088012139605]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"recall_score_weighted_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9141556470843268, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9174073271190115, 0.9174073271190115]}], \"matthews_correlation\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"matthews_correlation\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.5381263488490677, 0.526591325437469, 0.1788274042154197, 0.0, 0.5443602165911705, 0.4350830055933642, 0.32399181458437054, 0.3325957324027877, 0.3268870711210748, 0.32901726509672374, 0.37454156532431543, 0.3804074595484963, 0.04386927232041749, 0.0, 0.37423810689310055, 0.5538448682456516, 0.0, 0.26956115937419156, 0.31485900264434485, 0.4362334188615695, 0.323876503686899, 0.3937921127405609, 0.5060317030430397, 0.0, 0.1174836962540863, 0.0, 0.3285096673972966, 0.5308755495098276, 0.0, 0.4226448869544638, 0.0, 0.3292213215630979, 0.35745907633382035, 0.4979166728393857, 0.5290886961150412, 0.0, 0.5054670735894551, 0.32781250925606675, 0.40518818295882264, 0.459217494056975, 0.17555107485757562, 0.47844753803646023, 0.5223869985379224, 0.5266693715225292, 0.4628287712578657, 0.0, 0.5284191699698617, 0.5596533409857434, 0.529754257572449]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"matthews_correlation_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.5381263488490677, 0.5381263488490677, 0.5381263488490677, 0.5381263488490677, 0.5443602165911705, 0.5443602165911705, 0.5443602165911705, 0.5443602165911705, 0.5443602165911705, 0.5443602165911705, 0.5443602165911705, 0.5443602165911705, 0.5443602165911705, 0.5443602165911705, 0.5443602165911705, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5538448682456516, 0.5596533409857434, 0.5596533409857434]}], \"AUC_micro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"AUC_micro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9802960737026798, 0.9802601052671202, 0.9690463745165655, 0.9659954955338096, 0.9324408079805753, 0.9731100652455391, 0.9703198939800523, 0.9668941988984937, 0.965552614786373, 0.8275486073180586, 0.9733942788570683, 0.9734831050092696, 0.9678035696722084, 0.9635250927770878, 0.8454018650296439, 0.9247686382309883, 0.966220591962253, 0.9684286422470094, 0.8496425125667482, 0.9732184968737208, 0.8391054526954687, 0.8717544725227047, 0.978843313571684, 0.9666522789791561, 0.9575951975794474, 0.8879687838716672, 0.9702620738035268, 0.9795106328456853, 0.9550537226226468, 0.9756226113168802, 0.8879687838716672, 0.9717778421372902, 0.9739823679476972, 0.9796004364521063, 0.9792717768693823, 0.8879687838716672, 0.9786457832319513, 0.9711564632116072, 0.9754086992203723, 0.9771173455877271, 0.9637297660515017, 0.9790219715261006, 0.978310864182783, 0.9787701360855937, 0.9765172826829088, 0.8879687838716672, 0.9792871435775454, 0.9805214332918746, 0.979994350502583]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"AUC_micro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9802960737026798, 0.9805214332918746, 0.9805214332918746]}], \"precision_score_micro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"precision_score_micro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9141556470843268, 0.9146325601560807, 0.8921309343160633, 0.887968783871667, 0.8703663559505745, 0.906091480598309, 0.89958812052894, 0.9008020810752221, 0.9002384565358769, 0.7605462822458271, 0.9031432907001952, 0.903793626707132, 0.8891827444179492, 0.887968783871667, 0.7943637546065467, 0.8632126598742683, 0.887968783871667, 0.8963797962280513, 0.7269889442878822, 0.9066984608714502, 0.7552568827227402, 0.7897680468241924, 0.9126815521352698, 0.887968783871667, 0.8900498590938651, 0.887968783871667, 0.9001951008020811, 0.9130717537394322, 0.887968783871667, 0.9054411445913721, 0.887968783871667, 0.8997181877303273, 0.902319531758075, 0.9130283980056362, 0.9127249078690657, 0.887968783871667, 0.9098634294385433, 0.9002384565358769, 0.90422718404509, 0.9063082592672881, 0.8928679817905918, 0.911597658790375, 0.9122479947973119, 0.9118577931931497, 0.9066551051376545, 0.887968783871667, 0.9129850422718404, 0.9174073271190115, 0.9138088012139605]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"precision_score_micro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9141556470843268, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9146325601560807, 0.9174073271190115, 0.9174073271190115]}], \"f1_score_macro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"f1_score_macro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.7674986508001772, 0.7597517716947536, 0.5250511029902414, 0.47032965175358177, 0.7531955776787806, 0.7011549266902347, 0.618890437150141, 0.6200021482617284, 0.6174060616514913, 0.6242607627351464, 0.6558891439182613, 0.6590657367472614, 0.48438631474996396, 0.47032965175358177, 0.655745963246466, 0.7517923770912676, 0.47032965175358177, 0.5832666626238567, 0.6018433560631734, 0.7029948378305617, 0.6196979263201831, 0.6586395873950663, 0.7476618472354983, 0.47032965175358177, 0.4945584950103292, 0.47032965175358177, 0.6202832323745221, 0.7637856642306285, 0.47032965175358177, 0.6948490116642748, 0.47032965175358177, 0.624132749188829, 0.6407475549822998, 0.7410681903387152, 0.7628721719835052, 0.47032965175358177, 0.7500775188023077, 0.6189525506952092, 0.6830117181627933, 0.7226463217627149, 0.5209535080804967, 0.7279105859410789, 0.7590540887990892, 0.7618326353592746, 0.724749716385709, 0.47032965175358177, 0.7622931499886079, 0.7786675437712551, 0.7624464939229931]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"f1_score_macro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7674986508001772, 0.7786675437712551, 0.7786675437712551]}], \"balanced_accuracy\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"balanced_accuracy\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.7487416205914255, 0.7329903856224893, 0.5283884158579879, 0.5, 0.8418116928552435, 0.6677294151617159, 0.5875383505710482, 0.587651440566883, 0.5858682969603205, 0.7337131163937511, 0.6171073846825752, 0.6193436480701361, 0.5072676070941237, 0.5, 0.7549601053868316, 0.860904277378179, 0.5, 0.564068970590051, 0.7342896555146216, 0.6644125819326914, 0.7320341367311299, 0.7764482575843101, 0.716919645521084, 0.5, 0.5121256881652927, 0.5, 0.5881838170712937, 0.7445636517681156, 0.5, 0.655672340204044, 0.5, 0.5916304100625689, 0.6037690265563833, 0.7054697777118799, 0.7438364023697923, 0.5, 0.7274845797843545, 0.5872736843928131, 0.643518829300537, 0.6910627491570928, 0.5260916477509336, 0.6887891208205587, 0.7378717434836586, 0.7440544883171755, 0.6934687467900875, 0.5, 0.7419528760104626, 0.7618471411716752, 0.7399583353105958]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"balanced_accuracy_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.7487416205914255, 0.7487416205914255, 0.7487416205914255, 0.7487416205914255, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.8418116928552435, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179, 0.860904277378179]}], \"norm_macro_recall\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"markers\", \"name\": \"norm_macro_recall\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.4974832411828508, 0.4659807712449785, 0.056776831715975674, 0.0, 0.6836233857104871, 0.3354588303234317, 0.1750767011420963, 0.1753028811337661, 0.17173659392064114, 0.4674262327875023, 0.2342147693651504, 0.23868729614027212, 0.014535214188247192, 0.0, 0.509920210773663, 0.7218085547563577, 0.0, 0.12813794118010202, 0.468579311029243, 0.32882516386538263, 0.4640682734622598, 0.5528965151686202, 0.433839291042168, 0.0, 0.02425137633058534, 0.0, 0.17636763414258758, 0.48912730353623096, 0.0, 0.31134468040808805, 0.0, 0.1832608201251377, 0.20753805311276635, 0.41093955542375965, 0.4876728047395849, 0.0, 0.45496915956870937, 0.17454736878562632, 0.287037658601074, 0.3821254983141854, 0.05218329550186725, 0.3775782416411174, 0.47574348696731716, 0.4881089766343509, 0.386937493580175, 0.0, 0.48390575202092523, 0.5236942823433501, 0.4799166706211917]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\", \"14\", \"15\", \"16\", \"17\", \"18\", \"19\", \"20\", \"21\", \"22\", \"23\", \"24\", \"25\", \"26\", \"27\", \"28\", \"29\", \"30\", \"31\", \"32\", \"33\", \"34\", \"35\", \"36\", \"37\", \"38\", \"39\", \"40\", \"41\", \"42\", \"43\", \"44\", \"45\", \"46\", \"47\", \"48\"], \"mode\": \"lines\", \"name\": \"norm_macro_recall_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.4974832411828508, 0.4974832411828508, 0.4974832411828508, 0.4974832411828508, 0.6836233857104871, 0.6836233857104871, 0.6836233857104871, 0.6836233857104871, 0.6836233857104871, 0.6836233857104871, 0.6836233857104871, 0.6836233857104871, 0.6836233857104871, 0.6836233857104871, 0.6836233857104871, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577, 0.7218085547563577]}]}, \"metricName\": null, \"primaryMetricName\": \"accuracy\", \"showLegend\": false}, \"run_metrics\": [{\"name\": \"experiment_status\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], \"series\": [{\"data\": [\"DatasetEvaluation\", \"FeaturesGeneration\", \"DatasetFeaturization\", \"DatasetFeaturizationCompleted\", \"DatasetBalancing\", \"DatasetCrossValidationSplit\", \"ModelSelection\", \"BestRunExplainModel\", \"ModelExplanationDataSetSetup\", \"PickSurrogateModel\", \"EngineeredFeatureExplanations\", \"EngineeredFeatureExplanations\", \"RawFeaturesExplanations\", \"RawFeaturesExplanations\", \"BestRunExplainModel\"]}]}, {\"name\": \"experiment_status_description\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], \"series\": [{\"data\": [\"Gathering dataset statistics.\", \"Generating features for the dataset.\", \"Beginning to fit featurizers and featurize the dataset.\", \"Completed fit featurizers and featurizing the dataset.\", \"Performing class balancing sweeping\", \"Generating individually featurized CV splits.\", \"Beginning model selection.\", \"Best run model explanations started\", \"Model explanations data setup completed\", \"Choosing LightGBM as the surrogate model for explanations\", \"Computation of engineered features started\", \"Computation of engineered features completed\", \"Computation of raw features started\", \"Computation of raw features completed\", \"Best run model explanations completed\"]}]}, {\"name\": \"weighted_accuracy\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.9560281405495372]}]}, {\"name\": \"log_loss\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.19807490901665323]}]}, {\"name\": \"recall_score_macro\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.7618471411716752]}]}, {\"name\": \"AUC_weighted\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.9463987033748344]}]}, {\"name\": \"recall_score_micro\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.9174073271190115]}]}, {\"name\": \"f1_score_micro\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.9174073271190115]}]}, {\"name\": \"average_precision_score_micro\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.9812870923189342]}]}, {\"name\": \"AUC_macro\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.9463987033748342]}]}, {\"name\": \"precision_score_weighted\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.9128267541883142]}]}, {\"name\": \"accuracy\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.9174073271190115]}]}, {\"name\": \"precision_score_macro\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.7990786988388037]}]}, {\"name\": \"f1_score_weighted\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.9146564968169095]}]}, {\"name\": \"average_precision_score_macro\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.8253415965296659]}]}, {\"name\": \"average_precision_score_weighted\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.9554410206467688]}]}, {\"name\": \"recall_score_weighted\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.9174073271190115]}]}, {\"name\": \"matthews_correlation\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.5596533409857434]}]}, {\"name\": \"AUC_micro\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.9805214332918746]}]}, {\"name\": \"precision_score_micro\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.9174073271190115]}]}, {\"name\": \"f1_score_macro\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.7786675437712551]}]}, {\"name\": \"balanced_accuracy\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.7618471411716752]}]}, {\"name\": \"norm_macro_recall\", \"run_id\": \"AutoML_770c77d2-9751-454d-8d86-98b0c225ada9\", \"categories\": [0], \"series\": [{\"data\": [0.5236942823433501]}]}], \"run_logs\": \"\\nRun is completed.\", \"graph\": {}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"INFO\", \"sdk_version\": \"1.18.0\"}, \"loading\": false}"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "****************************************************************************************************\n",
      "DATA GUARDRAILS: \n",
      "\n",
      "TYPE:         Class balancing detection\n",
      "STATUS:       ALERTED\n",
      "DESCRIPTION:  To decrease model bias, please cancel the current run and fix balancing problem.\n",
      "              Learn more about imbalanced data: https://aka.ms/AutomatedMLImbalancedData\n",
      "DETAILS:      Imbalanced data can lead to a falsely perceived positive effect of a model's accuracy because the input data has bias towards one class.\n",
      "+---------------------------------+---------------------------------+--------------------------------------+\n",
      "|Size of the smallest class       |Name/Label of the smallest class |Number of samples in the training data|\n",
      "+=================================+=================================+======================================+\n",
      "|2584                             |1                                |23065                                 |\n",
      "+---------------------------------+---------------------------------+--------------------------------------+\n",
      "\n",
      "****************************************************************************************************\n",
      "\n",
      "TYPE:         Missing feature values imputation\n",
      "STATUS:       PASSED\n",
      "DESCRIPTION:  No feature missing values were detected in the training data.\n",
      "              Learn more about missing value imputation: https://aka.ms/AutomatedMLFeaturization\n",
      "\n",
      "****************************************************************************************************\n",
      "\n",
      "TYPE:         High cardinality feature detection\n",
      "STATUS:       PASSED\n",
      "DESCRIPTION:  Your inputs were analyzed, and no high cardinality features were detected.\n",
      "              Learn more about high cardinality feature handling: https://aka.ms/AutomatedMLFeaturization\n",
      "\n",
      "****************************************************************************************************\n",
      "\n",
      "****************************************************************************************************\n",
      "ITERATION: The iteration being evaluated.\n",
      "PIPELINE: A summary description of the pipeline being evaluated.\n",
      "DURATION: Time taken for the current iteration.\n",
      "METRIC: The result of computing score on the fitted pipeline.\n",
      "BEST: The best observed score thus far.\n",
      "****************************************************************************************************\n",
      "\n",
      " ITERATION   PIPELINE                                       DURATION      METRIC      BEST\n",
      "         0   MaxAbsScaler LightGBM                          0:00:40       0.9142    0.9142\n",
      "         1   MaxAbsScaler XGBoostClassifier                 0:00:38       0.9146    0.9146\n",
      "         2   MaxAbsScaler RandomForest                      0:00:31       0.8921    0.9146\n",
      "         3   MaxAbsScaler RandomForest                      0:00:32       0.8880    0.9146\n",
      "         4   MaxAbsScaler SGD                               0:00:26       0.8704    0.9146\n",
      "         5   MaxAbsScaler SGD                               0:00:25       0.9061    0.9146\n",
      "         6   MaxAbsScaler ExtremeRandomTrees                0:00:26       0.8996    0.9146\n",
      "         7   MaxAbsScaler ExtremeRandomTrees                0:00:41       0.9008    0.9146\n",
      "         8   MaxAbsScaler ExtremeRandomTrees                0:00:30       0.9002    0.9146\n",
      "         9   MaxAbsScaler ExtremeRandomTrees                0:00:32       0.7605    0.9146\n",
      "        10   MaxAbsScaler SGD                               0:00:25       0.9031    0.9146\n",
      "        11   MaxAbsScaler SGD                               0:00:31       0.9038    0.9146\n",
      "        12   MaxAbsScaler RandomForest                      0:00:27       0.8892    0.9146\n",
      "        13   StandardScalerWrapper ExtremeRandomTrees       0:00:34       0.8880    0.9146\n",
      "        14   MaxAbsScaler RandomForest                      0:00:28       0.7944    0.9146\n",
      "        15   MaxAbsScaler SGD                               0:00:27       0.8632    0.9146\n",
      "        16   MaxAbsScaler RandomForest                      0:00:33       0.8880    0.9146\n",
      "        17   MaxAbsScaler ExtremeRandomTrees                0:00:34       0.8964    0.9146\n",
      "        18   SparseNormalizer ExtremeRandomTrees            0:00:40       0.7270    0.9146\n",
      "        19   MaxAbsScaler SGD                               0:00:39       0.9067    0.9146\n",
      "        20   MaxAbsScaler ExtremeRandomTrees                0:00:30       0.7553    0.9146\n",
      "        21   MaxAbsScaler RandomForest                      0:00:29       0.7898    0.9146\n",
      "        22   MaxAbsScaler LightGBM                          0:00:34       0.9127    0.9146\n",
      "        23   MaxAbsScaler ExtremeRandomTrees                0:00:44       0.8880    0.9146\n",
      "        24   MaxAbsScaler ExtremeRandomTrees                0:01:08       0.8900    0.9146\n",
      "        25   TruncatedSVDWrapper XGBoostClassifier          0:00:30       0.8880    0.9146\n",
      "        26   MaxAbsScaler ExtremeRandomTrees                0:00:35       0.9002    0.9146\n",
      "        27   SparseNormalizer XGBoostClassifier             0:00:45       0.9131    0.9146\n",
      "        28   MaxAbsScaler ExtremeRandomTrees                0:00:38       0.8880    0.9146\n",
      "        29   MaxAbsScaler LightGBM                          0:00:33       0.9054    0.9146\n",
      "        30   TruncatedSVDWrapper XGBoostClassifier          0:00:35       0.8880    0.9146\n",
      "        31   StandardScalerWrapper ExtremeRandomTrees       0:00:49       0.8997    0.9146\n",
      "        32   SparseNormalizer LightGBM                      0:00:34       0.9023    0.9146\n",
      "        33   StandardScalerWrapper XGBoostClassifier        0:00:42       0.9130    0.9146\n",
      "        34   SparseNormalizer XGBoostClassifier             0:01:16       0.9127    0.9146\n",
      "        35   TruncatedSVDWrapper XGBoostClassifier          0:00:31       0.8880    0.9146\n",
      "        36   MaxAbsScaler LightGBM                          0:00:30       0.9099    0.9146\n",
      "        37   MaxAbsScaler ExtremeRandomTrees                0:00:53       0.9002    0.9146\n",
      "        38   StandardScalerWrapper LightGBM                 0:00:28       0.9042    0.9146\n",
      "        39   StandardScalerWrapper LightGBM                 0:00:32       0.9063    0.9146\n",
      "        40   StandardScalerWrapper XGBoostClassifier        0:00:25       0.8929    0.9146\n",
      "        41   SparseNormalizer XGBoostClassifier             0:00:37       0.9116    0.9146\n",
      "        42   SparseNormalizer XGBoostClassifier             0:00:28       0.9122    0.9146\n",
      "        43   SparseNormalizer XGBoostClassifier             0:01:04       0.9119    0.9146\n",
      "        44   MaxAbsScaler LightGBM                          0:00:27       0.9067    0.9146\n",
      "        45   StandardScalerWrapper XGBoostClassifier        0:00:34       0.8880    0.9146\n",
      "        46   SparseNormalizer XGBoostClassifier             0:00:40       0.9130    0.9146\n",
      "        47    VotingEnsemble                                0:01:12       0.9174    0.9174\n",
      "        48    StackEnsemble                                 0:01:24       0.9138    0.9174\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'runId': 'AutoML_770c77d2-9751-454d-8d86-98b0c225ada9',\n",
       " 'target': 'local',\n",
       " 'status': 'Completed',\n",
       " 'startTimeUtc': '2020-11-23T14:53:43.605472Z',\n",
       " 'endTimeUtc': '2020-11-23T15:26:42.769357Z',\n",
       " 'properties': {'num_iterations': '1000',\n",
       "  'training_type': 'TrainFull',\n",
       "  'acquisition_function': 'EI',\n",
       "  'primary_metric': 'accuracy',\n",
       "  'train_split': '0',\n",
       "  'acquisition_parameter': '0',\n",
       "  'num_cross_validation': '5',\n",
       "  'target': 'local',\n",
       "  'AMLSettingsJsonString': '{\"path\":null,\"name\":\"automl-experiment\",\"subscription_id\":\"55e71b9d-a209-42c0-8818-ca9cc885909c\",\"resource_group\":\"aml-quickstarts-127549\",\"workspace_name\":\"quick-starts-ws-127549\",\"region\":\"southcentralus\",\"compute_target\":\"local\",\"spark_service\":null,\"azure_service\":\"Microsoft.AzureNotebookVM\",\"_local_managed_run_id\":null,\"many_models\":false,\"pipeline_fetch_max_batch_size\":1,\"iterations\":1000,\"primary_metric\":\"accuracy\",\"task_type\":\"classification\",\"data_script\":null,\"validation_size\":0.0,\"n_cross_validations\":5,\"y_min\":null,\"y_max\":null,\"num_classes\":2,\"featurization\":\"auto\",\"_ignore_package_version_incompatibilities\":false,\"is_timeseries\":false,\"max_cores_per_iteration\":1,\"max_concurrent_iterations\":1,\"iteration_timeout_minutes\":null,\"mem_in_mb\":null,\"enforce_time_on_windows\":false,\"experiment_timeout_minutes\":30,\"experiment_exit_score\":null,\"whitelist_models\":null,\"blacklist_algos\":[\"TensorFlowLinearClassifier\",\"TensorFlowDNN\"],\"supported_models\":[\"LinearSVM\",\"SGD\",\"BernoulliNaiveBayes\",\"GradientBoosting\",\"KNN\",\"SVM\",\"AveragedPerceptronClassifier\",\"TensorFlowLinearClassifier\",\"MultinomialNaiveBayes\",\"RandomForest\",\"LogisticRegression\",\"DecisionTree\",\"XGBoostClassifier\",\"TensorFlowDNN\",\"ExtremeRandomTrees\",\"LightGBM\"],\"auto_blacklist\":true,\"blacklist_samples_reached\":false,\"exclude_nan_labels\":true,\"verbosity\":20,\"_debug_log\":\"automl.log\",\"show_warnings\":false,\"model_explainability\":true,\"service_url\":null,\"sdk_url\":null,\"sdk_packages\":null,\"enable_onnx_compatible_models\":false,\"enable_split_onnx_featurizer_estimator_models\":false,\"vm_type\":null,\"telemetry_verbosity\":20,\"send_telemetry\":true,\"enable_dnn\":false,\"scenario\":\"SDK-1.13.0\",\"environment_label\":null,\"force_text_dnn\":false,\"enable_feature_sweeping\":true,\"enable_early_stopping\":false,\"early_stopping_n_iters\":10,\"metrics\":null,\"enable_ensembling\":true,\"enable_stack_ensembling\":true,\"ensemble_iterations\":15,\"enable_tf\":false,\"enable_subsampling\":null,\"subsample_seed\":null,\"enable_nimbusml\":false,\"enable_streaming\":false,\"force_streaming\":false,\"track_child_runs\":true,\"allowed_private_models\":[],\"label_column_name\":\"y\",\"weight_column_name\":null,\"cv_split_column_names\":null,\"enable_local_managed\":false,\"cost_mode\":1,\"lag_length\":0,\"metric_operation\":\"maximize\",\"preprocess\":true}',\n",
       "  'DataPrepJsonString': None,\n",
       "  'EnableSubsampling': None,\n",
       "  'runTemplate': 'AutoML',\n",
       "  'azureml.runsource': 'automl',\n",
       "  'display_task_type': 'classification',\n",
       "  'dependencies_versions': '{\"azureml-widgets\": \"1.18.0\", \"azureml-train\": \"1.18.0\", \"azureml-train-restclients-hyperdrive\": \"1.18.0\", \"azureml-train-core\": \"1.18.0\", \"azureml-train-automl\": \"1.18.0\", \"azureml-train-automl-runtime\": \"1.18.0\", \"azureml-train-automl-client\": \"1.18.0\", \"azureml-tensorboard\": \"1.18.0\", \"azureml-telemetry\": \"1.18.0\", \"azureml-sdk\": \"1.18.0\", \"azureml-samples\": \"0+unknown\", \"azureml-pipeline\": \"1.18.0\", \"azureml-pipeline-steps\": \"1.18.0\", \"azureml-pipeline-core\": \"1.18.0\", \"azureml-opendatasets\": \"1.18.0\", \"azureml-model-management-sdk\": \"1.0.1b6.post1\", \"azureml-mlflow\": \"1.18.0\", \"azureml-interpret\": \"1.18.0\", \"azureml-explain-model\": \"1.18.0\", \"azureml-defaults\": \"1.18.0\", \"azureml-dataset-runtime\": \"1.18.0\", \"azureml-dataprep\": \"2.4.2\", \"azureml-dataprep-rslex\": \"1.2.2\", \"azureml-dataprep-native\": \"24.0.0\", \"azureml-datadrift\": \"1.18.0\", \"azureml-core\": \"1.18.0\", \"azureml-contrib-services\": \"1.18.0\", \"azureml-contrib-server\": \"1.18.0\", \"azureml-contrib-reinforcementlearning\": \"1.18.0\", \"azureml-contrib-pipeline-steps\": \"1.18.0\", \"azureml-contrib-notebook\": \"1.18.0\", \"azureml-contrib-interpret\": \"1.18.0\", \"azureml-contrib-gbdt\": \"1.18.0\", \"azureml-contrib-fairness\": \"1.18.0\", \"azureml-contrib-dataset\": \"1.18.0\", \"azureml-cli-common\": \"1.18.0\", \"azureml-automl-runtime\": \"1.18.0\", \"azureml-automl-core\": \"1.18.0\", \"azureml-accel-models\": \"1.18.0\"}',\n",
       "  '_aml_system_scenario_identification': 'Local.Parent',\n",
       "  'ClientSdkVersion': '1.18.0',\n",
       "  'ClientType': 'SDK',\n",
       "  'environment_cpu_name': 'AzureML-AutoML',\n",
       "  'environment_cpu_version': '44',\n",
       "  'environment_gpu_name': 'AzureML-AutoML-GPU',\n",
       "  'environment_gpu_version': '32',\n",
       "  'root_attribution': 'automl',\n",
       "  'attribution': 'AutoML',\n",
       "  'Orchestrator': 'AutoML',\n",
       "  '_azureml.ComputeTargetType': 'local',\n",
       "  'ProblemInfoJsonString': '{\"dataset_num_categorical\": 0, \"is_sparse\": true, \"subsampling\": false, \"dataset_classes\": 2, \"dataset_features\": 120, \"dataset_samples\": 23065, \"single_frequency_class_detected\": false}',\n",
       "  'feature_skus': 'automatedml_sdk_guardrails'},\n",
       " 'inputDatasets': [],\n",
       " 'outputDatasets': [],\n",
       " 'logFiles': {}}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# from azureml.widgets import RunDetails\n",
    "RunDetails(automl_run).show()\n",
    "\n",
    "automl_run.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieve and Save Best Model\n",
    "\n",
    "Below we select the best model from all the training iterations using get_output method.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "['accuracy_table', 'confusion_matrix', 'explanation/33d138e1/classes.interpret.json', 'explanation/33d138e1/expected_values.interpret.json', 'explanation/33d138e1/features.interpret.json', 'explanation/33d138e1/global_names/0.interpret.json', 'explanation/33d138e1/global_rank/0.interpret.json', 'explanation/33d138e1/global_values/0.interpret.json', 'explanation/33d138e1/local_importance_values.interpret.json', 'explanation/33d138e1/per_class_names/0.interpret.json', 'explanation/33d138e1/per_class_rank/0.interpret.json', 'explanation/33d138e1/per_class_values/0.interpret.json', 'explanation/33d138e1/rich_metadata.interpret.json', 'explanation/33d138e1/visualization_dict.interpret.json', 'explanation/56c439a1/classes.interpret.json', 'explanation/56c439a1/eval_data_viz.interpret.json', 'explanation/56c439a1/expected_values.interpret.json', 'explanation/56c439a1/features.interpret.json', 'explanation/56c439a1/global_names/0.interpret.json', 'explanation/56c439a1/global_rank/0.interpret.json', 'explanation/56c439a1/global_values/0.interpret.json', 'explanation/56c439a1/local_importance_values.interpret.json', 'explanation/56c439a1/per_class_names/0.interpret.json', 'explanation/56c439a1/per_class_rank/0.interpret.json', 'explanation/56c439a1/per_class_values/0.interpret.json', 'explanation/56c439a1/rich_metadata.interpret.json', 'explanation/56c439a1/visualization_dict.interpret.json', 'outputs/conda_env_v_1_0_0.yml', 'outputs/env_dependencies.json', 'outputs/model.pkl', 'outputs/pipeline_graph.json', 'outputs/scoring_file_v_1_0_0.py']"
      ],
      "text/plain": [
       "['accuracy_table',\n",
       " 'confusion_matrix',\n",
       " 'explanation/33d138e1/classes.interpret.json',\n",
       " 'explanation/33d138e1/expected_values.interpret.json',\n",
       " 'explanation/33d138e1/features.interpret.json',\n",
       " 'explanation/33d138e1/global_names/0.interpret.json',\n",
       " 'explanation/33d138e1/global_rank/0.interpret.json',\n",
       " 'explanation/33d138e1/global_values/0.interpret.json',\n",
       " 'explanation/33d138e1/local_importance_values.interpret.json',\n",
       " 'explanation/33d138e1/per_class_names/0.interpret.json',\n",
       " 'explanation/33d138e1/per_class_rank/0.interpret.json',\n",
       " 'explanation/33d138e1/per_class_values/0.interpret.json',\n",
       " 'explanation/33d138e1/rich_metadata.interpret.json',\n",
       " 'explanation/33d138e1/visualization_dict.interpret.json',\n",
       " 'explanation/56c439a1/classes.interpret.json',\n",
       " 'explanation/56c439a1/eval_data_viz.interpret.json',\n",
       " 'explanation/56c439a1/expected_values.interpret.json',\n",
       " 'explanation/56c439a1/features.interpret.json',\n",
       " 'explanation/56c439a1/global_names/0.interpret.json',\n",
       " 'explanation/56c439a1/global_rank/0.interpret.json',\n",
       " 'explanation/56c439a1/global_values/0.interpret.json',\n",
       " 'explanation/56c439a1/local_importance_values.interpret.json',\n",
       " 'explanation/56c439a1/per_class_names/0.interpret.json',\n",
       " 'explanation/56c439a1/per_class_rank/0.interpret.json',\n",
       " 'explanation/56c439a1/per_class_values/0.interpret.json',\n",
       " 'explanation/56c439a1/rich_metadata.interpret.json',\n",
       " 'explanation/56c439a1/visualization_dict.interpret.json',\n",
       " 'outputs/conda_env_v_1_0_0.yml',\n",
       " 'outputs/env_dependencies.json',\n",
       " 'outputs/model.pkl',\n",
       " 'outputs/pipeline_graph.json',\n",
       " 'outputs/scoring_file_v_1_0_0.py']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Retrieve model\n",
    "\n",
    "best_run, fitted_model = automl_run.get_output()\n",
    "\n",
    "# get name of files of best_run\n",
    "best_run.get_file_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PreFittedSoftVotingClassifier(classification_labels=None,\n",
      "                              estimators=[('1',\n",
      "                                           Pipeline(memory=None,\n",
      "                                                    steps=[('maxabsscaler',\n",
      "                                                            MaxAbsScaler(copy=True)),\n",
      "                                                           ('xgboostclassifier',\n",
      "                                                            XGBoostClassifier(base_score=0.5,\n",
      "                                                                              booster='gbtree',\n",
      "                                                                              colsample_bylevel=1,\n",
      "                                                                              colsample_bynode=1,\n",
      "                                                                              colsample_bytree=1,\n",
      "                                                                              gamma=0,\n",
      "                                                                              learning_rate=0.1,\n",
      "                                                                              max_delta_step=0,\n",
      "                                                                              max_depth=3,\n",
      "                                                                              min_child_weight=1,\n",
      "                                                                              miss...\n",
      "                                                                                 l1_ratio=0.836734693877551,\n",
      "                                                                                 learning_rate='constant',\n",
      "                                                                                 loss='modified_huber',\n",
      "                                                                                 max_iter=1000,\n",
      "                                                                                 n_jobs=1,\n",
      "                                                                                 penalty='l2',\n",
      "                                                                                 power_t=0.2222222222222222,\n",
      "                                                                                 random_state=None,\n",
      "                                                                                 tol=0.0001))],\n",
      "                                                    verbose=False))],\n",
      "                              flatten_transform=None,\n",
      "                              weights=[0.38461538461538464, 0.15384615384615385,\n",
      "                                       0.07692307692307693, 0.15384615384615385,\n",
      "                                       0.07692307692307693, 0.07692307692307693,\n",
      "                                       0.07692307692307693])\n"
     ]
    }
   ],
   "source": [
    "# save best model\n",
    "best_run.register_model(model_name = \"model.pkl\", model_path = './outputs/')\n",
    "print(fitted_model._final_estimator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "[('datatransformer', DataTransformer(enable_dnn=None, enable_feature_sweeping=None,\n",
       "                feature_sweeping_config=None, feature_sweeping_timeout=None,\n",
       "                featurization_config=None, force_text_dnn=None,\n",
       "                is_cross_validation=None, is_onnx_compatible=None, logger=None,\n",
       "                observer=None, task=None, working_dir=None)), ('prefittedsoftvotingclassifier', PreFittedSoftVotingClassifier(classification_labels=None,\n",
       "                              estimators=[('1',\n",
       "                                           Pipeline(memory=None,\n",
       "                                                    steps=[('maxabsscaler',\n",
       "                                                            MaxAbsScaler(copy=True)),\n",
       "                                                           ('xgboostclassifier',\n",
       "                                                            XGBoostClassifier(base_score=0.5,\n",
       "                                                                              booster='gbtree',\n",
       "                                                                              colsample_bylevel=1,\n",
       "                                                                              colsample_bynode=1,\n",
       "                                                                              colsample_bytree=1,\n",
       "                                                                              gamma=0,\n",
       "                                                                              learning_rate=0.1,\n",
       "                                                                              max_delta_step=0,\n",
       "                                                                              max_depth=3,\n",
       "                                                                              min_child_weight=1,\n",
       "                                                                              miss...\n",
       "                                                                                 l1_ratio=0.836734693877551,\n",
       "                                                                                 learning_rate='constant',\n",
       "                                                                                 loss='modified_huber',\n",
       "                                                                                 max_iter=1000,\n",
       "                                                                                 n_jobs=1,\n",
       "                                                                                 penalty='l2',\n",
       "                                                                                 power_t=0.2222222222222222,\n",
       "                                                                                 random_state=None,\n",
       "                                                                                 tol=0.0001))],\n",
       "                                                    verbose=False))],\n",
       "                              flatten_transform=None,\n",
       "                              weights=[0.38461538461538464, 0.15384615384615385,\n",
       "                                       0.07692307692307693, 0.15384615384615385,\n",
       "                                       0.07692307692307693, 0.07692307692307693,\n",
       "                                       0.07692307692307693]))]"
      ],
      "text/plain": [
       "[('datatransformer',\n",
       "  DataTransformer(enable_dnn=None, enable_feature_sweeping=None,\n",
       "                  feature_sweeping_config=None, feature_sweeping_timeout=None,\n",
       "                  featurization_config=None, force_text_dnn=None,\n",
       "                  is_cross_validation=None, is_onnx_compatible=None, logger=None,\n",
       "                  observer=None, task=None, working_dir=None)),\n",
       " ('prefittedsoftvotingclassifier',\n",
       "  PreFittedSoftVotingClassifier(classification_labels=None,\n",
       "                                estimators=[('1',\n",
       "                                             Pipeline(memory=None,\n",
       "                                                      steps=[('maxabsscaler',\n",
       "                                                              MaxAbsScaler(copy=True)),\n",
       "                                                             ('xgboostclassifier',\n",
       "                                                              XGBoostClassifier(base_score=0.5,\n",
       "                                                                                booster='gbtree',\n",
       "                                                                                colsample_bylevel=1,\n",
       "                                                                                colsample_bynode=1,\n",
       "                                                                                colsample_bytree=1,\n",
       "                                                                                gamma=0,\n",
       "                                                                                learning_rate=0.1,\n",
       "                                                                                max_delta_step=0,\n",
       "                                                                                max_depth=3,\n",
       "                                                                                min_child_weight=1,\n",
       "                                                                                miss...\n",
       "                                                                                   l1_ratio=0.836734693877551,\n",
       "                                                                                   learning_rate='constant',\n",
       "                                                                                   loss='modified_huber',\n",
       "                                                                                   max_iter=1000,\n",
       "                                                                                   n_jobs=1,\n",
       "                                                                                   penalty='l2',\n",
       "                                                                                   power_t=0.2222222222222222,\n",
       "                                                                                   random_state=None,\n",
       "                                                                                   tol=0.0001))],\n",
       "                                                      verbose=False))],\n",
       "                                flatten_transform=None,\n",
       "                                weights=[0.38461538461538464, 0.15384615384615385,\n",
       "                                         0.07692307692307693, 0.15384615384615385,\n",
       "                                         0.07692307692307693, 0.07692307692307693,\n",
       "                                         0.07692307692307693]))]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fitted_model.steps\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cleaning Up Cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "cpu_cluster.delete()"
   ]
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "python3-azureml"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}