supply-chain/world_of_supply/world-of-supply.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multi-echelon Supply Chain Optimization Using Deep Reinforcement Learning\n",
    "\n",
    "This notebook demonstrates how replenishment and transportation decision in a complex multi-echelon supply chain can be optimized using reinforcement learning methods.\n",
    "\n",
    "### Use Case\n",
    "We assume a multi-echelon supply chain that serves stochastic tyme-varying demand. The economic model of the chain includes storage and transportation costs, capacity constraints, and other complexities. Our goal is to learn optimal policy that controls replenishment and transportation decisions balancing various costs and stock-out risks/losses. \n",
    "\n",
    "### Prototype: Approach and Data\n",
    "We create a relatively advanced simulator (`World of Supply`) of a multi-echelon supply chain that includes multiple inventory management and transportation controls. We then use an off-the-shelf deep reinforcement learning component to learn the control policy.\n",
    "\n",
    "### Usage and Productization\n",
    "This prototype is provided mainly for educational and experimentation purposes. Productization of this approach can be challenging because of a relatively large action space (multiple control variables) which results in instability of the learning process and highly irregular control policies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Imports and settings\n",
    "#\n",
    "import numpy as np\n",
    "from tqdm import tqdm as tqdm\n",
    "import importlib\n",
    "\n",
    "import world_of_supply_tools as wst\n",
    "importlib.reload(wst)\n",
    "\n",
    "wst.print_hardware_status()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Core Simulation Logic and Rendering\n",
    "\n",
    "In this section, we test the core simulator and renderer (without RL adapters and integrations)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import world_of_supply_environment as ws\n",
    "import world_of_supply_renderer as wsr\n",
    "import world_of_supply_tools as wst\n",
    "for module in [ws, wsr, wst]:\n",
    "    importlib.reload(module)\n",
    "        \n",
    "# Measure the simulation rate, steps/sec\n",
    "eposod_len = 1000\n",
    "n_episods = 10\n",
    "world = ws.WorldBuilder.create()\n",
    "tracker = wst.SimulationTracker(eposod_len, n_episods, world.facilities.keys())\n",
    "with tqdm(total=eposod_len * n_episods) as pbar:\n",
    "    for i in range(n_episods):\n",
    "        world = ws.WorldBuilder.create()\n",
    "        policy = ws.SimpleControlPolicy()\n",
    "        for t in range(eposod_len):\n",
    "            outcome = world.act(policy.compute_control(world))\n",
    "            tracker.add_sample(i, t, world.economy.global_balance().total(), \n",
    "                               {k: v.total() for k, v in outcome.facility_step_balance_sheets.items() } )\n",
    "            pbar.update(1)        \n",
    "tracker.render()\n",
    "    \n",
    "# Test rendering\n",
    "renderer = wsr.AsciiWorldRenderer()\n",
    "frame_seq = []\n",
    "world = ws.WorldBuilder.create()\n",
    "policy = ws.SimpleControlPolicy()\n",
    "for epoch in tqdm(range(300)):\n",
    "    frame = renderer.render(world)\n",
    "    frame_seq.append(np.asarray(frame))\n",
    "    world.act(policy.compute_control(world))\n",
    "\n",
    "print('Rendering the animation...')\n",
    "wsr.AsciiWorldRenderer.plot_sequence_images(frame_seq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Policy Training\n",
    "\n",
    "In this section, we run RLlib policy trainers. These trainers evaluate the hand coded policy, learn a new policy from scrath, or learn a new policy by playing against the hand coded policy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import world_of_supply_rllib_models as wsm\n",
    "importlib.reload(wsm)\n",
    "import world_of_supply_rllib as wsrl\n",
    "importlib.reload(wsrl)\n",
    "import world_of_supply_rllib_training as wsrt\n",
    "importlib.reload(wsrt)\n",
    "\n",
    "wsrt.print_model_summaries()\n",
    "\n",
    "# Policy training\n",
    "#trainer = wsrt.play_baseline(n_iterations = 1)\n",
    "trainer = wsrt.train_ppo(n_iterations = 600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Policy Evaluation\n",
    "\n",
    "In this section, we evaluate the trained policy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rendering One Episod for the Trained Policy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import world_of_supply_renderer as wsren\n",
    "import world_of_supply_tools as wst\n",
    "import world_of_supply_rllib as wsrl\n",
    "import world_of_supply_rllib_training as wstr\n",
    "for module in [wsren, wst, wsrl, wstr]:\n",
    "    importlib.reload(module)\n",
    "\n",
    "# Parameters of the tracing simulation\n",
    "policy_mode = 'baseline'   # 'baseline' or 'trained'\n",
    "episod_duration = 500\n",
    "steps_to_render = None#(0, episod_duration)  # (0, episod_duration) or None\n",
    "\n",
    "# Create the environment\n",
    "renderer = wsren.AsciiWorldRenderer()\n",
    "frame_seq = []\n",
    "env_config_for_rendering = wstr.env_config.copy()\n",
    "env_config_for_rendering.update({\n",
    "    'downsampling_rate': 1\n",
    "})\n",
    "env = wsrl.WorldOfSupplyEnv(env_config_for_rendering)\n",
    "env.set_iteration(1, 1)\n",
    "print(f\"Environment: Producer action space {env.action_space_producer}, Consumer action space {env.action_space_consumer}, Observation space {env.observation_space}\")\n",
    "states = env.reset()\n",
    "infos = None\n",
    "    \n",
    "def load_policy(agent_id):\n",
    "    if policy_mode == 'baseline':\n",
    "        if wsrl.Utils.is_producer_agent(agent_id):\n",
    "            return wsrl.ProducerSimplePolicy(env.observation_space, env.action_space_producer, wsrl.SimplePolicy.get_config_from_env(env))\n",
    "        elif wsrl.Utils.is_consumer_agent(agent_id):\n",
    "            return wsrl.ConsumerSimplePolicy(env.observation_space, env.action_space_consumer, wsrl.SimplePolicy.get_config_from_env(env))\n",
    "        else:\n",
    "            raise Exception(f'Unknown agent type {agent_id}')\n",
    "    \n",
    "    if policy_mode == 'trained':\n",
    "        policy_map = wstr.policy_mapping_global.copy()\n",
    "        wstr.update_policy_map(policy_map)   \n",
    "        return trainer.get_policy(wstr.create_policy_mapping_fn(policy_map)(agent_id))\n",
    "\n",
    "policies = {}\n",
    "rnn_states = {}\n",
    "for agent_id in states.keys():\n",
    "    policies[agent_id] = load_policy(agent_id)\n",
    "    rnn_states[agent_id] = policies[agent_id].get_initial_state()\n",
    "    \n",
    "# Simulation loop\n",
    "tracker = wst.SimulationTracker(episod_duration, 1, env.agent_ids())\n",
    "for epoch in tqdm(range(episod_duration)):\n",
    "    \n",
    "    action_dict = {}\n",
    "    if epoch % wstr.env_config['downsampling_rate'] == 0:\n",
    "        for agent_id, state in states.items():\n",
    "            policy = policies[agent_id]\n",
    "            rnn_state = rnn_states[agent_id]\n",
    "            if infos is not None and agent_id in infos:\n",
    "                action_dict[agent_id], rnn_state, _ = policy.compute_single_action( state, info=infos[agent_id], state=rnn_state ) \n",
    "            else:\n",
    "                action_dict[agent_id], rnn_state, _ = policy.compute_single_action( state, state=rnn_state )   \n",
    "   \n",
    "    states, rewards, dones, infos = env.step(action_dict)\n",
    "    tracker.add_sample(0, epoch, env.world.economy.global_balance().total(), rewards)\n",
    "    \n",
    "    if steps_to_render is not None and epoch >= steps_to_render[0] and epoch < steps_to_render[1]:\n",
    "        frame = renderer.render(env.world)\n",
    "        frame_seq.append(np.asarray(frame))\n",
    " \n",
    "tracker.render()\n",
    "\n",
    "if steps_to_render is not None:\n",
    "    print('Rendering the animation...')\n",
    "    wsren.AsciiWorldRenderer.plot_sequence_images(frame_seq)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}