From 8ce4b5370c4a59064f6cd832dfbb244180a878b4 Mon Sep 17 00:00:00 2001 From: EastSmith <67545545+EastSmith@users.noreply.github.com> Date: Fri, 4 Jun 2021 16:09:14 +0800 Subject: [PATCH] Advantage Actor-Critic(A2C) (#973) * Create Advantage Actor-Critic(A2C).ipynb * Update Advantage Actor-Critic(A2C).ipynb * Update Advantage Actor-Critic(A2C).ipynb * Update Advantage Actor-Critic(A2C).ipynb * Update Advantage Actor-Critic(A2C).ipynb --- .../Advantage Actor-Critic(A2C).ipynb | 593 ++++++++++++++++++ 1 file changed, 593 insertions(+) create mode 100644 paddle2.0_docs/Advantage Actor-Critic(A2C)/Advantage Actor-Critic(A2C).ipynb diff --git a/paddle2.0_docs/Advantage Actor-Critic(A2C)/Advantage Actor-Critic(A2C).ipynb b/paddle2.0_docs/Advantage Actor-Critic(A2C)/Advantage Actor-Critic(A2C).ipynb new file mode 100644 index 00000000..8ca9d7af --- /dev/null +++ b/paddle2.0_docs/Advantage Actor-Critic(A2C)/Advantage Actor-Critic(A2C).ipynb @@ -0,0 +1,593 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "# **强化学习——Advantage Actor-Critic(A2C)**\n", + "**作者：**:[EastSmith](https://github.com/EastSmith)\n", + "\n", + "**日期：** 2021.05 \n", + "\n", + "**AI Studio项目**：[点击体验](https://aistudio.baidu.com/aistudio/projectdetail/1766508)\n", + "## **一、介绍**\n", + "### **让我们回顾一下以前的知识， 您可能知道，目前有两种主要的RL方法类型：** \n", + "\n", + "* **基于值**：他们试图找到或近似最佳值函数，这是一个动作和一个值之间的映射。 值越高，动作越好。 最著名的算法是Q学习及其所有增强的方法， 例如Deep Q Networks，Double Dueling Q Networks等 \n", + "* **基于策略的**：基于策略的算法（例如“ 策略梯度” 和REINFORCE）尝试直接找到最佳策略，而无需Q值作为中间步骤。\n", + "\n", + "当这两个算法流行以后，下一个显而易见的步骤是……尝试合并它们。 这就是演员——评论家的诞生方式。 演员评论家旨在利用基于价值和基于策略的优点，同时消除其弊端。 以及他们如何做到这一点？\n", + "\n", + "### **主要思想是将模型分为两部分：一个用于基于状态计算动作，另一个用于估计动作的Q值。**\n", + "\n", + "演员将状态作为输入并输出最佳动作。 它实质上是通过控制代理的行为来学习最佳策略 （基于策略） 。 另一方面，评论家通过计算值函数评估动作 （基于值）来 。 这两个模型参加了一场比赛，随着时间的流逝，他们各自的角色都变得更好。 结果是，与单独使用两种方法相比，整个体系结构将学会更有效地玩游戏。 \n", + "\n", + "让两个模型相互交互（或竞争）的想法在机器学习领域越来越流行。 例如， 生成对抗网络（Generative Adversarial Networks） 或 变体自动编码器（Variational Autoencoders）\n", + "\n", + "\n", + "### **演员——评论家：**\n", + "（可以参照的教程：[强化学习——Actor Critic Method-使用文档-PaddlePaddle深度学习平台](https://www.paddlepaddle.org.cn/documentation/docs/zh/tutorial/reinforcement_learning/Actor_Critic_Method/Actor_Critic_Method.html)）\n", + "\n", + "演员——评论家的一个很好的比喻是一个小男孩和他的母亲。 这个孩子（演员）不断尝试新事物并探索他周围的环境。 他吃自己的玩具，触摸热烤箱，用头撞在墙上（谁知道他为什么这样做）。 他的母亲（评论家）看着他，并批评或称赞他。 这个孩子听母亲讲给他的话，并调整自己的行为。 随着孩子的成长，他学会了什么动作是坏事还是好事，并且他实质上学会了玩称为生活的游戏。 这与演员评论家的工作方式完全相同。\n", + "\n", + "参与者演员可以是类似于**神经网络的函数逼近器**，其任务是针对给定状态产生最佳动作。 当然，它可以是全连接的神经网络，也可以是卷积或其他任何东西。 评论家是另一个函数逼近器，它接收参与者输入的环境和动作作为输入，将它们连接起来并输出评分值（Q值）。 让我提醒您几秒钟，Q值本质上是将来的最大奖励。\n", + "\n", + "这两个网络的训练是分别进行的，评论家使用梯度上升（找到全局最大值而不是最小值）来更新它们的权重。 随着时间的流逝，演员正在学会做出更好的动作（他开始学习策略），而评论家在评估这些动作方面也越来越好。 重要的是要注意，权重的更新发生在每个步骤（TD学习），而不是发生在事件的结尾，这与策略梯度相反。\n", + "\n", + "事实证明，演员评论家能够学习大型复杂的环境，并且已在很多著名的2d和3d游戏中使用，例如Doom，Super Mario等。\n", + "\n", + "### **优势-演员-评论家 Advantage-Actor-Critic（A2C）**\n", + "什么是优势？ Q值实际上可以分解为两部分：状态值函数V（s）和优势值A（s，a）：\n", + "\n", + "Q（s，a）= V（s）+ A（s，a）\n", + "\n", + "![](https://ai-studio-static-online.cdn.bcebos.com/aa58a55591394f478cfbde40227812e1ab21bc088a7546c09e440b4b353cc761)\n", + "\n", + "优势函数能够评估在给定状态下与其他行为相比更好的行为，而众所周知，价值函数是评估在此状态下行为的良好程度。\n", + "\n", + "你猜这是怎么回事，对不对？ 与其让评论家学习Q值，不如让评论家学习Advantage值 。 这样，对行为的评估不仅基于行为的良好程度，而且还取决于行为可以改善的程度。 优势函数的优势是它减少了策略网络的数值差异并稳定了模型。 \n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "## **二、环境配置**\n", + "本教程基于Paddle 2.1 编写，如果您的环境不是本版本，请先参考官网[安装](https://www.paddlepaddle.org.cn/install/quick) Paddle 2.1 。" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "import math\r\n", + "import random\r\n", + "import os\r\n", + "import gym\r\n", + "import numpy as np\r\n", + "\r\n", + "import paddle\r\n", + "import paddle.nn as nn\r\n", + "import paddle.optimizer as optim\r\n", + "import paddle.nn.functional as F\r\n", + "from paddle.distribution import Categorical\r\n", + "\r\n", + "import matplotlib.pyplot as plt\r\n", + "from visualdl import LogWriter" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "## **三、实施“优势-演员-评论家 Advantage-Actor-Critic（A2C）”算法**\n", + "### **构建多个进程玩CartPole-v0**\n", + "A2C会构建多个进程，包括多个并行的 worker，与独立的环境进行交互，收集独立的经验。详细代码在multiprocessing_env.py里。简单介绍一下创建多环境的过程：env = gym.make(env_name)只能创建一个线程，智能体只能和一个环境进行交互，而使用 SubprocVecEnv(envs)可以创建多个并行的环境，用num_envs定义并行环境的数量。需要注意的是如果创建的是多个并行的环境envs的话，那么envs.step()需要输入的是成组的动作，每个环境对应一组动作，相应的envs返回的next_state, reward等也是成组的。\n", + "![](https://ai-studio-static-online.cdn.bcebos.com/5d443811448d4f13bf6a16b43bb0339e885b3bfee1b04d23be1665f925877c5d)\n", + "\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "\r\n", + "#This code is from openai baseline\r\n", + "#https://github.com/openai/baselines/tree/master/baselines/common/vec_env\r\n", + "\r\n", + "import numpy as np\r\n", + "from multiprocessing import Process, Pipe\r\n", + "\r\n", + "def worker(remote, parent_remote, env_fn_wrapper):\r\n", + " parent_remote.close()\r\n", + " env = env_fn_wrapper.x()\r\n", + " while True:\r\n", + " cmd, data = remote.recv()\r\n", + " if cmd == 'step':\r\n", + " ob, reward, done, info = env.step(data)\r\n", + " # ob, reward, done, info = env.step(1)\r\n", + "\r\n", + " if done:\r\n", + " ob = env.reset()\r\n", + " remote.send((ob, reward, done, info))\r\n", + " elif cmd == 'reset':\r\n", + " ob = env.reset()\r\n", + " remote.send(ob)\r\n", + " elif cmd == 'reset_task':\r\n", + " ob = env.reset_task()\r\n", + " remote.send(ob)\r\n", + " elif cmd == 'close':\r\n", + " remote.close()\r\n", + " break\r\n", + " elif cmd == 'get_spaces':\r\n", + " remote.send((env.observation_space, env.action_space))\r\n", + " else:\r\n", + " raise NotImplementedError\r\n", + "\r\n", + "class VecEnv(object):\r\n", + " \"\"\"\r\n", + " An abstract asynchronous, vectorized environment.\r\n", + " \"\"\"\r\n", + " def __init__(self, num_envs, observation_space, action_space):\r\n", + " self.num_envs = num_envs\r\n", + " self.observation_space = observation_space\r\n", + " self.action_space = action_space\r\n", + "\r\n", + " def reset(self):\r\n", + " \"\"\"\r\n", + " Reset all the environments and return an array of\r\n", + " observations, or a tuple of observation arrays.\r\n", + " If step_async is still doing work, that work will\r\n", + " be cancelled and step_wait() should not be called\r\n", + " until step_async() is invoked again.\r\n", + " \"\"\"\r\n", + " pass\r\n", + "\r\n", + " def step_async(self, actions):\r\n", + " \"\"\"\r\n", + " Tell all the environments to start taking a step\r\n", + " with the given actions.\r\n", + " Call step_wait() to get the results of the step.\r\n", + " You should not call this if a step_async run is\r\n", + " already pending.\r\n", + " \"\"\"\r\n", + " pass\r\n", + "\r\n", + " def step_wait(self):\r\n", + " \"\"\"\r\n", + " Wait for the step taken with step_async().\r\n", + " Returns (obs, rews, dones, infos):\r\n", + " - obs: an array of observations, or a tuple of\r\n", + " arrays of observations.\r\n", + " - rews: an array of rewards\r\n", + " - dones: an array of \"episode done\" booleans\r\n", + " - infos: a sequence of info objects\r\n", + " \"\"\"\r\n", + " pass\r\n", + "\r\n", + " def close(self):\r\n", + " \"\"\"\r\n", + " Clean up the environments' resources.\r\n", + " \"\"\"\r\n", + " pass\r\n", + "\r\n", + " def step(self, actions):\r\n", + " self.step_async(actions)\r\n", + " return self.step_wait()\r\n", + "\r\n", + " \r\n", + "class CloudpickleWrapper(object):\r\n", + " \"\"\"\r\n", + " Uses cloudpickle to serialize contents (otherwise multiprocessing tries to use pickle)\r\n", + " \"\"\"\r\n", + " def __init__(self, x):\r\n", + " self.x = x\r\n", + " def __getstate__(self):\r\n", + " import cloudpickle\r\n", + " return cloudpickle.dumps(self.x)\r\n", + " def __setstate__(self, ob):\r\n", + " import pickle\r\n", + " self.x = pickle.loads(ob)\r\n", + "\r\n", + " \r\n", + "class SubprocVecEnv(VecEnv):\r\n", + " def __init__(self, env_fns, spaces=None):\r\n", + " \"\"\"\r\n", + " envs: list of gym environments to run in subprocesses\r\n", + " \"\"\"\r\n", + " self.waiting = False\r\n", + " self.closed = False\r\n", + " nenvs = len(env_fns)\r\n", + " self.nenvs = nenvs\r\n", + " self.remotes, self.work_remotes = zip(*[Pipe() for _ in range(nenvs)])\r\n", + " self.ps = [Process(target=worker, args=(work_remote, remote, CloudpickleWrapper(env_fn)))\r\n", + " for (work_remote, remote, env_fn) in zip(self.work_remotes, self.remotes, env_fns)]\r\n", + " for p in self.ps:\r\n", + " p.daemon = True # if the main process crashes, we should not cause things to hang\r\n", + " p.start()\r\n", + " for remote in self.work_remotes:\r\n", + " remote.close()\r\n", + "\r\n", + " self.remotes[0].send(('get_spaces', None))\r\n", + " observation_space, action_space = self.remotes[0].recv()\r\n", + " VecEnv.__init__(self, len(env_fns), observation_space, action_space)\r\n", + "\r\n", + " def step_async(self, actions):\r\n", + " for remote, action in zip(self.remotes, actions):\r\n", + " remote.send(('step', action))\r\n", + " self.waiting = True\r\n", + "\r\n", + " def step_wait(self):\r\n", + " results = [remote.recv() for remote in self.remotes]\r\n", + " self.waiting = False\r\n", + " obs, rews, dones, infos = zip(*results)\r\n", + " return np.stack(obs), np.stack(rews), np.stack(dones), infos\r\n", + "\r\n", + " def reset(self):\r\n", + " for remote in self.remotes:\r\n", + " remote.send(('reset', None))\r\n", + " return np.stack([remote.recv() for remote in self.remotes])\r\n", + "\r\n", + " def reset_task(self):\r\n", + " for remote in self.remotes:\r\n", + " remote.send(('reset_task', None))\r\n", + " return np.stack([remote.recv() for remote in self.remotes])\r\n", + "\r\n", + " def close(self):\r\n", + " if self.closed:\r\n", + " return\r\n", + " if self.waiting:\r\n", + " for remote in self.remotes: \r\n", + " remote.recv()\r\n", + " for remote in self.remotes:\r\n", + " remote.send(('close', None))\r\n", + " for p in self.ps:\r\n", + " p.join()\r\n", + " self.closed = True\r\n", + " \r\n", + " def __len__(self):\r\n", + " return self.nenvs" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "writer = LogWriter(logdir=\"./log\") \r\n", + "\r\n", + "#from multiprocessing_env import SubprocVecEnv\r\n", + "\r\n", + "num_envs = 8\r\n", + "env_name = \"CartPole-v0\"\r\n", + "\r\n", + "def make_env():\r\n", + " def _thunk():\r\n", + " env = gym.make(env_name)\r\n", + " return env\r\n", + " return _thunk\r\n", + "\r\n", + "plt.ion()\r\n", + "envs = [make_env() for i in range(num_envs)]\r\n", + "envs = SubprocVecEnv(envs) # 8 env\r\n", + "\r\n", + "env = gym.make(env_name) # a single env\r\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "### **定义网络结构并开始训练**\n", + "self.critic部分定义的是“评论家”，self.actor部分定义的是“演员”。“评论家”网络观察输入并“打分”，“演员”网络接收输入并给出行动的类别分布，这里用到了API——paddle.distribution.Categorical，后续调用sample(shape)生成指定维度的样本、调用entropy()返回类别分布的信息熵、调用log_prob(value)返回所选择类别的对数概率，其他用法可以查看飞桨API文档。\n", + "\n", + "![](https://ai-studio-static-online.cdn.bcebos.com/5ecb0c2d202d43e99cdbe378bc7ebb0fbb438e1e9b1b41469cf633e73c5e688b)\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "\r\n", + "class ActorCritic(nn.Layer):\r\n", + " def __init__(self, num_inputs, num_outputs, hidden_size, std=0.0):\r\n", + " super(ActorCritic, self).__init__()\r\n", + " nn.initializer.set_global_initializer(nn.initializer.XavierNormal(), nn.initializer.Constant(value=0.))\r\n", + " \r\n", + " self.critic = nn.Sequential(\r\n", + " nn.Linear(num_inputs, hidden_size),\r\n", + " nn.ReLU(),\r\n", + " nn.Linear(hidden_size, 1)\r\n", + " )\r\n", + " \r\n", + " self.actor = nn.Sequential(\r\n", + " nn.Linear(num_inputs, hidden_size),\r\n", + " nn.ReLU(),\r\n", + " nn.Linear(hidden_size, num_outputs),\r\n", + " nn.Softmax(axis=1),\r\n", + " )\r\n", + " \r\n", + " def forward(self, x):\r\n", + " value = self.critic(x)\r\n", + " probs = self.actor(x)\r\n", + " dist = Categorical(probs)\r\n", + " return dist, value\r\n", + "\r\n", + "\r\n", + "def test_env(vis=False):\r\n", + " state = env.reset()\r\n", + " if vis: env.render()\r\n", + " done = False\r\n", + " total_reward = 0\r\n", + " while not done:\r\n", + " state = paddle.to_tensor(state,dtype=\"float32\").unsqueeze(0)\r\n", + " dist, _ = model(state)\r\n", + " next_state, reward, done, _ = env.step(dist.sample([1]).cpu().numpy()[0][0]) \r\n", + " state = next_state\r\n", + " if vis: env.render()\r\n", + " total_reward += reward\r\n", + " return total_reward\r\n", + "\r\n", + "\r\n", + "def compute_returns(next_value, rewards, masks, gamma=0.99):\r\n", + " R = next_value\r\n", + " returns = []\r\n", + " for step in reversed(range(len(rewards))):\r\n", + " R = rewards[step] + gamma * R * masks[step]\r\n", + " returns.insert(0, R)\r\n", + " return returns\r\n", + "\r\n", + "def plot(frame_idx, rewards):\r\n", + " plt.plot(rewards,'b-')\r\n", + " plt.title('frame %s. reward: %s' % (frame_idx, rewards[-1]))\r\n", + " plt.pause(0.0001)\r\n", + "\r\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "### **实例化模型和定义优化器**\n", + "hidden_size是网络的隐藏层的“神经元”数目，lr是优化器的学习率，咱使用经典的Adam优化器。num_steps是收集轨迹的步数，值设置的越大，更新网络前收集的轨迹越长。" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "\r\n", + "num_inputs = envs.observation_space.shape[0]\r\n", + "num_outputs = envs.action_space.n\r\n", + "\r\n", + "#Hyper params:\r\n", + "hidden_size = 256\r\n", + "lr = 1e-3\r\n", + "num_steps = 8\r\n", + "\r\n", + "model = ActorCritic(num_inputs, num_outputs, hidden_size)\r\n", + "optimizer = optim.Adam(parameters=model.parameters(),learning_rate=lr)\r\n", + "save_model_path = \"models/A2C_model.pdparams\"\r\n", + "if os.path.exists(save_model_path):\r\n", + " model_state_dict = paddle.load(save_model_path)\r\n", + " model.set_state_dict(model_state_dict )\r\n", + " print(' Model loaded')\r\n", + "\r\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "## **四、开始循环训练过程：**\n", + "收集经验—>计算损失—>反向传播\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "collapsed": false + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "frame 19300. reward: 200.0\n", + "frame 19400. reward: 200.0\n", + "frame 19500. reward: 200.0\n", + "frame 19600. reward: 200.0\n", + "frame 19700. reward: 200.0\n", + "frame 19800. reward: 200.0\n", + "frame 19900. reward: 200.0\n", + "frame 20000. reward: 200.0\n" + ] + } + ], + "source": [ + "# 首先定义最大的训练帧数，并行的环境envs每执行一步step()算一帧。如果按照前面定义的\r\n", + "# 是8组环境并行，那么envs就需要输入8组动作，同时会输出8组回报（reward）、下一\r\n", + "# 观测状态（next_state）。\r\n", + "\r\n", + "max_frames = 20000\r\n", + "frame_idx = 0\r\n", + "test_rewards = []\r\n", + "\r\n", + "\r\n", + "state = envs.reset()\r\n", + "\r\n", + "while frame_idx < max_frames:\r\n", + "\r\n", + " log_probs = []\r\n", + " values = []\r\n", + " rewards = []\r\n", + " masks = []\r\n", + " entropy = 0\r\n", + "\r\n", + " # rollout trajectory\r\n", + " # 现在模型展开num_steps步的轨迹：模型会根据观测状态返回动作的分布、状态价值，然后\r\n", + " # 根据动作分布采样动作，接着环境step一步进入到下一个状态，并返回reward。\r\n", + " for _ in range(num_steps):\r\n", + " state = paddle.to_tensor(state,dtype=\"float32\")\r\n", + " dist, value = model(state)\r\n", + "\r\n", + " action = dist.sample([1]).squeeze(0)\r\n", + " next_state, reward, done, _ = envs.step(action.cpu().numpy())\r\n", + "\r\n", + " log_prob = dist.log_prob(action)\r\n", + " entropy += dist.entropy().mean()\r\n", + " \r\n", + " log_probs.append(log_prob)\r\n", + " values.append(value)\r\n", + " rewards.append(paddle.to_tensor(reward,dtype=\"float32\").unsqueeze(1))\r\n", + " masks.append(paddle.to_tensor(1 - done).unsqueeze(1))\r\n", + " \r\n", + " state = next_state\r\n", + " frame_idx += 1\r\n", + " \r\n", + " Plot = False\r\n", + " # 程序每隔100帧会进行一次评估，评估的方式是运行2次test_env()并计算返回的\r\n", + " # total_reward的均值，这里用VisualDL记录它，文章的最后会展示模型运行效果。\r\n", + " if frame_idx % 100 == 0:\r\n", + " test_rewards.append(np.mean([test_env() for _ in range(2)]))\r\n", + " writer.add_scalar(\"test_rewards\", value=test_rewards[-1], step=frame_idx) \r\n", + " if Plot:\r\n", + " plot(frame_idx, test_rewards)\r\n", + " else:\r\n", + " print('frame {}. reward: {}'.format(frame_idx, test_rewards[-1]))\r\n", + "\r\n", + " # 程序会记录展开轨迹的动作对数似然概率log_probs、模型估计价值values、回报rewards等，\r\n", + " # 并计算优势值advantage 。由于是多环境并行，可以用paddle.concat将这些值分别拼接起来，\r\n", + " # 随后计算出演员网络的损失actor_loss、评论家网络的损失critic_loss，在最终loss中有一项\r\n", + " # 是动作分布熵的均值，希望能增大网络的探索能力。 \r\n", + " next_state = paddle.to_tensor(next_state,dtype=\"float32\")\r\n", + " _, next_value = model(next_state)\r\n", + " returns = compute_returns(next_value, rewards, masks)\r\n", + " \r\n", + " log_probs = paddle.concat(log_probs)\r\n", + " returns = paddle.concat(returns).detach()\r\n", + " values = paddle.concat(values)\r\n", + "\r\n", + " advantage = returns - values\r\n", + "\r\n", + " actor_loss = -(log_probs * advantage.detach()).mean()\r\n", + " critic_loss = advantage.pow(2).mean()\r\n", + "\r\n", + " loss = actor_loss + 0.5 * critic_loss - 0.01 * entropy\r\n", + " # 用VisualDL记录训练的actor_loss、critic_loss以及合并后的loss。然后再反向传播，优化神\r\n", + " # 经网络的参数，开始下一轮的训练循环。\r\n", + " writer.add_scalar(\"actor_loss\", value=actor_loss, step=frame_idx)\r\n", + " writer.add_scalar(\"critic_loss\", value=critic_loss, step=frame_idx)\r\n", + " writer.add_scalar(\"loss\", value=loss, step=frame_idx)\r\n", + " ##动态学习率，每隔2000帧缩放一次\r\n", + " if frame_idx % 2000 ==0:\r\n", + " lr = 0.92*lr\r\n", + " optimizer.set_lr(lr) \r\n", + "\r\n", + " optimizer.clear_grad()\r\n", + " loss.backward()\r\n", + " optimizer.step()\r\n", + "\r\n", + " \r\n", + "if not os.path.exists(os.path.dirname(save_model_path)):\r\n", + " os.makedirs(os.path.dirname(save_model_path))\r\n", + "# paddle.save(model.state_dict(), save_model_path)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "## **五、VisualDL里展示模型运行的效果**\n", + "在gym的CartPole环境（env）里面，小车需要左右移动来保持杆子竖直。左移或者右移小车之后，env会返回一个“+1”的reward，如果杠子倾角过大或者小车超范围游戏就结束了。其中，在CartPole-v0环境里reward达到200也会结束游戏。" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "![](https://ai-studio-static-online.cdn.bcebos.com/d1f795ede3fe434eb4b0e12f5e985a8c76072df13c944188bd20040f0665176c)\n", + "\n", + "\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": false + }, + "source": [ + "\n", + "\n", + "## **六、总结和建议**\n", + "* 深度强化学习中，很多基础算法都是单线程的，也就是一个 agent 去跟环境交互产生经验。基础版 Actor-Critic ，由于环境是固定不变的，agent 的动作又是连续的，这样收集到的经验就有很强的时序关联，而且在有限的时间内也只能探索到部分状态和动作空间。\n", + "\n", + "* 为了打破经验之间的耦合，可以采用Experiencre Replay的方法，让 agent 能够在后续的训练中访问到以前的历史经验，这就是 DQN 和 DDPG 这类基于值的（DDPG虽然也属于 Actor-Critic 架构，但本质上是 DQN 在连续空间中的扩展）算法所采用的方式。而对于基于策略类的算法，agent 收集的经验都是以 episode为单位的，跑完一个episode 后经验就要丢掉，更好的方式是采用多线程的并行架构，这样既能解决前面的问题，又能高效利用计算资源，提升训练效率。\n", + "\n", + "* Advantage Actor-Critic（A2C） 算法引入了并行架构，各个 worker 都会独立的跟自己的环境去交互，得到独立的采样经验，而这些经验之间也是相互独立的，这样就打破了经验之间的耦合，起到跟 Experiencre Replay 相当的效果。因此通常 A2C和A3C 是不需要使用 Replay Buffer 的，这种结构本身就可以替代了。\n", + "\n", + "![](https://ai-studio-static-online.cdn.bcebos.com/88b967da1ba74e049b3ff28dd9083d1e527ba734dc064a798374f99199f84086)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "PaddlePaddle 2.0.0b0 (Python 3.5)", + "language": "python", + "name": "py35-paddle1.2.0" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.4" + } + }, + "nbformat": 4, + "nbformat_minor": 1 +}