DDPG.md (#93)

Author: XiangDuojie

docs/source/_static/figures/pseucodes/pseucode-DDPG.png

@@ -0,0 +1,185 @@
+# Deep Deterministic Policy Gradient (DDPG)
+
+**Paper Link:** [**https://arxiv.org/abs/1509.02971**](https://arxiv.org/abs/1509.02971).
+
+Deep Deterministic Policy Gradient (DDPG) is a model-free algorithm in Deep Reinforcement Learning (DRL) that combines elements of policy gradient and deep neural networks. It was developed by Timothy P. Lillicrap et al. in 2015. DDPG has been widely applied in continuous control tasks, achieving notable results in scenarios such as robotic control and simulated environments.
+
+This table lists some general features about DDPG algorithm:
+
+| Features of DDPG   | Values | Description                                              |
+|-------------------|--------|----------------------------------------------------------|
+| On-policy         | ❌      | The evaluate policy is the same as the target policy.    |
+| Off-policy        | ✅      | The evaluate policy is different from the target policy. | 
+| Model-free        | ✅      | No need to prepare an environment dynamics model.        | 
+| Model-based       | ❌      | Need an environment model to train the policy.           | 
+| Discrete Action   | ✅      | Deal with discrete action space.                         |   
+| Continuous Action | ✅      | Deal with continuous action space.                       |    
+
+## Actor-Critic Framework
+
+DDPG builds upon the Actor-Critic framework. The actor network is responsible for generating actions based on the current state. It outputs a deterministic action directly, parameterized by a deep neural network. The critic network, on the other hand, estimates the Q-value of a state-action pair. The critic network takes both the state and the action as inputs and outputs the estimated Q-value, which indicates the long-term expected reward of taking a particular action in a given state.
+
+
+## Policy Gradient
+
+The actor in DDPG is updated using the policy gradient method. The policy gradient is calculated by taking the gradient of the expected return with respect to the actor's parameters. The goal is to find the policy that maximizes the expected return. The update rule for the actor network is based on the gradient of the Q-value with respect to the actor's parameters, which is calculated using the chain rule of calculus.
+The update for the actor network parameters $\theta^\mu$ is given by:
+
+$$
+\nabla_{\theta^\mu} J \approx \frac{1}{N} \sum_{i=1}^{N} \nabla_a Q(s, a \mid \theta^Q) \bigg|_{s=s^i, a=\mu(s^i \mid \theta^\mu)} \nabla_{\theta^\mu} \mu(s \mid \theta^\mu) \bigg|_{s^i}
+$$
+
+where $J$ is the expected return, $N$ is the number of samples in a mini-batch,$\theta^Q$  are the parameters of the critic network, and $\mu(s \mid \theta^\mu)$  is the actor network.
+
+## Critic Network Update
+The critic network in DDPG is updated using the temporal difference (TD) error. The TD error is calculated as the difference between the target Q-value and the predicted Q-value. The target Q-value is calculated using the Bellman equation, similar to Q-learning. The update rule for the critic network is based on minimizing the mean-squared error (MSE) between the predicted Q-value and the target Q-value.
+The target Q-value $y^i$ for a sample $(s^i, a^i, r^i, s^{i+1})$ is given by:
+
+$$
+y^i = r^i + \gamma Q'(s^{i+1}, \mu'(s^{i+1} \mid \theta^{\mu'}) \mid \theta^{Q'})
+$$
+
+where $\mu'$ and $\theta^{Q'}$ are the target actor network and the parameters of target critic network  respectively, $Q'$ is the target critic network and $\gamma$ is the discount factor.
+The critic network parameters $\theta^Q$ are updated by minimizing the loss function:
+
+$$
+L = \frac{1}{N} \sum_{i=1}^{N} (y^i - Q(s^i, a^i \mid \theta^Q))^2
+$$
+
+## Target Networks and Experience Replay
+Similar to DQN, DDPG uses target networks to stabilize the learning process. Separate target actor and target critic networks are maintained. The parameters of the target networks are updated slowly from the main networks using a soft update rule:
+
+$$
+\theta' \leftarrow \tau \theta + (1 - \tau) \theta'
+$$
+
+where $\tau$ is a small positive number, typically close to 1, controlling the rate of update.
+DDPG also employs experience replay. An experience replay buffer stores the agent's experiences $(s^i, a^i, r^i, s^{i+1})$ .Mini-batches of experiences are randomly sampled from the buffer to train the actor and critic networks. This helps to break the correlation between consecutive samples and improves the stability and generalization of the learning algorithm.
+
+## Exploration
+DDPG uses an exploration strategy to encourage the agent to explore the environment. Typically, a noise process is added to the actions generated by the actor network:
+
+$$
+\mu'(s_t) = \mu(s_t \mid \theta^{\mu}_t) + \mathcal{N}
+$$
+
+This noise $\mathcal{N}$ can be, for example, Gaussian noise. The exploration noise helps the agent to explore different parts of the action space and discover better policies. As the training progresses, the amount of exploration noise can be gradually reduced.
+
+Strengths of DDPG:
+- Can handle continuous action spaces, making it suitable for a wide range of control tasks in robotics and other fields.
+- Stabilizes the learning process through the use of target networks and experience replay, enabling more reliable training.
+- Demonstrated effectiveness in various continuous control scenarios, achieving good performance in tasks like robotic manipulation and locomotion.
+
+## Algorithm
+
+The full algorithm for training DDPG is presented in Algorithm 1:
+
+```{eval-rst}
+.. image:: ./../../../../_static/figures/pseucodes/pseucode-DDPG.png
+    :width: 80%
+    :align: center
+```
+
+## Run DDPG in XuanCe
+
+Before running DDPG in XuanCe, you need to prepare a conda environment and install ``xuance`` following 
+the [**installation steps**](./../../../usage/installation.rst#install-xuance).
+
+### Run Build-in Demos
+
+After completing the installation, you can open a Python console and run DDPG directly using the following commands:
+
+```python3
+import xuance
+runner = xuance.get_runner(method='ddpg',
+                           env='classic_control',  # Choices: claasi_control, box2d, atari.
+                           env_id='CartPole-v1',  # Choices: CartPole-v1, LunarLander-v2, ALE/Breakout-v5, etc.
+                           is_test=False)
+runner.run()  # Or runner.benchmark()
+```
+
+### Run With Self-defined Configs
+
+If you want to run DDPG with different configurations, you can build a new ``.yaml`` file, e.g., ``my_config.yaml``.
+Then, run the DDPG by the following code block:
+
+```python3
+import xuance as xp
+runner = xp.get_runner(method='ddpg',
+                       env='classic_control',  # Choices: claasi_control, box2d, atari.
+                       env_id='CartPole-v1',  # Choices: CartPole-v1, LunarLander-v2, ALE/Breakout-v5, etc.
+                       config_path="my_config.yaml",  # The path of my_config.yaml file should be correct.
+                       is_test=False)
+runner.run()  # Or runner.benchmark()
+```
+
+To learn more about the configurations, please visit the 
+[**tutorial of configs**](./../../configs/configuration_examples.rst).
+
+### Run With Customized Environment
+
+If you would like to run XuanCe's DDPG in your own environment that was not included in XuanCe, 
+you need to define the new environment following the steps in 
+[**New Environment Tutorial**](./../../../usage/new_envs.rst).
+Then, [**prepapre the configuration file**](./../../../usage/new_envs.rst#step-2-create-the-config-file-and-read-the-configurations) 
+ ``ddpg_myenv.yaml``.
+
+After that, you can run DDPG in your own environment with the following code:
+
+```python3
+import argparse
+from xuance.common import get_configs
+from xuance.environment import REGISTRY_ENV
+from xuance.environment import make_envs
+from xuance.torch.agents import DDPG_Agent
+
+configs_dict = get_configs(file_dir="ddpg_myenv.yaml")
+configs = argparse.Namespace(**configs_dict)
+REGISTRY_ENV[configs.env_name] = MyNewEnv
+
+envs = make_envs(configs)  # Make parallel environments.
+Agent = DDPG_Agent(config=configs, envs=envs)  # Create a DDPG agent from XuanCe.
+Agent.train(configs.running_steps // configs.parallels)  # Train the model for numerous steps.
+Agent.save_model("final_train_model.pth")  # Save the model to model_dir.
+Agent.finish()  # Finish the training.
+```
+
+## Citation
+
+```{code-block} bash
+@article{lillicrap2015continuous,
+  title={Continuous control with deep reinforcement learning},
+  author={Lillicrap, Timothy P and Hunt, Jonathan J and Pritzel, Alexander and Heess, Nicolas and Erez, Tom and Tassa, Yuval and Silver, David and Wierstra, Daan},
+  journal={arXiv preprint arXiv:1509.02971},
+  year={2015}
+}
+```
+
+## APIs
+
+### PyTorch
+
+```{eval-rst}
+.. automodule:: xuance.torch.agents.policy_gradient.ddpg_agent
+    :members:
+    :undoc-members:
+    :show-inheritance:
+```
+
+### TensorFlow2
+
+```{eval-rst}
+.. automodule:: xuance.tensorflow.agents.policy_gradient.ddpg_agent
+    :members:
+    :undoc-members:
+    :show-inheritance:
+```
+
+### MindSpore
+
+```{eval-rst}
+.. automodule:: xuance.mindspore.agents.policy_gradient.ddpg_agent
+    :members:
+    :undoc-members:
+    :show-inheritance:
+```