Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add dueling in rainbow iqn #137

Closed
wants to merge 1 commit into from
Closed

Add dueling in rainbow iqn #137

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ We are warmly welcoming external contributors! :)
7. [Prioritized Experience Replay (PER with DDPG)](https://github.com/medipixel/rl_algorithms/tree/master/algorithms/per)
8. [From Demonstrations (DDPGfD, SACfD, DQfD)](https://github.com/medipixel/rl_algorithms/tree/master/algorithms/fd)
9. [Rainbow DQN](https://github.com/medipixel/rl_algorithms/tree/master/algorithms/dqn)
10. [Rainbow IQN (without DuelingNet)](https://github.com/medipixel/rl_algorithms/tree/master/algorithms/dqn)
10. [Rainbow IQN](https://github.com/medipixel/rl_algorithms/tree/master/algorithms/dqn)

## Getting started
We have tested each algorithm on some of the following environments.
Expand Down
28 changes: 27 additions & 1 deletion algorithms/dqn/networks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,7 @@ def __init__(
hidden_sizes=hidden_sizes,
hidden_activation=hidden_activation,
linear_layer=linear_layer,
use_output_layer=False,
init_fn=init_fn,
)

Expand All @@ -208,6 +209,17 @@ def __init__(
)
self.quantile_fc_layer = init_fn(self.quantile_fc_layer)

# set advantage layer
hidden_size = hidden_sizes[-1]
self.adv_hidden_layer = self.linear_layer(hidden_size, hidden_size)
self.adv_layer = self.linear_layer(hidden_size, output_size)
self.adv_layer = init_fn(self.adv_layer)

# set value layer
self.val_hidden_layer = self.linear_layer(hidden_size, hidden_size)
self.val_layer = self.linear_layer(hidden_size, 1)
self.val_layer = init_fn(self.val_layer)

def forward_(
self, state: torch.Tensor, n_tau_samples: int = None
) -> Tuple[torch.Tensor, torch.Tensor]:
Expand All @@ -234,7 +246,21 @@ def forward_(
# Hadamard product
quantile_net = state_tiled * quantile_net

quantile_values = super(IQNMLP, self).forward(quantile_net)
# advantage
last_common_hidden = super(IQNMLP, self).forward(quantile_net)
adv_hidden = self.hidden_activation(self.adv_hidden_layer(last_common_hidden))
advantages = self.adv_layer(adv_hidden)
advantages = advantages.view(-1, self.output_size, n_tau_samples)

# value
val_hidden = self.hidden_activation(self.val_hidden_layer(last_common_hidden))
values = self.val_layer(val_hidden)
values = values.view(-1, 1, n_tau_samples)

# dueling
advantages_mean = advantages.mean(dim=1, keepdim=True)
quantile_values = values + advantages - advantages_mean
quantile_values = quantile_values.view(-1, self.output_size)

return quantile_values, quantiles

Expand Down