Update priority

[tphanson]

Since I followed PER which is described in this paper, I believe that there must exist a way to update priority every time we compute TD error on every minibatch. However, I carefully looked at all your examples but it seems no function like this existing.

[acassirer]

Hey Tu,

I was surprised to see that we indeed do no have any examples of how to update priorities in the example. The method you are looking for is Client.mutate_priorities. There is also an in-graph version in TFClient.update_priorities but you probably want to focus on the former.

I'll keep this issue until the examples have been updated but an example of how to update the priorities could look something like this:

# Batches 2 sequences together.
# Shapes of items is now [2, 3, 10, 10].
dataset = dataset.batch(2)

for sample in dataset.take(1):
  # Results in the following format.
  print(sample.info.key)          # ([2, 3], uint64)
  print(sample.info.probability)  # ([2, 3], float64)
  
  observation, action = sample.data
  print(observation)              # ([2, 3, 10, 10], uint8)
  print(action)                   # ([2, 3, 2], float32)

  # Set the priority of all sampled items to 0.5.
  client.mutate_priorities('my_table', {
      int(key[0]): 0.5            # key[0] => uint64 scalar.
      for key in sample.info.key  # for ([3], uint64) in ([2, 3], uint64)
  })

[sontuphan]

Thank you @acassirer. That's very helpful to me.

[sontuphan]

Thank you @acassirer. That's very helpful to me.

Just notice that I used another github account to reply you :) It's still me who posted the question. Thanks again.

[ebrevdo]

Leaving this open until Albin updates examples.

[abhishekbisht1429]

Hi @acassirer, I am using ReverbReplayBuffer from tf_agents and obtained the dataset using its as_dataset function. However sample.info gives me 'tuple' object has no attribute 'info' error. Could you please tell me how can I obtain keys to update priorities in tf_agents.replay_buffers.ReverbReplayBuffer ?

[acassirer]

Hey @abhishekbisht1429,

I can't speak for the transformations that tf_agents are applying to the output of the Reverb dataset but my guess is that they lost the data container types along the way. If this is the case then I would expect the priorities to be found in sample[0][0] (as this would be the same as doing sample.info.key on the named tuple).

[abhishekbisht1429]

Thanks @acassirer for your comment.

After some search and trials I figured it out, leaving this comment here in case someone stumbles upon this problem. Each item in the dataset returned by ReverbReplayBuffer.as_dataset() is a 2-tuple, the first element of which is the actual data and the second element is a named tuple containing auxiliary information about the data. To access the keys corresponding to data item one can use sample[1].key where sample is the item contained the Dataset object returned by ReverbReplayBuffer.as_dataset() function. It will be a tensor of the same shape as the data. So one might need to reshape it in order to use it with ReverbReplayBuffer.update_priorities.