Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Performance] Accelerate GAE #1142

Merged
merged 18 commits into from
May 10, 2023
Merged

[Performance] Accelerate GAE #1142

merged 18 commits into from
May 10, 2023

Commits on Apr 6, 2023

  1. init

    vmoens committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    b40e9a9 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2023

  1. Configuration menu
    Copy the full SHA
    1669802 View commit details
    Browse the repository at this point in the history
  2. init

    vmoens committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    fe31219 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2023

  1. Configuration menu
    Copy the full SHA
    658380e View commit details
    Browse the repository at this point in the history
  2. Optimize vec_generalized_advantage_estimate

    An optimized vecotrized version for the generalized advantage estimation is used in case
    gamma and lambda are scalars.
    
    When handling consecutive trajectories of the form
    ```
    reward = [r00, r01, r02, r03, r10, r11]
    done = [False, False, False, True, False, False]
    ```
    , `vec_generalized_advantage_estimate`
    needs to build a giant famma tensor of size [Batch, T, T] with a decayed gamma
    tensor that suits each trajectory. Thus it needs to allocate a big tensor `[B,
    T, T]` and do a heavy matrix multiplication.  In case gamma and lambda are
    scalars, this can be optimized by building a single tensor of the form
    ```
    r_transformed = [[r00, r01, r02, r03]
    		 [r10, r11, 0, 0]]
    ```
    and applying the gamma filter `[r00 + gamma r01 + gamma ** 2 r02 + ..., ro1 + gamma r02 + gamma ** 2 r03 + ...,]`
    to calculate the GAE.
    vmoens authored and Blonck committed May 9, 2023
    Configuration menu
    Copy the full SHA
    5a585d0 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    70bd48d View commit details
    Browse the repository at this point in the history
  4. Refactor code:

    * move helper methods to util
    * reuse existing helper methods
    * remove wip file
    Blonck committed May 9, 2023
    Configuration menu
    Copy the full SHA
    268056e View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    0c6dc55 View commit details
    Browse the repository at this point in the history
  6. Add hard check that _fast_vec_gae is faster than original implementation

    In case gamma and lmbda are scalars, `fast_vec_gae` should be always faster than
    `vec_generalized_advantage_estimate` if len(T) is large enough.
    Blonck committed May 9, 2023
    Configuration menu
    Copy the full SHA
    245e68f View commit details
    Browse the repository at this point in the history
  7. Revert "Add hard check that _fast_vec_gae is faster than original imp…

    …lementation"
    
    This reverts commit 245e68f.
    Blonck committed May 9, 2023
    Configuration menu
    Copy the full SHA
    cd65976 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    9c32f01 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    41945f5 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    88b9788 View commit details
    Browse the repository at this point in the history
  11. Improve perfomance of _inv_pad_sequence

    in case there is only one split, _inv_pad_sequence can skip its calculation.
    Blonck committed May 9, 2023
    Configuration menu
    Copy the full SHA
    291386a View commit details
    Browse the repository at this point in the history

Commits on May 10, 2023

  1. Configuration menu
    Copy the full SHA
    6d5da1b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4a634c6 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    914d44b View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    a921333 View commit details
    Browse the repository at this point in the history