[Feature] pickle-free RB checkpointing

[vmoens]

Description

Allows for a pickle-free replay buffer checkpointing, where tensors are saved under mmap tensors and metadata in json files.
Not compatible with ListStorage.
Depends on pytorch/tensordict#582

Example:

import os
import tempfile

import tqdm
from torchrl.data import LazyMemmapStorage, TensorDictReplayBuffer
from torchrl.data.replay_buffers.samplers import PrioritizedSampler, RandomSampler
import torch
from tensordict import TensorDict

def print_directory_tree(path, indent=""):
    """
    Print the directory tree starting from the specified path.

    Parameters:
    - path (str): The path of the directory to print.
    - indent (str): The current indentation level for formatting.
    """
    if os.path.isdir(path):
        print(indent + os.path.basename(path) + "/")
        indent += "    "
        for item in os.listdir(path):
            print_directory_tree(os.path.join(path, item), indent)
    else:
        print(indent + os.path.basename(path))

S = 1_000_000
sampler = PrioritizedSampler(S, 1.1, 1.0)
# sampler = RandomSampler()
storage = LazyMemmapStorage(S)
rb = TensorDictReplayBuffer(storage=storage, sampler=sampler)

for _ in tqdm.tqdm(range(100)):
    td = TensorDict({"obs": torch.randn(100, 3, 4), "next": {"obs": torch.randn(100, 3, 4)}, "td_error": torch.rand(100)}, [100])
    rb.extend(td)
    sample = rb.sample(32)
    rb.update_tensordict_priority(sample)

with tempfile.TemporaryDirectory() as tmpdir:
    rb.dumps(tmpdir)
    print_directory_tree(tmpdir)

    sampler = PrioritizedSampler(S, 1.1, 1.0)
    # sampler = RandomSampler()
    storage = LazyMemmapStorage(S)
    rb_load = TensorDictReplayBuffer(storage=storage, sampler=sampler)
    rb_load.loads(tmpdir)

    assert len(rb) == len(rb_load)

    torch.manual_seed(0)
    s1 = rb.sample(32)
    print(s1)

    torch.manual_seed(0)
    s2 = rb_load.sample(32)
    print(s2)

    print(s1["index"])
    print(s2["index"])

The saved folder is printed as

tmpbz5bng25/
    storage/
        index.memmap
        storage_metadata.json
        _data/
            next/
                obs.memmap
                meta.json
            obs.memmap
            td_error.memmap
            meta.json
        meta.json
    writer/
        metadata.json
    sampler/
        sampler_metadata.json
        sumtree.memmap
        mintree.memmap
    buffer_metadata.json

The structure of the samples match and a random sample with the same seed provides the exact same samples with indices

tensor([5002, 7710,  899, 1339, 3105, 6370, 4950, 8955, 4614, 6349, 3524, 4043,
         222, 1696, 2961, 5216, 6984, 8011, 1621, 2852, 6836, 9145, 4000, 8738,
        4239, 5550, 9528,  363, 1854, 3767, 3084, 9323])

TODO:

• Make sure the checkpointing can occur twice at the same location
• docstrings

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1733

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (5 Unrelated Failures)

As of commit 4fc7c31 with merge base 841f8d9 ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Build M1 Wheels / pytorch/rl / wheel-py3_8-cpu (gh)
• RLHF Tests on Linux / unittests (3.9, 12.1) / linux-job (gh)
• Unit-tests on Windows CPU / unittests / windows-job (gh)

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Habitat Tests on Linux / tests (3.9, 11.6) / linux-job (gh)
• Unit-tests on Windows GPU / unittests / windows-job (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $\large\color{#35bf28}3$. Worsened: $\large\color{#d91a1a}3$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	62.9151ms	62.2233ms	16.0712 Ops/s	15.2832 Ops/s	$\textbf{\color{#35bf28}+5.16\%}$
test_sync	40.6082ms	34.5239ms	28.9655 Ops/s	25.0034 Ops/s	$\textbf{\color{#35bf28}+15.85\%}$
test_async	68.1264ms	33.0288ms	30.2766 Ops/s	30.8303 Ops/s	$\color{#d91a1a}-1.80\%$
test_simple	0.4861s	0.4328s	2.3108 Ops/s	2.3175 Ops/s	$\color{#d91a1a}-0.29\%$
test_transformed	0.6443s	0.5980s	1.6724 Ops/s	1.6697 Ops/s	$\color{#35bf28}+0.16\%$
test_serial	1.3708s	1.3254s	0.7545 Ops/s	0.7447 Ops/s	$\color{#35bf28}+1.32\%$
test_parallel	1.3470s	1.2842s	0.7787 Ops/s	0.7800 Ops/s	$\color{#d91a1a}-0.17\%$
test_step_mdp_speed[True-True-True-True-True]	0.1627ms	22.6787μs	44.0942 KOps/s	44.4613 KOps/s	$\color{#d91a1a}-0.83\%$
test_step_mdp_speed[True-True-True-True-False]	58.3160μs	13.6311μs	73.3617 KOps/s	73.2030 KOps/s	$\color{#35bf28}+0.22\%$
test_step_mdp_speed[True-True-True-False-True]	44.6430μs	13.7151μs	72.9125 KOps/s	73.1714 KOps/s	$\color{#d91a1a}-0.35\%$
test_step_mdp_speed[True-True-True-False-False]	30.4770μs	8.2356μs	121.4243 KOps/s	123.3177 KOps/s	$\color{#d91a1a}-1.54\%$
test_step_mdp_speed[True-True-False-True-True]	75.4130μs	23.9349μs	41.7801 KOps/s	41.8433 KOps/s	$\color{#d91a1a}-0.15\%$
test_step_mdp_speed[True-True-False-True-False]	36.4680μs	14.8831μs	67.1902 KOps/s	66.8146 KOps/s	$\color{#35bf28}+0.56\%$
test_step_mdp_speed[True-True-False-False-True]	51.6060μs	15.0438μs	66.4725 KOps/s	66.7706 KOps/s	$\color{#d91a1a}-0.45\%$
test_step_mdp_speed[True-True-False-False-False]	34.1030μs	9.5265μs	104.9702 KOps/s	105.7078 KOps/s	$\color{#d91a1a}-0.70\%$
test_step_mdp_speed[True-False-True-True-True]	78.7890μs	25.1035μs	39.8350 KOps/s	39.0040 KOps/s	$\color{#35bf28}+2.13\%$
test_step_mdp_speed[True-False-True-True-False]	56.9340μs	16.3464μs	61.1757 KOps/s	61.4928 KOps/s	$\color{#d91a1a}-0.52\%$
test_step_mdp_speed[True-False-True-False-True]	45.6350μs	14.8761μs	67.2217 KOps/s	66.7026 KOps/s	$\color{#35bf28}+0.78\%$
test_step_mdp_speed[True-False-True-False-False]	71.3540μs	9.5869μs	104.3087 KOps/s	105.0892 KOps/s	$\color{#d91a1a}-0.74\%$
test_step_mdp_speed[True-False-False-True-True]	74.4190μs	26.4385μs	37.8236 KOps/s	37.6605 KOps/s	$\color{#35bf28}+0.43\%$
test_step_mdp_speed[True-False-False-True-False]	62.8980μs	17.4917μs	57.1701 KOps/s	57.1340 KOps/s	$\color{#35bf28}+0.06\%$
test_step_mdp_speed[True-False-False-False-True]	44.0320μs	16.1448μs	61.9394 KOps/s	62.2388 KOps/s	$\color{#d91a1a}-0.48\%$
test_step_mdp_speed[True-False-False-False-False]	33.7230μs	10.5811μs	94.5084 KOps/s	93.5146 KOps/s	$\color{#35bf28}+1.06\%$
test_step_mdp_speed[False-True-True-True-True]	75.2900μs	25.6060μs	39.0533 KOps/s	39.3439 KOps/s	$\color{#d91a1a}-0.74\%$
test_step_mdp_speed[False-True-True-True-False]	55.2630μs	16.1837μs	61.7906 KOps/s	60.5707 KOps/s	$\color{#35bf28}+2.01\%$
test_step_mdp_speed[False-True-True-False-True]	54.8430μs	17.5193μs	57.0798 KOps/s	57.5938 KOps/s	$\color{#d91a1a}-0.89\%$
test_step_mdp_speed[False-True-True-False-False]	33.3620μs	10.7782μs	92.7795 KOps/s	91.5565 KOps/s	$\color{#35bf28}+1.34\%$
test_step_mdp_speed[False-True-False-True-True]	88.7960μs	26.9227μs	37.1434 KOps/s	37.6420 KOps/s	$\color{#d91a1a}-1.32\%$
test_step_mdp_speed[False-True-False-True-False]	66.6850μs	17.5295μs	57.0466 KOps/s	56.7667 KOps/s	$\color{#35bf28}+0.49\%$
test_step_mdp_speed[False-True-False-False-True]	57.0760μs	18.5995μs	53.7649 KOps/s	54.4279 KOps/s	$\color{#d91a1a}-1.22\%$
test_step_mdp_speed[False-True-False-False-False]	75.4110μs	11.9541μs	83.6532 KOps/s	83.3894 KOps/s	$\color{#35bf28}+0.32\%$
test_step_mdp_speed[False-False-True-True-True]	91.3430μs	27.6776μs	36.1304 KOps/s	35.4040 KOps/s	$\color{#35bf28}+2.05\%$
test_step_mdp_speed[False-False-True-True-False]	55.8740μs	18.6822μs	53.5269 KOps/s	53.2022 KOps/s	$\color{#35bf28}+0.61\%$
test_step_mdp_speed[False-False-True-False-True]	64.6380μs	18.2483μs	54.7996 KOps/s	53.2847 KOps/s	$\color{#35bf28}+2.84\%$
test_step_mdp_speed[False-False-True-False-False]	54.8340μs	11.9006μs	84.0292 KOps/s	82.6149 KOps/s	$\color{#35bf28}+1.71\%$
test_step_mdp_speed[False-False-False-True-True]	76.6820μs	28.7189μs	34.8203 KOps/s	34.5102 KOps/s	$\color{#35bf28}+0.90\%$
test_step_mdp_speed[False-False-False-True-False]	68.2370μs	19.8288μs	50.4317 KOps/s	50.4791 KOps/s	$\color{#d91a1a}-0.09\%$
test_step_mdp_speed[False-False-False-False-True]	42.5300μs	19.3583μs	51.6575 KOps/s	50.8523 KOps/s	$\color{#35bf28}+1.58\%$
test_step_mdp_speed[False-False-False-False-False]	64.2300μs	12.9377μs	77.2932 KOps/s	77.2606 KOps/s	$\color{#35bf28}+0.04\%$
test_values[generalized_advantage_estimate-True-True]	14.0804ms	12.2214ms	81.8237 Ops/s	82.1451 Ops/s	$\color{#d91a1a}-0.39\%$
test_values[vec_generalized_advantage_estimate-True-True]	38.1451ms	26.8332ms	37.2673 Ops/s	37.0595 Ops/s	$\color{#35bf28}+0.56\%$
test_values[td0_return_estimate-False-False]	0.2331ms	0.1857ms	5.3840 KOps/s	5.4859 KOps/s	$\color{#d91a1a}-1.86\%$
test_values[td1_return_estimate-False-False]	26.4852ms	25.8370ms	38.7042 Ops/s	38.5905 Ops/s	$\color{#35bf28}+0.29\%$
test_values[vec_td1_return_estimate-False-False]	36.8707ms	26.6234ms	37.5610 Ops/s	36.4389 Ops/s	$\color{#35bf28}+3.08\%$
test_values[td_lambda_return_estimate-True-False]	45.3492ms	36.7656ms	27.1994 Ops/s	27.7371 Ops/s	$\color{#d91a1a}-1.94\%$
test_values[vec_td_lambda_return_estimate-True-False]	36.4647ms	26.7146ms	37.4327 Ops/s	37.0874 Ops/s	$\color{#35bf28}+0.93\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	8.2687ms	8.1493ms	122.7102 Ops/s	123.3221 Ops/s	$\color{#d91a1a}-0.50\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	12.8013ms	1.9365ms	516.3859 Ops/s	554.8081 Ops/s	$\textbf{\color{#d91a1a}-6.93\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	11.1474ms	0.4534ms	2.2055 KOps/s	2.2799 KOps/s	$\color{#d91a1a}-3.26\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	48.8113ms	40.1179ms	24.9265 Ops/s	25.2061 Ops/s	$\color{#d91a1a}-1.11\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	11.7456ms	2.5509ms	392.0158 Ops/s	391.5953 Ops/s	$\color{#35bf28}+0.11\%$
test_dqn_speed	8.4719ms	1.6207ms	616.9995 Ops/s	598.9677 Ops/s	$\color{#35bf28}+3.01\%$
test_ddpg_speed	72.3734ms	3.9200ms	255.1031 Ops/s	275.3603 Ops/s	$\textbf{\color{#d91a1a}-7.36\%}$
test_sac_speed	11.1166ms	10.1538ms	98.4855 Ops/s	97.2126 Ops/s	$\color{#35bf28}+1.31\%$
test_redq_speed	29.9535ms	19.7146ms	50.7238 Ops/s	51.8625 Ops/s	$\color{#d91a1a}-2.20\%$
test_redq_deprec_speed	26.6949ms	15.7117ms	63.6467 Ops/s	64.2872 Ops/s	$\color{#d91a1a}-1.00\%$
test_td3_speed	21.7666ms	10.5754ms	94.5590 Ops/s	94.4240 Ops/s	$\color{#35bf28}+0.14\%$
test_cql_speed	50.5758ms	38.7258ms	25.8226 Ops/s	25.5123 Ops/s	$\color{#35bf28}+1.22\%$
test_a2c_speed	18.9132ms	8.1665ms	122.4518 Ops/s	118.1227 Ops/s	$\color{#35bf28}+3.66\%$
test_ppo_speed	19.1483ms	8.5085ms	117.5302 Ops/s	113.2807 Ops/s	$\color{#35bf28}+3.75\%$
test_reinforce_speed	9.0100ms	7.2702ms	137.5486 Ops/s	133.8437 Ops/s	$\color{#35bf28}+2.77\%$
test_iql_speed	44.9871ms	34.3962ms	29.0730 Ops/s	28.8146 Ops/s	$\color{#35bf28}+0.90\%$
test_sample_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	2.9376ms	1.9700ms	507.6251 Ops/s	514.0858 Ops/s	$\color{#d91a1a}-1.26\%$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	3.2949ms	2.1141ms	473.0104 Ops/s	476.5658 Ops/s	$\color{#d91a1a}-0.75\%$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	3.6043ms	2.1231ms	471.0018 Ops/s	478.7991 Ops/s	$\color{#d91a1a}-1.63\%$
test_sample_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.1433ms	2.0015ms	499.6232 Ops/s	508.0996 Ops/s	$\color{#d91a1a}-1.67\%$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	3.3815ms	2.1182ms	472.1009 Ops/s	475.2456 Ops/s	$\color{#d91a1a}-0.66\%$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	3.5415ms	2.1264ms	470.2775 Ops/s	472.1901 Ops/s	$\color{#d91a1a}-0.41\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	2.5659ms	1.9540ms	511.7614 Ops/s	510.5988 Ops/s	$\color{#35bf28}+0.23\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	3.4308ms	2.1203ms	471.6303 Ops/s	478.6612 Ops/s	$\color{#d91a1a}-1.47\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	3.2113ms	2.1195ms	471.8109 Ops/s	472.8666 Ops/s	$\color{#d91a1a}-0.22\%$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.1583ms	1.9784ms	505.4503 Ops/s	497.1094 Ops/s	$\color{#35bf28}+1.68\%$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	3.6753ms	2.1578ms	463.4261 Ops/s	473.4800 Ops/s	$\color{#d91a1a}-2.12\%$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	3.6927ms	2.1412ms	467.0382 Ops/s	464.3769 Ops/s	$\color{#35bf28}+0.57\%$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.1670ms	1.9806ms	504.8937 Ops/s	504.1485 Ops/s	$\color{#35bf28}+0.15\%$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	3.2030ms	2.1049ms	475.0712 Ops/s	464.7567 Ops/s	$\color{#35bf28}+2.22\%$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	3.3591ms	2.1179ms	472.1624 Ops/s	477.2363 Ops/s	$\color{#d91a1a}-1.06\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.6735ms	1.9536ms	511.8723 Ops/s	509.5704 Ops/s	$\color{#35bf28}+0.45\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	3.6365ms	2.1072ms	474.5563 Ops/s	465.6575 Ops/s	$\color{#35bf28}+1.91\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	3.9943ms	2.1357ms	468.2297 Ops/s	472.0129 Ops/s	$\color{#d91a1a}-0.80\%$
test_populate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1731s	18.0667ms	55.3505 Ops/s	54.1027 Ops/s	$\color{#35bf28}+2.31\%$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	0.1180s	16.9381ms	59.0384 Ops/s	59.4888 Ops/s	$\color{#d91a1a}-0.76\%$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	0.1140s	16.8184ms	59.4588 Ops/s	52.7334 Ops/s	$\textbf{\color{#35bf28}+12.75\%}$
test_populate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1118s	16.7447ms	59.7205 Ops/s	59.2592 Ops/s	$\color{#35bf28}+0.78\%$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	0.1188s	18.9676ms	52.7215 Ops/s	59.5743 Ops/s	$\textbf{\color{#d91a1a}-11.50\%}$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	0.1178s	16.9409ms	59.0286 Ops/s	59.7943 Ops/s	$\color{#d91a1a}-1.28\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1160s	16.9939ms	58.8447 Ops/s	59.5020 Ops/s	$\color{#d91a1a}-1.10\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	0.1132s	16.8341ms	59.4033 Ops/s	59.9139 Ops/s	$\color{#d91a1a}-0.85\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	0.1150s	16.8583ms	59.3180 Ops/s	59.1263 Ops/s	$\color{#35bf28}+0.32\%$

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 92. Improved: $\large\color{#35bf28}4$. Worsened: $\large\color{#d91a1a}2$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	0.1205s	0.1195s	8.3669 Ops/s	7.9861 Ops/s	$\color{#35bf28}+4.77\%$
test_sync	0.1026s	0.1024s	9.7645 Ops/s	9.6781 Ops/s	$\color{#35bf28}+0.89\%$
test_async	0.2709s	0.1017s	9.8288 Ops/s	9.9660 Ops/s	$\color{#d91a1a}-1.38\%$
test_single_pixels	0.1431s	0.1426s	7.0137 Ops/s	6.8878 Ops/s	$\color{#35bf28}+1.83\%$
test_sync_pixels	98.9870ms	96.3499ms	10.3788 Ops/s	10.5184 Ops/s	$\color{#d91a1a}-1.33\%$
test_async_pixels	0.1863s	90.1381ms	11.0941 Ops/s	11.0860 Ops/s	$\color{#35bf28}+0.07\%$
test_simple	0.9315s	0.8641s	1.1573 Ops/s	1.1315 Ops/s	$\color{#35bf28}+2.28\%$
test_transformed	1.1736s	1.1075s	0.9029 Ops/s	0.8812 Ops/s	$\color{#35bf28}+2.46\%$
test_serial	2.4834s	2.4143s	0.4142 Ops/s	0.4035 Ops/s	$\color{#35bf28}+2.66\%$
test_parallel	2.5331s	2.4748s	0.4041 Ops/s	0.4022 Ops/s	$\color{#35bf28}+0.46\%$
test_step_mdp_speed[True-True-True-True-True]	98.2140μs	33.3779μs	29.9599 KOps/s	29.2569 KOps/s	$\color{#35bf28}+2.40\%$
test_step_mdp_speed[True-True-True-True-False]	63.1330μs	19.8401μs	50.4030 KOps/s	49.4269 KOps/s	$\color{#35bf28}+1.97\%$
test_step_mdp_speed[True-True-True-False-True]	46.5620μs	19.7651μs	50.5942 KOps/s	49.0509 KOps/s	$\color{#35bf28}+3.15\%$
test_step_mdp_speed[True-True-True-False-False]	51.3830μs	11.6575μs	85.7820 KOps/s	84.3114 KOps/s	$\color{#35bf28}+1.74\%$
test_step_mdp_speed[True-True-False-True-True]	68.8030μs	35.1269μs	28.4682 KOps/s	28.5071 KOps/s	$\color{#d91a1a}-0.14\%$
test_step_mdp_speed[True-True-False-True-False]	79.8740μs	21.3821μs	46.7680 KOps/s	45.5486 KOps/s	$\color{#35bf28}+2.68\%$
test_step_mdp_speed[True-True-False-False-True]	47.8720μs	21.1618μs	47.2549 KOps/s	46.3383 KOps/s	$\color{#35bf28}+1.98\%$
test_step_mdp_speed[True-True-False-False-False]	38.1810μs	13.5105μs	74.0165 KOps/s	73.3348 KOps/s	$\color{#35bf28}+0.93\%$
test_step_mdp_speed[True-False-True-True-True]	67.7930μs	37.0224μs	27.0107 KOps/s	26.9799 KOps/s	$\color{#35bf28}+0.11\%$
test_step_mdp_speed[True-False-True-True-False]	58.6130μs	23.1678μs	43.1633 KOps/s	42.2529 KOps/s	$\color{#35bf28}+2.15\%$
test_step_mdp_speed[True-False-True-False-True]	59.3930μs	21.5460μs	46.4124 KOps/s	46.5639 KOps/s	$\color{#d91a1a}-0.33\%$
test_step_mdp_speed[True-False-True-False-False]	36.7220μs	13.4554μs	74.3194 KOps/s	72.7338 KOps/s	$\color{#35bf28}+2.18\%$
test_step_mdp_speed[True-False-False-True-True]	91.3540μs	38.9625μs	25.6657 KOps/s	25.5189 KOps/s	$\color{#35bf28}+0.58\%$
test_step_mdp_speed[True-False-False-True-False]	55.6530μs	25.1462μs	39.7674 KOps/s	38.7271 KOps/s	$\color{#35bf28}+2.69\%$
test_step_mdp_speed[True-False-False-False-True]	69.5930μs	23.0974μs	43.2948 KOps/s	41.9656 KOps/s	$\color{#35bf28}+3.17\%$
test_step_mdp_speed[True-False-False-False-False]	36.3110μs	15.2978μs	65.3688 KOps/s	64.5861 KOps/s	$\color{#35bf28}+1.21\%$
test_step_mdp_speed[False-True-True-True-True]	66.1430μs	36.6634μs	27.2751 KOps/s	26.8271 KOps/s	$\color{#35bf28}+1.67\%$
test_step_mdp_speed[False-True-True-True-False]	53.2220μs	23.3674μs	42.7946 KOps/s	41.5047 KOps/s	$\color{#35bf28}+3.11\%$
test_step_mdp_speed[False-True-True-False-True]	50.5020μs	25.6738μs	38.9502 KOps/s	38.8456 KOps/s	$\color{#35bf28}+0.27\%$
test_step_mdp_speed[False-True-True-False-False]	46.7220μs	15.0498μs	66.4460 KOps/s	64.2572 KOps/s	$\color{#35bf28}+3.41\%$
test_step_mdp_speed[False-True-False-True-True]	72.2040μs	38.5857μs	25.9163 KOps/s	25.0134 KOps/s	$\color{#35bf28}+3.61\%$
test_step_mdp_speed[False-True-False-True-False]	52.5720μs	25.4382μs	39.3110 KOps/s	38.8786 KOps/s	$\color{#35bf28}+1.11\%$
test_step_mdp_speed[False-True-False-False-True]	61.2230μs	27.6824μs	36.1240 KOps/s	35.8732 KOps/s	$\color{#35bf28}+0.70\%$
test_step_mdp_speed[False-True-False-False-False]	38.1820μs	16.8521μs	59.3399 KOps/s	58.0550 KOps/s	$\color{#35bf28}+2.21\%$
test_step_mdp_speed[False-False-True-True-True]	75.6330μs	40.4051μs	24.7494 KOps/s	24.4361 KOps/s	$\color{#35bf28}+1.28\%$
test_step_mdp_speed[False-False-True-True-False]	64.4130μs	26.9024μs	37.1714 KOps/s	36.4839 KOps/s	$\color{#35bf28}+1.88\%$
test_step_mdp_speed[False-False-True-False-True]	53.9120μs	27.6087μs	36.2204 KOps/s	36.8488 KOps/s	$\color{#d91a1a}-1.71\%$
test_step_mdp_speed[False-False-True-False-False]	43.5120μs	16.9089μs	59.1403 KOps/s	58.2549 KOps/s	$\color{#35bf28}+1.52\%$
test_step_mdp_speed[False-False-False-True-True]	0.1183ms	41.7540μs	23.9498 KOps/s	23.3405 KOps/s	$\color{#35bf28}+2.61\%$
test_step_mdp_speed[False-False-False-True-False]	62.4130μs	29.4041μs	34.0089 KOps/s	34.1385 KOps/s	$\color{#d91a1a}-0.38\%$
test_step_mdp_speed[False-False-False-False-True]	81.4640μs	28.6735μs	34.8754 KOps/s	34.6057 KOps/s	$\color{#35bf28}+0.78\%$
test_step_mdp_speed[False-False-False-False-False]	50.5030μs	18.8222μs	53.1287 KOps/s	52.8140 KOps/s	$\color{#35bf28}+0.60\%$
test_values[generalized_advantage_estimate-True-True]	27.2273ms	25.4965ms	39.2210 Ops/s	37.5329 Ops/s	$\color{#35bf28}+4.50\%$
test_values[vec_generalized_advantage_estimate-True-True]	83.8032ms	3.2342ms	309.1907 Ops/s	292.4349 Ops/s	$\textbf{\color{#35bf28}+5.73\%}$
test_values[td0_return_estimate-False-False]	0.1019ms	65.0136μs	15.3814 KOps/s	14.8973 KOps/s	$\color{#35bf28}+3.25\%$
test_values[td1_return_estimate-False-False]	57.8262ms	56.1779ms	17.8006 Ops/s	17.3573 Ops/s	$\color{#35bf28}+2.55\%$
test_values[vec_td1_return_estimate-False-False]	2.0832ms	1.7517ms	570.8724 Ops/s	568.2741 Ops/s	$\color{#35bf28}+0.46\%$
test_values[td_lambda_return_estimate-True-False]	92.4771ms	89.2151ms	11.2089 Ops/s	10.7016 Ops/s	$\color{#35bf28}+4.74\%$
test_values[vec_td_lambda_return_estimate-True-False]	2.0886ms	1.7339ms	576.7218 Ops/s	565.7503 Ops/s	$\color{#35bf28}+1.94\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	24.6283ms	24.2220ms	41.2847 Ops/s	38.1233 Ops/s	$\textbf{\color{#35bf28}+8.29\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	0.8420ms	0.7178ms	1.3931 KOps/s	1.3467 KOps/s	$\color{#35bf28}+3.45\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.7607ms	0.6804ms	1.4698 KOps/s	1.4102 KOps/s	$\color{#35bf28}+4.23\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	1.5300ms	1.4669ms	681.7318 Ops/s	668.5641 Ops/s	$\color{#35bf28}+1.97\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	0.9648ms	0.7095ms	1.4095 KOps/s	1.3816 KOps/s	$\color{#35bf28}+2.02\%$
test_dqn_speed	7.6599ms	1.4490ms	690.1387 Ops/s	676.0716 Ops/s	$\color{#35bf28}+2.08\%$
test_ddpg_speed	4.6552ms	3.2688ms	305.9258 Ops/s	305.5267 Ops/s	$\color{#35bf28}+0.13\%$
test_sac_speed	10.2426ms	9.1577ms	109.1973 Ops/s	99.8618 Ops/s	$\textbf{\color{#35bf28}+9.35\%}$
test_redq_speed	17.0607ms	16.4881ms	60.6497 Ops/s	59.6981 Ops/s	$\color{#35bf28}+1.59\%$
test_redq_deprec_speed	0.1028s	14.1721ms	70.5609 Ops/s	78.1444 Ops/s	$\textbf{\color{#d91a1a}-9.70\%}$
test_td3_speed	9.4806ms	9.3063ms	107.4545 Ops/s	105.4723 Ops/s	$\color{#35bf28}+1.88\%$
test_cql_speed	32.7444ms	31.6499ms	31.5957 Ops/s	31.9509 Ops/s	$\color{#d91a1a}-1.11\%$
test_a2c_speed	8.5928ms	7.2583ms	137.7725 Ops/s	142.7019 Ops/s	$\color{#d91a1a}-3.45\%$
test_ppo_speed	8.8619ms	7.5385ms	132.6530 Ops/s	136.0143 Ops/s	$\color{#d91a1a}-2.47\%$
test_reinforce_speed	7.5583ms	6.2503ms	159.9935 Ops/s	165.0519 Ops/s	$\color{#d91a1a}-3.06\%$
test_iql_speed	28.0903ms	26.6232ms	37.5613 Ops/s	37.4979 Ops/s	$\color{#35bf28}+0.17\%$
test_sample_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	2.6531ms	2.4390ms	410.0082 Ops/s	394.5960 Ops/s	$\color{#35bf28}+3.91\%$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	3.6170ms	2.6094ms	383.2257 Ops/s	367.9250 Ops/s	$\color{#35bf28}+4.16\%$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	4.0404ms	2.6238ms	381.1314 Ops/s	366.3658 Ops/s	$\color{#35bf28}+4.03\%$
test_sample_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	2.6632ms	2.4397ms	409.8859 Ops/s	393.9723 Ops/s	$\color{#35bf28}+4.04\%$
test_sample_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	3.9662ms	2.6158ms	382.2879 Ops/s	365.7064 Ops/s	$\color{#35bf28}+4.53\%$
test_sample_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	3.5388ms	2.6164ms	382.2091 Ops/s	367.2982 Ops/s	$\color{#35bf28}+4.06\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	2.8366ms	2.4313ms	411.3089 Ops/s	395.1498 Ops/s	$\color{#35bf28}+4.09\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	4.1000ms	2.6111ms	382.9862 Ops/s	366.7949 Ops/s	$\color{#35bf28}+4.41\%$
test_sample_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	3.7191ms	2.6187ms	381.8714 Ops/s	367.6835 Ops/s	$\color{#35bf28}+3.86\%$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.0573ms	2.4438ms	409.1953 Ops/s	398.7138 Ops/s	$\color{#35bf28}+2.63\%$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	3.6138ms	2.6169ms	382.1262 Ops/s	366.4630 Ops/s	$\color{#35bf28}+4.27\%$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	3.9909ms	2.6151ms	382.3935 Ops/s	366.3323 Ops/s	$\color{#35bf28}+4.38\%$
test_iterate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.0506ms	2.4293ms	411.6345 Ops/s	398.4839 Ops/s	$\color{#35bf28}+3.30\%$
test_iterate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	3.4945ms	2.6126ms	382.7534 Ops/s	366.8575 Ops/s	$\color{#35bf28}+4.33\%$
test_iterate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	4.2613ms	2.6268ms	380.6859 Ops/s	365.5734 Ops/s	$\color{#35bf28}+4.13\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.0619ms	2.4304ms	411.4549 Ops/s	394.8091 Ops/s	$\color{#35bf28}+4.22\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	3.7639ms	2.6230ms	381.2387 Ops/s	366.5410 Ops/s	$\color{#35bf28}+4.01\%$
test_iterate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	3.7988ms	2.6199ms	381.6893 Ops/s	365.5442 Ops/s	$\color{#35bf28}+4.42\%$
test_populate_rb[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.2223s	19.5068ms	51.2641 Ops/s	50.7880 Ops/s	$\color{#35bf28}+0.94\%$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	0.1281s	17.4053ms	57.4537 Ops/s	55.4753 Ops/s	$\color{#35bf28}+3.57\%$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	0.1282s	17.5897ms	56.8513 Ops/s	55.2736 Ops/s	$\color{#35bf28}+2.85\%$
test_populate_rb[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1276s	15.2040ms	65.7723 Ops/s	55.2654 Ops/s	$\textbf{\color{#35bf28}+19.01\%}$
test_populate_rb[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	0.1285s	17.5812ms	56.8790 Ops/s	63.4691 Ops/s	$\textbf{\color{#d91a1a}-10.38\%}$
test_populate_rb[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	0.1299s	17.4911ms	57.1718 Ops/s	55.4492 Ops/s	$\color{#35bf28}+3.11\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1293s	17.4155ms	57.4200 Ops/s	55.6071 Ops/s	$\color{#35bf28}+3.26\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	0.1282s	17.6137ms	56.7739 Ops/s	55.3126 Ops/s	$\color{#35bf28}+2.64\%$
test_populate_rb[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	0.1282s	17.5699ms	56.9157 Ops/s	55.1187 Ops/s	$\color{#35bf28}+3.26\%$