Skip to content

Commit

Permalink
update benchmark tables
Browse files Browse the repository at this point in the history
  • Loading branch information
@kengz
kengz committed Sep 23, 2019
1 parent 63bb516 commit e5988f0
Show file tree
Hide file tree
Showing 2 changed files with 26 additions and 26 deletions.
26 changes: 13 additions & 13 deletions BENCHMARK.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,24 +64,24 @@ The specs for these are contained in the [`slm_lab/spec/benchmark`](https://gith

>Click on the algorithm to see the result upload Pull Request.
| Env. \ Alg. | [A2C (GAE)](https://github.com/kengz/SLM-Lab/pull/405) | [A2C (n-step)](https://github.com/kengz/SLM-Lab/pull/405) | [PPO](https://github.com/kengz/SLM-Lab/pull/405) | [SAC](https://github.com/kengz/SLM-Lab/pull/409) |
| Env. \ Alg. | [A2C (GAE)](https://github.com/kengz/SLM-Lab/pull/416) | [A2C (n-step)](https://github.com/kengz/SLM-Lab/pull/416) | [PPO](https://github.com/kengz/SLM-Lab/pull/416) | [SAC](https://github.com/kengz/SLM-Lab/pull/416) |
|:---|---|---|---|---|
| RoboschoolAnt <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64073199-7c376a80-cc4f-11e9-87bd-026fc0a75156.png"><img src="https://user-images.githubusercontent.com/8209263/64073200-7c376a80-cc4f-11e9-87d3-4a4fc5080537.png"></details> | 1029.51 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822013-d4930300-c903-11e9-91f8-13e9bda44b59.png"></details> | 1148.76 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822318-d90beb80-c904-11e9-8ddf-16c105508e5e.png"></details> | 1931.35 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822580-a57d9100-c905-11e9-8d31-6df248d62a74.png"></details> | **2914.75** <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64072328-76d32380-cc41-11e9-9750-9c6e3a52b6f6.png"></details> |
| RoboschoolAtlasForwardWalk <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64073201-7c376a80-cc4f-11e9-891c-8269f2f880e3.png"><img src="https://user-images.githubusercontent.com/8209263/64073202-7cd00100-cc4f-11e9-868a-0cb959b64c46.png"></details> | 68.15 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822079-16bc4480-c904-11e9-9b0b-d63267f820aa.png"></details> | 73.46 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822362-f17c0600-c904-11e9-888b-2511bc59df9a.png"></details> | 148.81 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822614-c7771380-c905-11e9-9978-5fc9a6b0f218.png"></details> | **942.39** <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822963-f93caa00-c906-11e9-869f-df8f6032d469.png"></details> |
| RoboschoolHalfCheetah <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64073203-7cd00100-cc4f-11e9-879d-06d13fc5e8ac.png"><img src="https://user-images.githubusercontent.com/8209263/64073204-7cd00100-cc4f-11e9-8809-cb43b38baec2.png"></details> | 895.24 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822128-3a7f8a80-c904-11e9-8e48-12c3214197ec.png"></details> | 409.59 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822377-02c51280-c905-11e9-9ff3-5a96f3c7ba05.png"></details> | 1838.69 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822643-dcec3d80-c905-11e9-89df-a30f84842bb1.png"></details> | **2496.54** <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64072874-05986e00-cc4b-11e9-8533-66afb636be79.png"></details> |
| RoboschoolHopper <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64073205-7cd00100-cc4f-11e9-9cb0-552a687df369.png"><img src="https://user-images.githubusercontent.com/8209263/64073206-7cd00100-cc4f-11e9-8982-b4a7416f3728.png"></details> | 286.67 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822146-45d2b600-c904-11e9-976d-e61d1c215427.png"></details> | -187.91 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822416-196b6980-c905-11e9-8410-f2ff57a63983.png"></details> | 2079.22 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822669-ed9cb380-c905-11e9-9732-9cce76683f60.png"></details> | **2251.36** <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64072357-d6313380-cc41-11e9-8e0c-4adde788fb5d.png"></details> |
| RoboschoolInvertedDoublePendulum <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64073207-7cd00100-cc4f-11e9-9207-e25f6d666cb8.png"><img src="https://user-images.githubusercontent.com/8209263/64073208-7d689780-cc4f-11e9-8deb-9c517a8aabde.png"></details> | 1769.74 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822174-5f73fd80-c904-11e9-848f-959aaff30f11.png"></details> | 486.76 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822446-2f792a00-c905-11e9-972e-545493c136be.png"></details> | 7967.03 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822686-0016ed00-c906-11e9-8064-7eecab895646.png"></details> | **8085.04** <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64072341-a5e99500-cc41-11e9-9f29-011f74bcbd55.png"></details> |
| RoboschoolInvertedPendulum <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64073209-7d689780-cc4f-11e9-8f2e-8bffe7aafaa0.png"><img src="https://user-images.githubusercontent.com/8209263/64073210-7d689780-cc4f-11e9-95b3-a11db92aeec5.png"></details> | **1000.0** <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822205-7b779f00-c904-11e9-9ac7-d686fc640a3b.png"></details> | 997.54 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822472-44ee5400-c905-11e9-8c93-2973786ba561.png"></details> | 930.29 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822719-19b83480-c906-11e9-9ec7-245f9dcf4431.png"></details> | 941.45 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64072887-3ed0de00-cc4b-11e9-8044-3e0a07f2c526.png"></details> |
| RoboschoolReacher <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64073211-7d689780-cc4f-11e9-94dc-90c0d9cdfca4.png"><img src="https://user-images.githubusercontent.com/8209263/64073212-7d689780-cc4f-11e9-8c4f-fc5f26cebc33.png"></details> | 14.57 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822262-a661f300-c904-11e9-80be-8a7cb986fa77.png"></details> | -6.18 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822533-75ce8900-c905-11e9-87a6-dbdc1c0a74cc.png"></details> | 19.18 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822853-9ea34e00-c906-11e9-8fef-93e18f17c5fc.png"></details> | **19.99** <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64072892-54460800-cc4b-11e9-8784-fed41c3db49e.png"></details> |
| RoboschoolWalker2d <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64073213-7d689780-cc4f-11e9-88b1-18f9f826387a.png"><img src="https://user-images.githubusercontent.com/8209263/64073214-7e012e00-cc4f-11e9-8304-7833fc32403e.png"></details> | 413.26 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822280-b548a580-c904-11e9-943b-168bcd58ea01.png"></details> | 141.83 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822553-87b02c00-c905-11e9-99b0-87da689a3bd2.png"></details> | 1368.25 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/63822867-afec5a80-c906-11e9-8b14-453bf1f381d2.png"></details> | **1894.05** <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64072897-69bb3200-cc4b-11e9-9a0d-18dc04e83a3f.png"></details> |
| RoboschoolAnt <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/65396960-d0220480-dd60-11e9-94ed-b6be9310a3e8.png"></details> | 787 | 1396 | 1843 | **2915** |
| RoboschoolAtlasForwardWalk <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/65396961-d0220480-dd60-11e9-8c0c-bc60a2c80f80.png"></details> | 59.87 | 88.04 | 172 | **800** |
| RoboschoolHalfCheetah <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/65396962-d0ba9b00-dd60-11e9-904d-4592f81a1432.png"></details> | 712 | 439 | 1960 | **2497** |
| RoboschoolHopper <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/65396963-d0ba9b00-dd60-11e9-9e8c-87e442d61dd4.png"></details> | 710 | 285 | 2042 | **2045** |
| RoboschoolInvertedDoublePendulum <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/65396967-d0ba9b00-dd60-11e9-9f30-514aee1fb88f.png"></details> | 996 | 4410 | 8076 | **8085** |
| RoboschoolInvertedPendulum <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/65396968-d1533180-dd60-11e9-8dba-c838a58b9fd3.png"></details> | **995** | 978 | 986 | 941 |
| RoboschoolReacher <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/65396970-d1533180-dd60-11e9-9f5b-e57502b1b094.png"></details> | 12.9 | 10.16 | 19.51 | **19.99** |
| RoboschoolWalker2d <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/65396971-d1533180-dd60-11e9-9c07-b83215f19a20.png"></details> | 280 | 220 | 1660 | **1894** |

Humanoid environments are significantly harder. Note that due to the number of frames required, we could only run Async-SAC.

| Env. \ Alg. | [A2C (GAE)](https://github.com/kengz/SLM-Lab/pull/409) | [A2C (n-step)](https://github.com/kengz/SLM-Lab/pull/409) | [PPO](https://github.com/kengz/SLM-Lab/pull/409) | [Async-SAC](https://github.com/kengz/SLM-Lab/pull/409) |
| Env. \ Alg. | [A2C (GAE)](https://github.com/kengz/SLM-Lab/pull/416) | [A2C (n-step)](https://github.com/kengz/SLM-Lab/pull/416) | [PPO](https://github.com/kengz/SLM-Lab/pull/416) | [Async-SAC](https://github.com/kengz/SLM-Lab/pull/416) |
|:---|---|---|---|---|
| RoboschoolHumanoid | 122.23 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64072980-a176a980-cc4c-11e9-9973-e933306b2e0c.png"></details> | -6029.02 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64072987-b94e2d80-cc4c-11e9-8774-977c6176b014.png"></details> | 1554.03 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64073010-11852f80-cc4d-11e9-9257-1ebfda9817a6.png"></details> | **2621.46** <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64073022-54df9e00-cc4d-11e9-8003-e357c5735c22.png"></details> |
| RoboschoolHumanoidFlagrun | 93.48 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64072978-9754ab00-cc4c-11e9-9e3f-eb4ecd96baf0.png"></details> | -2079.02 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64072994-d08d1b00-cc4c-11e9-8747-22d99339b7db.png"></details> | 1635.64 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64073014-26fa5980-cc4d-11e9-94d7-c78ec868fd7b.png"></details> | **1937.77** <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64073030-704aa900-cc4d-11e9-83f3-858d3a20dcd9.png"></details> |
| RoboschoolHumanoidFlagrunHarder | -472.34 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64072973-8a37bc00-cc4c-11e9-91c2-bef0a3b30371.png"></details> | -24620.71 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64073003-f0bcda00-cc4c-11e9-981f-f4e12181a153.png"></details> | **610.09** <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64073019-3bd6ed00-cc4d-11e9-9d23-6579c09971a5.png"></details> | 280.18 <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/64073039-93755880-cc4d-11e9-8e3e-93f9e3beef16.png"></details> |
| RoboschoolHumanoid <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/65396964-d0ba9b00-dd60-11e9-9177-9bad7329ba48.png"></details> | 99.31 | 54.58 | 2388 | **2621** |
| RoboschoolHumanoidFlagrun <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/65396965-d0ba9b00-dd60-11e9-8796-cc2f0efe0818.png"></details> | 73.57 | 178 | 2014 | **2056** |
| RoboschoolHumanoidFlagrunHarder <details><summary><i>graph</i></summary><img src="https://user-images.githubusercontent.com/8209263/65396966-d0ba9b00-dd60-11e9-8814-1d8c06fb0527.png"></details> | -429 | 253 | **680** | 280 |


#### Asynchronous Algorithms Benchmark
Expand Down
Loading

0 comments on commit e5988f0

Please sign in to comment.