Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate first benchmark agent runs for agentos.org leaderboard #300

Open
andyk opened this issue Mar 4, 2022 · 0 comments
Open

Generate first benchmark agent runs for agentos.org leaderboard #300

andyk opened this issue Mar 4, 2022 · 0 comments
Assignees

Comments

@andyk
Copy link
Contributor

andyk commented Mar 4, 2022

We should populate our public registry w/ training and evaluation Runs of the most popular OSS agent x environment pairs. We'll start with 2 agents and one env., and then add more agents and more envs.

We currently have SB3 PPO playing cartpole.
FINE GRAINED TODOS:

  • Factor out CartPoleEnv into a common component whose GitHubRepo points to OpenAI's repo.
  • add PAPAG agent playing cartpole env.
  • add Ilya PAPAG agent playing gym-pong env.
  • add SB3 agent playing gym-pong env.

Below are our current shortlists of other agents and envs to benchmark:

Agents/algos

  1. SB3
  2. Ilya PAPAG PPO agent
  3. RLlib
  4. Acme
  5. RLkit
  6. Open-clip

Environments

  1. Atari games (one easy, one of the hardest)
  2. PyBullet or Mujoco simulator (humanoid? half cheetah? soccer?)
  3. Robotic arm simulator
  4. Classic control?
  5. DeepmindLab?
  6. Minecraft?
  7. [STRETCH] Hottest NLP (??) or vision benchmarks (ImageNet)

We should publish our code for these runs as new scripts in documentation/demos/ so that they double as helpful learning resources & examples of how to use the AgentOS & PCS APIs.

Note the registry is currently hosted at https://aos-web.herokuapp.com, after #331 it will be hosted at agentos.org/registry.

@andyk andyk moved this to Todo in Sprint 0.2.0 Mar 4, 2022
@andyk andyk changed the title Launch official Registry (setup url, populate first few envs/agents) Improve public Agent Registry (setup url, populate first few envs/agents) Mar 12, 2022
@andyk andyk changed the title Improve public Agent Registry (setup url, populate first few envs/agents) Populate first agent runs at agentos.org/registry Mar 22, 2022
@andyk andyk removed this from Sprint 0.2.0 Mar 25, 2022
@andyk andyk moved this to Todo in Sprint 0.2.1 Mar 25, 2022
@andyk andyk moved this from Todo to In Progress in Sprint 0.2.1 Apr 13, 2022
@andyk andyk changed the title Populate first agent runs at agentos.org/registry Generate first agent runs for agentos.org benchmark Apr 13, 2022
@andyk andyk changed the title Generate first agent runs for agentos.org benchmark Generate first benchmark agent runs for agentos.org leaderboard Apr 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In Progress
Development

No branches or pull requests

2 participants