Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generalization Metrics in ULTRA #615

Closed
2 of 4 tasks
AlexLewandowski opened this issue Feb 23, 2021 · 0 comments · Fixed by #639
Closed
2 of 4 tasks

Generalization Metrics in ULTRA #615

AlexLewandowski opened this issue Feb 23, 2021 · 0 comments · Fixed by #639
Assignees
@AlexLewandowski @JenishPatel99
Labels
ULTRA Tickets related to Unprotected Left Turns
Milestone
ULTRA

Comments

@AlexLewandowski
Copy link
Contributor

AlexLewandowski commented Feb 23, 2021 •
edited
Loading

End goal is leaderboard for generalization research

  • Record training performance on the same schedule as evaluation
  • Implement a way to calculate generalization gap (for solve rate / time to solve / env_score). This can be a post-process script or a separate LogInfo?
  • Add different difficulty levels
  • Implement a mechanism for the agent to go to a higher difficulty level (after 90% test or train solve rate)
@AlexLewandowski AlexLewandowski added the ULTRA Tickets related to Unprotected Left Turns label Feb 23, 2021
@AlexLewandowski AlexLewandowski added this to the ULTRA milestone Feb 23, 2021
@AlexLewandowski AlexLewandowski linked a pull request Mar 3, 2021 that will close this issue
Update evaluate and train to evaluate per episode #639
Merged
2 tasks
This was referenced Mar 12, 2021
Closed
Closed
@JenishPatel99 JenishPatel99 linked a pull request Mar 16, 2021 that will close this issue
ULTRA WIP: Add grade-based structure (cla fix) #675
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AlexLewandowski AlexLewandowski

@JenishPatel99 JenishPatel99

Labels
ULTRA Tickets related to Unprotected Left Turns
Projects
None yet
Milestone
ULTRA
3 participants
@AlexLewandowski @Adaickalavan @JenishPatel99