Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow users to specify the Docker image to use with Testbed #986

Merged
merged 2 commits into from
Dec 15, 2023

Conversation

afourney
Copy link
Member

@afourney afourney commented Dec 14, 2023

Why are these changes needed?

When scenarios are run on the Testbed, they previously started from a clean "python:3.11" docker image, pulled from Docker Hub. Dependencies were then installed from the requirements file. This whole process would repeat for each task being evaluated.

This was mostly fine when running simple AutoGen scenarios, but has gotten to be somewhat slow as AutoGen dependencies have gotten heavier, or as new benchmarks require additional dependencies.

Importantly, if no image is specified, this code will build an appropriate one for testing. Specifically it will install AutoGen's dependencies -- but not AutoGen itself, as we often want to test a particular branch pulled and installed from Git. It also includes the 10 libraries listed here: https://learnpython.com/blog/most-popular-python-packages/ , namely:

  • numpy
  • pandas
  • matplotlib
  • seaborn
  • scikit-learn
  • requests
  • urllib3
  • nltk
  • pillow
  • pytest

NOTE: The list of libraries to install by default deserves careful consideration. On the one hand, we likely want to assume some basic core libraries are available (e.g., requests, urllib3). On the other hand, determining which libraries are important for a task is often part of solving the task itself (equivalent to automatic tool selection). The decision of which libraries to include should therefore be made very carefully and in advance of considering any benchmarks or problem sets. My current thinking on this is to only pre-install libraries that:

-are basic AutoGen requirements (and so would be installed anyways)
-are common enough to find themselves on lists provided by independent/neutral 3rd parties (as above)

If a given benchmark requires other libraries to run (e.g., sympy for MATH), then it is reasonable to create a Docker Image just for that benchmark.

Related issue number

Closes #985 and commend made by @kevin666aa in #792

Checks

@afourney afourney added the proj-autogenbench Issues related to AutoGenBench. label Dec 14, 2023
@afourney afourney requested review from qingyun-wu, yiranwu0, LeoLjl and a team December 14, 2023 22:30
@afourney afourney self-assigned this Dec 14, 2023
@codecov-commenter
Copy link

codecov-commenter commented Dec 14, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (3a768c3) 26.63% compared to head (7962d07) 26.63%.
Report is 2 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #986   +/-   ##
=======================================
  Coverage   26.63%   26.63%           
=======================================
  Files          28       28           
  Lines        3777     3777           
  Branches      858      858           
=======================================
  Hits         1006     1006           
  Misses       2700     2700           
  Partials       71       71           
Flag Coverage Δ
unittests 26.58% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@victordibia
Copy link
Collaborator

PR looks good.
I think having known useful libraries will improve test bed dev experience. Also devs can update the dockerfile as needed.

samples/tools/testbed/Dockerfile Outdated Show resolved Hide resolved
Copy link
Collaborator

@yiranwu0 yiranwu0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@sonichi sonichi added this pull request to the merge queue Dec 15, 2023
Merged via the queue into main with commit 4dcb415 Dec 15, 2023
16 checks passed
rlam3 pushed a commit to rlam3/autogen that referenced this pull request Dec 19, 2023
…t#986)

* Allow users to specify the Docker image to use (or build a good AutoGen default image if not specified).

* Added lmm and graphs to dockerfile
@sonichi sonichi deleted the testbed_docker branch January 1, 2024 17:36
@afourney afourney mentioned this pull request Jan 2, 2024
3 tasks
whiskyboy pushed a commit to whiskyboy/autogen that referenced this pull request Apr 17, 2024
* outline

* revision

* eval function signature

* first draft

* link

* format

* example

* cleanup

* average

* move figure

* tldr

* bold

* bold

* tag
whiskyboy pushed a commit to whiskyboy/autogen that referenced this pull request Apr 17, 2024
…t#986)

* Allow users to specify the Docker image to use (or build a good AutoGen default image if not specified).

* Added lmm and graphs to dockerfile
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
proj-autogenbench Issues related to AutoGenBench.
Projects
None yet
6 participants