Testbed folders

[afourney]

Why are these changes needed?

The current testbed templating format works great for single-file scenarios, but is less flexible when multiple files need to be included in a test (e.g., including a PDF or image to operate over). This PR moves the templating format to one that accepts whole folders. Backwards compatibility is maintained.

Related issue number

This PR will enable progress on #691, #692, and other benchmarks (e.g., newly released GAIA)

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (ae7066b) 27.77% compared to head (d712076) 27.77%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #792   +/-   ##
=======================================
  Coverage   27.77%   27.77%           
=======================================
  Files          27       27           
  Lines        3500     3500           
  Branches      794      794           
=======================================
  Hits          972      972           
  Misses       2457     2457           
  Partials       71       71           

Flag	Coverage Δ
unittests	27.71% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[yiranwu0]

I was looking at this testbed and has a question about Docker. Currently the run_scenario will pull the python:3.11 docker image and do pip install every time. Is it possible to pass in names of a customized docker images to run (or am I overlooking)? I think it is even better if users could just pass in a docker_build.txt with the required python packages to create a new docker image if the intention is to run a large scale experiment.

[LeoLjl]

I was looking at this testbed and has a question about Docker. Currently the run_scenario will pull the python:3.11 docker image and do pip install every time. Is it possible to pass in names of a customized docker images to run (or am I overlooking)? I think it is even better if users could just pass in a docker_build.txt with the required python packages to create a new docker image if the intention is to run a large scale experiment.

I added package names to include/requirements.txt so that every time a new dokcer image is created, these packages are automatically installed. This could be a workaround if installation is fast.

[afourney]

Yes, adding support for a custom Docker image, or a Docker file would be a logical next step. At present, there are a couple of options. (1) you can update or specify a requirements.txt file. Or (2) you can customize the global_startup.sh or scenario_startup.sh files to install by other means. The new folder-oriented specification means that you can include zips, packages, or other dependencies in the includes or scenario template folders as well, allowing for testing on private content.