Merge branch 'main' into tinyra-refactor

Author: gagb

.devcontainer/Dockerfile

@@ -14,9 +14,13 @@ RUN apt-get update \
    && apt-get -y install --no-install-recommends build-essential npm \
    && apt-get autoremove -y \
    && apt-get clean -y \
-   && rm -rf /var/lib/apt/lists/*
+   && wget https://github.com/quarto-dev/quarto-cli/releases/download/v1.4.549/quarto-1.4.549-linux-amd64.deb \
+   && dpkg -i quarto-1.4.549-linux-amd64.deb \
+   && rm -rf /var/lib/apt/lists/* quarto-1.4.549-linux-amd64.deb
 ENV DEBIAN_FRONTEND=dialog
 
 # For docs
 RUN npm install --global yarn
 RUN pip install pydoc-markdown
+RUN pip install pyyaml
+RUN pip install colored

samples/dockers/Dockerfiles.md renamed to .devcontainer/README.md

@@ -1,28 +1,43 @@
-# README for AutoGen Docker Samples
+# Dockerfiles and Devcontainer Configurations for AutoGen
 
-Welcome to the `autogen/samples/dockers` directory! Here you'll find Dockerfiles that are essential for setting up your AutoGen development environment. Each Dockerfile is tailored for different use cases and requirements. Below is a brief overview of each and how you can utilize them effectively.
+Welcome to the `.devcontainer` directory! Here you'll find Dockerfiles and devcontainer configurations that are essential for setting up your AutoGen development environment. Each Dockerfile is tailored for different use cases and requirements. Below is a brief overview of each and how you can utilize them effectively.
+
+These configurations can be used with Codespaces and locally.
 
 ## Dockerfile Descriptions
 
-### Dockerfile.base
+### base
 
-- **Purpose**: This Dockerfile is designed for basic setups. It includes common Python libraries and essential dependencies required for general usage of AutoGen.
+- **Purpose**: This Dockerfile, i.e., `./Dockerfile`, is designed for basic setups. It includes common Python libraries and essential dependencies required for general usage of AutoGen.
 - **Usage**: Ideal for those just starting with AutoGen or for general-purpose applications.
-- **Building the Image**: Run `docker build -f Dockerfile.base -t autogen_base_img .` in this directory.
+- **Building the Image**: Run `docker build -f ./Dockerfile -t autogen_base_img .` in this directory.
+- **Using with Codespaces**: `Code > Codespaces > Click on +` By default + creates a Codespace on the current branch.
 
-### Dockerfile.full
+### full
 
-- **Purpose**: This Dockerfile is for advanced features. It includes additional dependencies and is configured for more complex or feature-rich AutoGen applications.
+- **Purpose**: This Dockerfile, i.e., `./full/Dockerfile` is for advanced features. It includes additional dependencies and is configured for more complex or feature-rich AutoGen applications.
 - **Usage**: Suited for advanced users who need the full range of AutoGen's capabilities.
-- **Building the Image**: Execute `docker build -f Dockerfile.full -t autogen_full_img .`.
+- **Building the Image**: Execute `docker build -f full/Dockerfile -t autogen_full_img .`.
+- **Using with Codespaces**: `Code > Codespaces > Click on ...> New with options > Choose "full" as devcontainer configuration`. This image may require a Codespace with at least 64GB of disk space.
 
-### Dockerfile.dev
+### dev
 
-- **Purpose**: Tailored for AutoGen project developers, this Dockerfile includes tools and configurations aiding in development and contribution.
+- **Purpose**: Tailored for AutoGen project developers, this Dockerfile, i.e., `./dev/Dockerfile` includes tools and configurations aiding in development and contribution.
 - **Usage**: Recommended for developers who are contributing to the AutoGen project.
-- **Building the Image**: Run `docker build -f Dockerfile.dev -t autogen_dev_img .`.
+- **Building the Image**: Run `docker build -f dev/Dockerfile -t autogen_dev_img .`.
+- **Using with Codespaces**: `Code > Codespaces > Click on ...> New with options > Choose "dev" as devcontainer configuration`. This image may require a Codespace with at least 64GB of disk space.
 - **Before using**: We highly encourage all potential contributors to read the [AutoGen Contributing](https://microsoft.github.io/autogen/docs/Contribute) page prior to submitting any pull requests.
 
+
+### studio
+
+- **Purpose**: Tailored for AutoGen project developers, this Dockerfile, i.e., `./studio/Dockerfile`, includes tools and configurations aiding in development and contribution.
+- **Usage**: Recommended for developers who are contributing to the AutoGen project.
+- **Building the Image**: Run `docker build -f studio/Dockerfile -t autogen_studio_img .`.
+- **Using with Codespaces**: `Code > Codespaces > Click on ...> New with options > Choose "studio" as devcontainer configuration`.
+- **Before using**: We highly encourage all potential contributors to read the [AutoGen Contributing](https://microsoft.github.io/autogen/docs/Contribute) page prior to submitting any pull requests.
+
+
 ## Customizing Dockerfiles
 
 Feel free to modify these Dockerfiles for your specific project needs. Here are some common customizations:
@@ -33,9 +48,9 @@ Feel free to modify these Dockerfiles for your specific project needs. Here are
     `FROM python:3.11-slim-bookworm` to `FROM python:3.10-slim-bookworm`
 - **Setting Environment Variables**: Add environment variables using the `ENV` command for any application-specific configurations. We have prestaged the line needed to inject your OpenAI_key into the docker environment as a environmental variable. Others can be staged in the same way. Just uncomment the line.
     `# ENV OPENAI_API_KEY="{OpenAI-API-Key}"` to `ENV OPENAI_API_KEY="{OpenAI-API-Key}"`
-- **Need a less "Advanced" Autogen build**: If the Dockerfile.full is to much but you need more than advanced then update this line in the Dockerfile.full file.
+- **Need a less "Advanced" Autogen build**: If the `./full/Dockerfile` is to much but you need more than advanced then update this line in the Dockerfile file.
 `RUN pip install pyautogen[teachable,lmm,retrievechat,mathchat,blendsearch] autogenra` to install just what you need. `RUN pip install pyautogen[retrievechat,blendsearch] autogenra`
-- **Can't Dev without your favorite CLI tool**: if you need particular OS tools to be installed in your Docker container you can add those packages here right after the sudo for the Dockerfile.base and Dockerfile.full files. In the example below we are installing net-tools and vim to the environment.
+- **Can't Dev without your favorite CLI tool**: if you need particular OS tools to be installed in your Docker container you can add those packages here right after the sudo for the `./base/Dockerfile` and `./full/Dockerfile` files. In the example below we are installing net-tools and vim to the environment.
 
     ```code
     RUN apt-get update \

samples/dockers/Dockerfile.dev renamed to .devcontainer/dev/Dockerfile

@@ -4,7 +4,7 @@ FROM python:3.11-slim-bookworm
 # Update and install necessary packages
 RUN apt-get update && apt-get -y update
 # added vim and nano for convenience
-RUN apt-get install -y sudo git npm vim nano curl
+RUN apt-get install -y sudo git npm vim nano curl wget
 
 # Setup a non-root user 'autogen' with sudo access
 RUN adduser --disabled-password --gecos '' autogen
@@ -32,6 +32,14 @@ RUN sudo pip install pydoc-markdown
 RUN cd website
 RUN yarn install --frozen-lockfile --ignore-engines
 
+RUN arch=$(arch | sed s/aarch64/arm64/ | sed s/x86_64/amd64/) && \
+    wget -q https://github.com/quarto-dev/quarto-cli/releases/download/v1.4.549/quarto-1.4.549-linux-${arch}.tar.gz && \
+    mkdir -p /home/autogen/quarto/ && \
+    tar -xzf quarto-1.4.549-linux-${arch}.tar.gz --directory /home/autogen/quarto/ && \
+    rm quarto-1.4.549-linux-${arch}.tar.gz
+
+ENV PATH="${PATH}:/home/autogen/quarto/quarto-1.4.549/bin/"
+
 # Exposes the Yarn port for Docusaurus
 EXPOSE 3000
 

.devcontainer/dev/devcontainer.json

@@ -0,0 +1,3 @@
+{
+    "dockerFile": "Dockerfile"
+}

.devcontainer/devcontainer.json

@@ -4,7 +4,8 @@
             "extensions": [
                 "ms-python.python",
                 "ms-toolsai.jupyter",
-                "visualstudioexptteam.vscodeintellicode"
+                "visualstudioexptteam.vscodeintellicode",
+                "GitHub.copilot"
             ],
             "settings": {
                 "terminal.integrated.profiles.linux": {

samples/dockers/Dockerfile.full renamed to .devcontainer/full/Dockerfile

@@ -0,0 +1,3 @@
+{
+    "dockerFile": "Dockerfile"
+}

.devcontainer/full/devcontainer.json

@@ -0,0 +1,22 @@
+#-------------------------------------------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See LICENSE file in the project root for license information.
+#-------------------------------------------------------------------------------------------------------------
+
+FROM mcr.microsoft.com/vscode/devcontainers/python:3.10
+
+#
+# Update the OS and maybe install packages
+#
+ENV DEBIAN_FRONTEND=noninteractive
+RUN apt-get update \
+   && apt-get upgrade -y  \
+   && apt-get -y install --no-install-recommends build-essential npm \
+   && apt-get autoremove -y \
+   && apt-get clean -y \
+   && rm -rf /var/lib/apt/lists/*
+ENV DEBIAN_FRONTEND=dialog
+
+# For docs
+RUN npm install --global yarn
+RUN pip install pydoc-markdown

.devcontainer/studio/Dockerfile

@@ -0,0 +1,21 @@
+{
+    "customizations":  {
+        "vscode": {
+            "extensions": [
+                "ms-python.python",
+                "ms-toolsai.jupyter",
+                "visualstudioexptteam.vscodeintellicode"
+            ],
+            "settings": {
+                "terminal.integrated.profiles.linux": {
+                    "bash": {
+                        "path": "/bin/bash"
+                    }
+                },
+                "terminal.integrated.defaultProfile.linux": "bash"
+            }
+        }
+    },
+    "dockerFile": "Dockerfile",
+    "updateContentCommand": "cd samples/apps/autogen-studio && pip install -e . && sudo npm install -g gatsby-cli && cd frontend && yarn install && yarn build"
+}

.devcontainer/studio/devcontainer.json

@@ -5,30 +5,31 @@ name: Build
 
 on:
   push:
-    branches: ['main']
+    branches: ["main"]
     paths:
-      - 'autogen/**'
-      - 'test/**'
-      - '.github/workflows/build.yml'
-      - 'setup.py'
+      - "autogen/**"
+      - "test/**"
+      - ".github/workflows/build.yml"
+      - "setup.py"
   pull_request:
-    branches: ['main']
+    branches: ["main"]
   merge_group:
     types: [checks_requested]
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref }}
   cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
-
+permissions: {}
 jobs:
   build:
-
     runs-on: ${{ matrix.os }}
+    env:
+      AUTOGEN_USE_DOCKER: ${{ matrix.os != 'ubuntu-latest'  && 'False' }}
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-latest, macos-latest, windows-2019]
-        python-version: ["3.8", "3.9", "3.10", "3.11"]
+        os: [ubuntu-latest, macos-latest, windows-latest]
+        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{ matrix.python-version }}
@@ -40,15 +41,27 @@ jobs:
           python -m pip install --upgrade pip wheel
           pip install -e .
           python -c "import autogen"
-          pip install -e. pytest mock
-      - name: Test with pytest
-        if: matrix.python-version != '3.10'
+          pip install pytest mock
+          pip install jupyter-client ipykernel
+          python -m ipykernel install --user --name python3
+      - name: Set AUTOGEN_USE_DOCKER based on OS
+        shell: bash
+        run: |
+          if [[ ${{ matrix.os }} != ubuntu-latest ]]; then
+            echo "AUTOGEN_USE_DOCKER=False" >> $GITHUB_ENV
+          fi
+      - name: Test with pytest skipping openai tests
+        if: matrix.python-version != '3.10' && matrix.os == 'ubuntu-latest'
         run: |
           pytest test --skip-openai
+      - name: Test with pytest skipping openai and docker tests
+        if: matrix.python-version != '3.10' && matrix.os != 'ubuntu-latest'
+        run: |
+          pytest test --skip-openai --skip-docker
       - name: Coverage
         if: matrix.python-version == '3.10'
         run: |
-          pip install -e .[test]
+          pip install -e .[test,redis]
           coverage run -a -m pytest test --ignore=test/agentchat/contrib --skip-openai
           coverage xml
       - name: Upload coverage to Codecov

.github/workflows/build.yml

@@ -11,7 +11,11 @@ on:
       - 'test/agentchat/contrib/**'
       - '.github/workflows/contrib-openai.yml'
       - 'setup.py'
-
+permissions: {}
+    # actions: read
+  # checks: read
+  # contents: read
+  # deployments: read
 jobs:
   RetrieveChatTest:
     strategy:
@@ -217,3 +221,81 @@ jobs:
         with:
           file: ./coverage.xml
           flags: unittests
+  WebSurfer:
+    strategy:
+      matrix:
+        os: [ubuntu-latest]
+        python-version: ["3.11"]
+    runs-on: ${{ matrix.os }}
+    environment: openai1
+    steps:
+      # checkout to pr branch
+      - name: Checkout
+        uses: actions/checkout@v3
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install packages and dependencies
+        run: |
+          docker --version
+          python -m pip install --upgrade pip wheel
+          pip install -e .[websurfer]
+          python -c "import autogen"
+          pip install coverage pytest
+      - name: Coverage
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
+          AZURE_OPENAI_API_BASE: ${{ secrets.AZURE_OPENAI_API_BASE }}
+          OAI_CONFIG_LIST: ${{ secrets.OAI_CONFIG_LIST }}
+          BING_API_KEY: ${{ secrets.BING_API_KEY }}
+        run: |
+          coverage run -a -m pytest test/agentchat/contrib/test_web_surfer.py
+          coverage xml
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v3
+        with:
+          file: ./coverage.xml
+          flags: unittests
+  ContextHandling:
+      strategy:
+        matrix:
+          os: [ubuntu-latest]
+          python-version: ["3.11"]
+      runs-on: ${{ matrix.os }}
+      environment: openai1
+      steps:
+        # checkout to pr branch
+        - name: Checkout
+          uses: actions/checkout@v3
+          with:
+            ref: ${{ github.event.pull_request.head.sha }}
+        - name: Set up Python ${{ matrix.python-version }}
+          uses: actions/setup-python@v4
+          with:
+            python-version: ${{ matrix.python-version }}
+        - name: Install packages and dependencies
+          run: |
+            docker --version
+            python -m pip install --upgrade pip wheel
+            pip install -e .
+            python -c "import autogen"
+            pip install coverage pytest
+        - name: Coverage
+          env:
+            OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+            AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
+            AZURE_OPENAI_API_BASE: ${{ secrets.AZURE_OPENAI_API_BASE }}
+            OAI_CONFIG_LIST: ${{ secrets.OAI_CONFIG_LIST }}
+            BING_API_KEY: ${{ secrets.BING_API_KEY }}
+          run: |
+            coverage run -a -m pytest test/agentchat/contrib/capabilities/test_context_handling.py
+            coverage xml
+        - name: Upload coverage to Codecov
+          uses: codecov/codecov-action@v3
+          with:
+            file: ./coverage.xml
+            flags: unittests