Update config_list_from_json function to support YAML parsing #2560

Mai0313 · 2024-05-01T07:22:16Z

Why are these changes needed?

Add Yaml file supported for OAI_CONFIG_LIST by simply changing from json.load to yaml.safe_load

Related issue number

#2559

Checks

I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
I've added tests (if relevant) corresponding to the changes introduced in this PR.
I've made sure all auto checks have passed.

Mai0313 · 2024-05-01T07:24:00Z

@microsoft-github-policy-service agree

@microsoft-github-policy-service agree company="Mediatek"

Mai0313 · 2024-05-02T00:08:00Z

Idk why there is no yaml module, i thought it was a default module for Python

ekzhu

Thanks for the PR! Could you also update the documentation: website/docs/topics/llm_configuration.ipynb

codecov-commenter · 2024-05-06T17:26:34Z

Codecov Report

Attention: Patch coverage is 20.00000% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 15.75%. Comparing base (372ac1e) to head (b97b99d).
Report is 3 commits behind head on main.

Files	Patch %	Lines
autogen/oai/openai_utils.py	20.00%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2560       +/-   ##
===========================================
- Coverage   33.11%   15.75%   -17.37%     
===========================================
  Files          86       86               
  Lines        9108     9109        +1     
  Branches     1938     2090      +152     
===========================================
- Hits         3016     1435     -1581     
- Misses       5837     7626     +1789     
+ Partials      255       48      -207

Flag	Coverage Δ
unittests	`15.75% <20.00%> (-17.37%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Mai0313 · 2024-05-06T17:34:06Z

website/docs/topics/llm_configuration.ipyn

oh sorry, it is my first time to use mdx format, is there anyway I can preview the changes before commit?
I tried this extension, but it seems not working for me.

- Add pytest for load_config_from_json for yaml format - Add docs for website - Add pyyaml to setup.py

gitguardian · 2024-05-06T18:18:07Z

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
10404695	Triggered	Generic High Entropy Secret	`b97b99d`	test/oai/test_utils.py	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secret safely. Learn here the best practices.
Revoke and rotate this secret.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

Mai0313 · 2024-05-06T18:20:40Z

@ekzhu
It is my first time to use mdx format and I do not know where I can preview the changes for mdx, so basically, the way I have done is copy-paste the previous content and changed to yaml version.

BTW, I have added pyyaml to setup.py to make sure no one get error when loading config, since we don't use json module but yaml.

Additionally, I have implemented pytest for the YAML version. Specifically, I utilized pytest.mark.parametrize to minimize code duplication.

ekzhu · 2024-05-13T10:38:25Z

It is my first time to use mdx format and I do not know where I can preview the changes for mdx, so basically, the way I have done is copy-paste the previous content and changed to yaml version.

See https://microsoft.github.io/autogen/docs/contributor-guide/documentation, the process_notebooks.py script will generate the MDX files.

ekzhu · 2024-05-13T10:44:13Z

setup.py

@@ -25,6 +25,7 @@
    # Disallowing 2.6.0 can be removed when this is fixed https://github.com/pydantic/pydantic/issues/8705
    "pydantic>=1.10,<3,!=2.6.0",  # could be both V1 and V2
    "docker",
+    "pyyaml",


We can add pyyaml to test dependencies. But for install dependency let's move this to an optional dependency like those below.

actually I dont see any of test dependency or optional dependency under setup.py
should I put it under test from extra_require?

You can add it like here:

autogen/setup.py

Lines 60 to 70 in 82903f5

extra_require = {

"test": [

"ipykernel",

"nbconvert",

"nbformat",

"pre-commit",

"pytest-cov>=5",

"pytest-asyncio",

"pytest>=6.1.1,<8",

"pandas",

],

ekzhu · 2024-05-13T10:46:37Z

autogen/oai/openai_utils.py

        else:
            # Else, it should be a JSON string by itself.
-            json_str = env_str
-        config_list = json.loads(json_str)
+            json_or_yaml_str = env_str


If the yaml dependency is not installed, we can fall back to use json module. We can do this by checking for yaml import error during import, and use that information to choose whether to use yaml load or json load.

oh, it sounds good

ekzhu · 2024-05-13T10:47:03Z

autogen/oai/openai_utils.py

@@ -475,21 +476,38 @@ def config_list_from_json(

    Args:
        env_or_file (str): The name of the environment variable, the filename, or the environment variable of the filename
-            that containing the JSON data.
+            that containing the JSON/YAML data.


Add explanation for installing pyyaml.

…from environment variable or file

thinkall · 2024-07-07T13:19:08Z

Some tests failed, could you fix them? For documentation, you can follow https://microsoft.github.io/autogen/docs/contributor-guide/documentation to check it locally.

Update config_list_from_json function to support YAML parsing

c4166cd

Mai0313 added 2 commits May 1, 2024 07:31

Add OAI_CONFIG_LIST_yaml_sample and update README and blog file

ad32f63

Merge branch 'main' into feat/yaml_support_for_config

3e04d2c

ekzhu requested changes May 2, 2024

View reviewed changes

ekzhu requested review from sonichi, BeibinLi and thinkall May 2, 2024 05:58

Merge branch 'main' into feat/yaml_support_for_config

1a42c46

Feat:

b97b99d

- Add pytest for load_config_from_json for yaml format - Add docs for website - Add pyyaml to setup.py

Merge branch 'main' into feat/yaml_support_for_config

6abc696

Mai0313 requested a review from ekzhu May 11, 2024 08:46

Merge branch 'main' into feat/yaml_support_for_config

1f0f407

ekzhu reviewed May 13, 2024

View reviewed changes

Mai0313 added 2 commits May 31, 2024 00:00

Merge branch 'main' into feat/yaml_support_for_config

15e4896

chore: Refactor openai_utils.py to support loading JSON or YAML data …

2410ebc

…from environment variable or file

Mai0313 had a problem deploying to openai1 June 29, 2024 16:42 — with GitHub Actions Failure

thinkall had a problem deploying to openai1 August 6, 2024 12:59 — with GitHub Actions Failure

Mai0313 and others added 2 commits August 8, 2024 16:48

Merge branch 'main' into feat/yaml_support_for_config

e385c2b

Merge branch 'main' into feat/yaml_support_for_config

ccdf75a

thinkall had a problem deploying to openai1 September 13, 2024 07:49 — with GitHub Actions Failure

Mai0313 closed this by deleting the head repository Sep 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update config_list_from_json function to support YAML parsing #2560

Update config_list_from_json function to support YAML parsing #2560

Mai0313 commented May 1, 2024

Mai0313 commented May 1, 2024

Mai0313 commented May 2, 2024

ekzhu left a comment

codecov-commenter commented May 6, 2024 •

edited

Loading

Mai0313 commented May 6, 2024

gitguardian bot commented May 6, 2024 •

edited

Loading

Mai0313 commented May 6, 2024

ekzhu commented May 13, 2024

ekzhu May 13, 2024

Mai0313 May 30, 2024

thinkall Jul 7, 2024

ekzhu May 13, 2024

Mai0313 May 17, 2024

ekzhu May 13, 2024

thinkall commented Jul 7, 2024

	extra_require = {
	"test": [
	"ipykernel",
	"nbconvert",
	"nbformat",
	"pre-commit",
	"pytest-cov>=5",
	"pytest-asyncio",
	"pytest>=6.1.1,<8",
	"pandas",
	],

Update config_list_from_json function to support YAML parsing #2560

Update config_list_from_json function to support YAML parsing #2560

Conversation

Mai0313 commented May 1, 2024

Why are these changes needed?

Related issue number

Checks

Mai0313 commented May 1, 2024

Mai0313 commented May 2, 2024

ekzhu left a comment

Choose a reason for hiding this comment

codecov-commenter commented May 6, 2024 • edited Loading

Codecov Report

Mai0313 commented May 6, 2024

gitguardian bot commented May 6, 2024 • edited Loading

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Mai0313 commented May 6, 2024

ekzhu commented May 13, 2024

ekzhu May 13, 2024

Choose a reason for hiding this comment

Mai0313 May 30, 2024

Choose a reason for hiding this comment

thinkall Jul 7, 2024

Choose a reason for hiding this comment

ekzhu May 13, 2024

Choose a reason for hiding this comment

Mai0313 May 17, 2024

Choose a reason for hiding this comment

ekzhu May 13, 2024

Choose a reason for hiding this comment

thinkall commented Jul 7, 2024

codecov-commenter commented May 6, 2024 •

edited

Loading

gitguardian bot commented May 6, 2024 •

edited

Loading