Code extraction fix #1834

PranitModak · 2024-03-01T17:49:58Z

Why are these changes needed?

Current Code Extractor is not able to extract single like codes given by agent.
It ideally should be able to extract any code given by the agent if it has a correct markdown format.

Related issue number

Checks

I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
I've added tests (if relevant) corresponding to the changes introduced in this PR.
I've made sure all auto checks have passed.

Current regex do not extract single line codes given by agent like below.

Please install required modules with ```sh pip3 install Flask flask_sqlalchemy flask_bcrypt flask_login ``` and execute the following Python code which uses Flask:

ekzhu · 2024-03-01T18:05:19Z

Could you write some tests for this? Also, could you check if we also need to update autogen.coding.markdown_code_extractor.MarkdownCodeExtractor?

sonichi · 2024-03-01T18:34:12Z

I think @BeibinLi or @LittleLittleCloud added some way to extract single line of code before.

PranitModak · 2024-03-01T19:57:21Z

Could you write some tests for this? Also, could you check if we also need to update autogen.coding.markdown_code_extractor.MarkdownCodeExtractor?

yes have tested using MarkdownCodeExtractor and it works perfectly now.

Test with a user proxy agent and a assistant with a system message containing a single line sh example like below :

"Give all installations commands in a single line for the code you suggest in this format ```sh pip3 install Flask flask_sqlalchemy flask_bcrypt flask_login ``` "

BeibinLi · 2024-03-01T21:46:04Z

The new changes allow code block with 3 "`" signs to be written in a single line, which is good.

However, it still cannot resolve standard code blocks with one "`" sign in a single line.

I have a single-line detection in line 124:

code_pattern = re.compile(CODE_BLOCK_PATTERN + r"|`([^`]+)`")

Should we combine your changes with the single line detection logic we had earlier?

I am open to any suggestions regarding CODE_BLOCK_PATTERN or extract_code function. We can simplify the function, remove the parameter detect_single_line_code, or set the parameter's default to True.

Another question, what would be the result for:

```print("hello world")```

```x = y + 1```

BeibinLi

If we allow 0 or more "\n" characters. Should we also allow 0 or more "\r" characters?

They are the same at the end of the day (depends on which encoding method or operation system is used for the text file).

ekzhu · 2024-03-03T21:27:04Z

I think we need to be a bit more careful in changing the default behavior of code block extraction. Currently we only extract well-formed multi-line code blocks surrounded with three backticks "``". This is instruction is not hard for LLMs like GPT-4 to follow but could be an issue for open source LMs. I suggest we provide options in the code executors to turn this on, rather than changing the default behavior.

ekzhu · 2024-03-03T21:28:25Z

@marklysze @GregorD1A1 how's your experience using open source models when it comes to follow code block formatting instructions.

marklysze · 2024-03-04T03:20:20Z

@marklysze @GregorD1A1 how's your experience using open source models when it comes to follow code block formatting instructions.

In my testing of open source models with code generation (based on these notebooks):
https://github.com/microsoft/autogen/blob/tutorial/website/docs/tutorial/code-executors.ipynb
https://github.com/microsoft/autogen/blob/main/notebook/agentchat_function_call.ipynb

I found a mix of outputs. Enhanced support for #3 and #4 below would be good. I'm not sure #2 can be accommodated for easily.

Note: I'll space out the "```" so it shows correctly in this comment.

Some with what we expect:

` ` `python
def fibonacci(n):
    if n <= 1:
        return n
    else:
        return fibonacci(n-1) + fibonacci(n-2)

fibonacci(14)
` ` `

Apart from those, there were a mix of outputs.

Wrong character and no ending characters.

'''

def fibonacci(n):
    if n <= 1:
        return n
    else:
        return(fibonacci(n-1) + fibonacci(n-2))
    
print('The 14th Fibonacci number is :', fibonacci(14))

"""

def fibo(n):
    if n < 2:
        return 1
    else:
        return fibo(n-1) + fibo(n-2)
print(fibo(14))

Correct characters but outputs multiple blocks of code with comments in-between:

` ` ` 
pip install fibonacci
` ` `
Alternatively, you can use a different implementation of the Fibonacci sequence, such as the one provided by the `scipy` package:
` ` `python
from scipy.math import fibonacci

x = fibonacci(14)
print(x)
` ` `

All in a single line

` ` `python import math  # import the math library def fibonacci(n): if n == 0 or n == 1: return n else: return (fibonacci(n-1) + fibonacci(n-2)) % 10 ** 6 print(f"The 14th Fibonacci number is {fibonacci(14)}")  # calculate and print the output in a separate code block ` ` `

See the output files that have been generated here for examples:
https://github.com/marklysze/AutoGenCodeTesting/tree/master/results

Grigorij-Dudnik · 2024-03-04T12:18:22Z

@ekzhu I didn't tested code extraction for open source models. Hovewer, at some point I come to the conclusion, that it's better to... not use code extraction at all. Instead, I defined few tools as "insert code after line x", "replace code in lines", "create new file with code". All the file operations defined in python level inside the tools. Such function calling approach (at least at my application, where I'm creating web app) provided much better results - while with code extractor there was often situation when LLM removed half of the existing file ocasionally, with function calling were not so big mistakes. Also, function calling approach is safier, LLM can do only predefined list of things, so not requires docker to protect OS. So my advice based on practical experience - use function calling instead of code extraction when possible.

Althrough I understand there are different applications of Autogen and sometimes code extractor is needed, so it's good @PranitModak improving that feature. To test it with open source LLM, I recommend to use server feature in LM Studio.

ekzhu · 2024-03-04T18:18:41Z

@ekzhu I didn't tested code extraction for open source models. Hovewer, at some point I come to the conclusion, that it's better to... not use code extraction at all. Instead, I defined few tools as "insert code after line x", "replace code in lines", "create new file with code". All the file operations defined in python level inside the tools. Such function calling approach (at least at my application, where I'm creating web app) provided much better results - while with code extractor there was often situation when LLM removed half of the existing file ocasionally, with function calling were not so big mistakes. Also, function calling approach is safier, LLM can do only predefined list of things, so not requires docker to protect OS. So my advice based on practical experience - use function calling instead of code extraction when possible.

Althrough I understand there are different applications of Autogen and sometimes code extractor is needed, so it's good @PranitModak improving that feature. To test it with open source LLM, I recommend to use server feature in LM Studio.

Thanks! Would you be able to contribute a notebook on using function calling for writing code? Are you using function calling to execute the written code as well?

Grigorij-Dudnik · 2024-03-09T10:28:29Z

Thanks! Would you be able to contribute a notebook on using function calling for writing code? Are you using function calling to execute the written code as well?

@ekzhu ok, I can prepare such. I'm not using function calling to run code, in my case I used it to improve code for FastApi and Vue.js applications, both of which was running as separate processes and got automatically reloaded after changes.

ekzhu · 2024-03-09T12:00:10Z

I see. I a just trying to understand the usage case. So is it right to say that the use case is given a code file to the agent, and the agent suggests changes to the code via tool call, which then makes the actual changes. Once the changes are made, the application whose source code is changed got reloaded.

Grigorij-Dudnik · 2024-03-11T15:42:45Z

Yes, that's right. I'll try to explain the thing in notebook.

ekzhu · 2024-03-14T18:40:34Z

@PranitModak as discussed above, let's make the this optional in the markdown code extractor class in autogen.coding.

gitguardian · 2024-07-20T21:21:57Z

⚠️ GitGuardian has uncovered 96 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
12853598	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	`eff19ac`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`06a0a5d`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`0524c77`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`d7ea410`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`e43a86c`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`841ed31`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`802f099`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`9a484d8`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`e973ac3`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`89650e7`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`e07b06b`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`abe4c41`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`7362fb9`	.github/workflows/dotnet-release.yml	View secret
12853599	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`abad9ff`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`c7bb588`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`b97b99d`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
12853602	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
11616921	Triggered	Generic High Entropy Secret	`a86d0fd`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`394561b`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`3eac646`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`f45b553`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`6563248`	notebook/agentchat_agentops.ipynb	View secret
12853598	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853598	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	`954ca45`	.github/workflows/dotnet-build.yml	View secret
12853599	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853599	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`3b79cc6`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`11baa52`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	`3b79cc6`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`11baa52`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`11baa52`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`3b79cc6`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret

and 16 others.

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

jackgerrits · 2024-09-23T19:10:03Z

@PranitModak are you still interested in working on this?

jackgerrits · 2024-09-25T20:13:51Z

Looks like this is dormant, if you want to pick this up again please feel free to reopen. Thanks!

PranitModak added 2 commits March 1, 2024 17:41

fix for single line markdown codes like shell commands.

85e8c52

Update code_utils.py

a32e545

PranitModak had a problem deploying to openai1 March 1, 2024 17:50 — with GitHub Actions Failure

ekzhu added the code-execution execute generated code label Mar 1, 2024

sonichi requested review from BeibinLi, abhaymathur21 and ekzhu March 1, 2024 18:33

BeibinLi requested changes Mar 1, 2024

View reviewed changes

ekzhu requested review from Grigorij-Dudnik and marklysze March 3, 2024 21:28

jackgerrits closed this Sep 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Code extraction fix #1834

Code extraction fix #1834

PranitModak commented Mar 1, 2024 •

edited

Loading

ekzhu commented Mar 1, 2024

sonichi commented Mar 1, 2024

PranitModak commented Mar 1, 2024 •

edited

Loading

BeibinLi commented Mar 1, 2024 •

edited

Loading

BeibinLi left a comment

ekzhu commented Mar 3, 2024 •

edited

Loading

ekzhu commented Mar 3, 2024

marklysze commented Mar 4, 2024 •

edited

Loading

Grigorij-Dudnik commented Mar 4, 2024

ekzhu commented Mar 4, 2024

Grigorij-Dudnik commented Mar 9, 2024

ekzhu commented Mar 9, 2024

Grigorij-Dudnik commented Mar 11, 2024

ekzhu commented Mar 14, 2024

gitguardian bot commented Jul 20, 2024 •

edited

Loading

jackgerrits commented Sep 23, 2024

jackgerrits commented Sep 25, 2024

Code extraction fix #1834

Code extraction fix #1834

Conversation

PranitModak commented Mar 1, 2024 • edited Loading

Why are these changes needed?

Related issue number

Checks

ekzhu commented Mar 1, 2024

sonichi commented Mar 1, 2024

PranitModak commented Mar 1, 2024 • edited Loading

BeibinLi commented Mar 1, 2024 • edited Loading

BeibinLi left a comment

Choose a reason for hiding this comment

ekzhu commented Mar 3, 2024 • edited Loading

ekzhu commented Mar 3, 2024

marklysze commented Mar 4, 2024 • edited Loading

Grigorij-Dudnik commented Mar 4, 2024

ekzhu commented Mar 4, 2024

Grigorij-Dudnik commented Mar 9, 2024

ekzhu commented Mar 9, 2024

Grigorij-Dudnik commented Mar 11, 2024

ekzhu commented Mar 14, 2024

gitguardian bot commented Jul 20, 2024 • edited Loading

⚠️ GitGuardian has uncovered 96 secrets following the scan of your pull request.

jackgerrits commented Sep 23, 2024

jackgerrits commented Sep 25, 2024

PranitModak commented Mar 1, 2024 •

edited

Loading

PranitModak commented Mar 1, 2024 •

edited

Loading

BeibinLi commented Mar 1, 2024 •

edited

Loading

ekzhu commented Mar 3, 2024 •

edited

Loading

marklysze commented Mar 4, 2024 •

edited

Loading

gitguardian bot commented Jul 20, 2024 •

edited

Loading