feat: Add experimental AI PR review #187
Conversation
scott-codecov
requested review from
giovanni-guidini,
joseph-sentry and
matt-codecov
scott-codecov
force-pushed
the
scott/ai-pr-review
branch
from
3c516d0
to
e62488f
Compare
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #187 +/- ##
==========================================
- Coverage 98.34% 98.30% -0.04%
==========================================
Files 376 380 +4
Lines 28088 28366 +278
==========================================
+ Hits 27622 27886 +264
- Misses 466 480 +14
Flags with carried forward coverage won't be shown. Click here to find out more.
This change has been scanned for critical changes. Learn more |
Codecov Report
@@ Coverage Diff @@
## main #187 +/- ##
==========================================
- Coverage 98.37% 98.34% -0.04%
==========================================
Files 347 351 +4
Lines 27443 27721 +278
==========================================
+ Hits 26998 27262 +264
- Misses 445 459 +14
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #187 +/- ##
==========================================
- Coverage 98.37% 98.34% -0.04%
==========================================
Files 347 351 +4
Lines 27443 27721 +278
==========================================
+ Hits 26998 27262 +264
- Misses 445 459 +14
Flags with carried forward coverage won't be shown. Click here to find out more.
|
|
|
||
| async def fetch_diff(self) -> str: | ||
| async with self.torngit.get_client() as client: | ||
| diff = await self.torngit.api( |
There was a problem hiding this comment.
I'm curious why you decided to implement this here instead of in torngit.
It's the same request from get_pull_request (but I think you are using more of the data)
I understand that this feature is experimental so I think it's better to keep it as self-contained as possible. I'm not asking you to change anything, just curious
There was a problem hiding this comment.
The difference here is the Accept header below. By passing that header, GitHub will return the raw Git patch (which you can also get while logged in by appending .diff to the end of a normal PR URL).
|
|
||
| try: | ||
| data = json.loads(res) | ||
| except json.decoder.JSONDecodeError: |
There was a problem hiding this comment.
We should have metrics for these errors so we can track how successful this thing is (in terms of delivering any response, obviously it wouldn't measure how effective the response is)
There was a problem hiding this comment.
Added in 3de8687
|
Thanks for the review @giovanni-guidini! Would you mind taking a quick look at codecov/shared#87 as well so I can get the EDIT: nevermind - @joseph-sentry did it - thanks! |
Enable the feature in your
codecov.ymland you'll start receiving GPT-4 powered PR reviews. The YAML is checked in thePullSyncTaskand, if that feature is enabled, then it spins off a newAiPrReviewTaskwhich handles the interactions with GitHub and OpenAI. This is GitHub only right now since the PR review API is a bit specific to GitHub (it's commenting on specific lines on code).The GPT-4 prompt could probably use a lot of work as I have very little experience with this. It's currently only passing in the Git patch and the comments received back from the OpenAI API have been somewhat lackluster. Providing additional context about the codebase could potentially help here but is out-of-scope for this initial POC.
Depends on codecov/shared#87 for the YAML schema update
TODO: updatesharedSHA once the PR linked above has been merged tomain