Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

contribute a prompt benchmarking script #292

Merged
merged 2 commits into from
Aug 28, 2024
Merged

Conversation

devoncarew
Copy link
Member

  • contribute a prompt benchmarking script

This contributes a contribute a prompt benchmarking script; it gets label information for ~100 existing issues - assumed to be correctly labelled - and compares them against the labels predicted by the prompt. This will allow us to benchmark prompt changes to know whether we're improving things or not. Note that there are no prompt improvements in this PR (just some updates for area label changes).

The benchmarking can be re-run via dart tool/bench.dart.


  • I’ve reviewed the contributor guide and applied the relevant portions to this PR.
Contribution guidelines:

Note that many Dart repos have a weekly cadence for reviewing PRs - please allow for some latency before initial review feedback.

Copy link

PR Health

Breaking changes ✔️
Package Change Current Version New Version Needed Version Looking good?
Changelog Entry ✔️
Package Changed Files

Changes to files need to be accounted for in their respective changelogs.

Coverage ✔️
File Coverage

This check for test coverage is informational (issues shown here will not fail the PR).

API leaks ✔️

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package Leaked API symbols
License Headers ✔️
// Copyright (c) 2024, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.
Files
no missing headers

All source files should start with a license header.

Package publish validation ✔️
Package Version Status
package:dart_flutter_team_lints 3.2.0 already published at pub.dev
package:firehose 0.9.2 already published at pub.dev

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

@devoncarew devoncarew requested a review from mosuem August 27, 2024 16:23
pkgs/sdk_triage_bot/tool/bench.md Show resolved Hide resolved
@devoncarew devoncarew merged commit 183fdd0 into main Aug 28, 2024
28 checks passed
@devoncarew devoncarew deleted the prompt_benchmarking branch August 28, 2024 23:47
copybara-service bot pushed a commit to dart-lang/sdk that referenced this pull request Sep 13, 2024
…l, http, lints, mockito, shelf, test, test_descriptor, typed_data, web, webkit_inspection_protocol, yaml

Revisions updated by `dart tools/rev_sdk_deps.dart`.

args (https://github.com/dart-lang/args/compare/1a24d61..e623652):
  e623652  2024-08-26  Kevin Moore  Fix angle brackets in doc comments (dart-archive/args#281)

browser_launcher (https://github.com/dart-lang/browser_launcher/compare/fa98c77..0acd188):
  0acd188  2024-08-22  Srujan Gaddam  Add CHANGELOG entry for dart-archive/browser_launcher#63 and publish 1.2.2 (dart-archive/browser_launcher#64)

collection (https://github.com/dart-lang/collection/compare/0c1f829..24b75d8):
  24b75d8  2024-08-26  Kevin Moore  Fix angle brackets in doc comments (dart-archive/collection#358)

csslib (https://github.com/dart-lang/csslib/compare/192d720..d486627):
  d486627  2024-08-26  Kevin Moore  Fix angle brackets in doc comments (dart-archive/csslib#206)

ecosystem (https://github.com/dart-lang/ecosystem/compare/8626bff..4ba6f15):
  4ba6f15  2024-09-04  Devon Carew  update the prompt to include `needs-info` (dart-lang/ecosystem#296)
  3496f54  2024-09-02  Moritz  Continue with labeling after error with issue transfer (dart-lang/ecosystem#293)
  bb45d73  2024-09-02  Moritz  Run `dart pub global run mono_repo generate` (dart-lang/ecosystem#295)
  8d5860b  2024-09-01  dependabot[bot]  Bump the github-actions group with 3 updates (dart-lang/ecosystem#297)
  6de3e8c  2024-08-29  Devon Carew  fix calculation for the labels to apply (dart-lang/ecosystem#294)
  183fdd0  2024-08-28  Devon Carew  contribute a prompt benchmarking script (dart-lang/ecosystem#292)
  f7191b7  2024-08-28  Moritz  Transfer issues in `package:trebuchet` (dart-lang/ecosystem#291)
  5da8a7a  2024-08-26  Moritz  Add `package:trebuchet` to repo. (dart-lang/ecosystem#289)
  dd8d12a  2024-08-23  Nate Bosch  Add check label to github action job name (dart-lang/ecosystem#290)

html (https://github.com/dart-lang/html/compare/0da420c..5516387):
  5516387  2024-08-28  Kevin Moore  Drop Pair util class (dart-archive/html#255)
  50837e5  2024-08-28  Nate Bosch  Fix doc comments (dart-archive/html#256)

http (https://github.com/dart-lang/http/compare/b97b8dc..dfeecf0):
  dfeecf0  2024-08-28  Brian Quinlan  Convert a `java.lang.OutOfMemoryError` into a `ClientException` (dart-lang/http#1299)
  a41141d  2024-08-28  Brian Quinlan  cupertino_http: Use `sharedDarwinSource` (dart-lang/http#1290)
  51beb4e  2024-08-28  Yaroslav Vorobev  fix(http_client_conformance_tests): multipart server (dart-lang/http#1292)
  bbfce40  2024-08-28  Brian Quinlan  Add a reference to AdapterWebSocketChannel for WebSocketChannel users (dart-lang/http#1297)
  7f21111  2024-08-21  Moritz  Add PR Health workflow (dart-lang/http#1296)

lints (https://github.com/dart-lang/lints/compare/894b500..9e4fd7d):
  9e4fd7d  2024-09-10  Lenz Paul  Fixes formatting error in the rules.md table (dart-lang/lints#206)
  6580f83  2024-09-05  Sam Rawlins  Remove avoid_null_checks_in_equality_operators from recommended (dart-lang/lints#201)

mockito (https://github.com/dart-lang/mockito/compare/eb4d1da..d0fda0c):
  d0fda0c  2024-09-11  Sam Rawlins  Bump CI tasks to use Dart 3.5.0 as stable
  2259e10  2024-09-11  Sam Rawlins  Pass a Dart language version to the Dart formatter.
  d7a8334  2024-09-10  Sam Rawlins  mockito: stop using deprecated analyzer APIs.

shelf (https://github.com/dart-lang/shelf/compare/9f2dffe..d53a8f9):
  d53a8f9  2024-09-06  Nate Bosch  Prepare to publish shelf_static (dart-lang/shelf#443)
  e725c73  2024-09-06  mx1up  Loosen mime package constraint (dart-lang/shelf#441)
  efbc566  2024-09-05  Nate Bosch  Remove links to core types with generics (dart-lang/shelf#442)

test (https://github.com/dart-lang/test/compare/cd3dbd5..9a2d155):
  9a2d155b  2024-09-11  Nate Bosch  Run package:test tests on mac (dart-lang/test#2280)
  569c6eb6  2024-09-01  dependabot[bot]  Bump the github-actions group with 2 updates (dart-lang/test#2275)
  db8cf091  2024-08-28  Nate Bosch  Remove the Pair utility (dart-lang/test#2271)
  032ef1dc  2024-08-27  Simon Binder  Support dart2wasm in node.js tests (dart-lang/test#2259)
  90481cf8  2024-08-23  Nate Bosch  Limit which health checks run (dart-lang/test#2270)
  337c7fb0  2024-08-21  Moritz  Update publish.yaml (dart-lang/test#2269)

test_descriptor (https://github.com/dart-lang/test_descriptor/compare/90743bc..3b85d38):
  3b85d38  2024-08-26  Kevin Moore  Fix angle brackets in doc comments (dart-lang/test_descriptor#70)

typed_data (https://github.com/dart-lang/typed_data/compare/365468a..2bb9e6e):
  2bb9e6e  2024-08-26  Kevin Moore  Require Dart 3.5, use TypeDataList in many places (dart-archive/typed_data#92)

web (https://github.com/dart-lang/web/compare/fb30192..933a37d):
  933a37d  2024-09-10  Srujan Gaddam  Update dart_style and use latestLanguageVersion (dart-lang/web#301)

webkit_inspection_protocol (https://github.com/google/webkit_inspection_protocol.dart/compare/119b877..e7418d7):
  e7418d7  2024-08-26  Kevin Moore  Update min SDK and a number of lints (google/webkit_inspection_protocol.dart#128)

yaml (https://github.com/dart-lang/yaml/compare/a645c39..b3d299e):
  b3d299e  2024-08-28  Kevin Moore  Drop Pair util class (dart-lang/yaml#169)

Change-Id: Ic3461659a40427c0deaafab8c79cade332f76edb
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/385302
Reviewed-by: Konstantin Shcheglov <[email protected]>
Auto-Submit: Devon Carew <[email protected]>
Commit-Queue: Konstantin Shcheglov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants