Skip to content

Eval hackathon #752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
VictorSanh merged 26 commits into from
Jun 29, 2022
Merged

Eval hackathon #752

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
4d4e3d4
test prompts for newsco
fyvo Apr 27, 2022
70823d0
sample prompts for zero-shot mt
fyvo Apr 27, 2022
4c5d800
one more pattern for wmt14/en-fr
fyvo Apr 27, 2022
e642b56
Fix an inverted fr-en prompt for wmt14
fyvo Apr 28, 2022
5a9c93f
Merge branch 'bigscience-workshop:eval-hackathon' into eval-hackathon
fyvo Apr 28, 2022
7272271
Prompts for WMT14 de-en, adapted directly from fr-en
fyvo Apr 28, 2022
2e0f4fa
Merge branch 'bigscience-workshop:eval-hackathon' into eval-hackathon
fyvo May 7, 2022
27d6a30
Merge branch 'bigscience-workshop:eval-hackathon' into eval-hackathon
fyvo May 8, 2022
a47afc1
Merge branch 'bigscience-workshop:eval-hackathon' into eval-hackathon
fyvo May 11, 2022
28cb0e8
Merge branch 'bigscience-workshop:eval-hackathon' into eval-hackathon
fyvo May 16, 2022
f086b81
fixed duplicate uuids, add a new set of prompts
fyvo May 16, 2022
fc14336
restore the original template
fyvo May 17, 2022
815c09b
Merge branch 'bigscience-workshop:eval-hackathon' into eval-hackathon
fyvo May 19, 2022
bbe3981
Merge branch 'bigscience-workshop:eval-hackathon' into eval-hackathon
fyvo May 20, 2022
7ebc5ba
Merge branch 'bigscience-workshop:eval-hackathon' into eval-hackathon
fyvo May 24, 2022
541cc2c
remove prompts in the French language to simplify copies
fyvo May 24, 2022
ba82480
remove faulty german prompts
fyvo May 24, 2022
b6a4fa9
Add prompts for WMT14 hi-En
fyvo May 25, 2022
62bcefe
Prompts for wmt14 cs-en
fyvo May 27, 2022
dd1f445
prompts for wmt14-wmt19 all news tasks
fyvo May 27, 2022
9711123
Merge branch 'bigscience-workshop:eval-hackathon' into eval-hackathon
fyvo Jun 6, 2022
fb9d6ef
added 4 new glm like prompts for wmt* tasks
fyvo Jun 16, 2022
14a8ff4
Merge branch 'bigscience-workshop:eval-hackathon' into eval-hackathon
fyvo Jun 27, 2022
5e3b223
Merge branch 'bigscience-workshop:eval-hackathon' into eval-hackathon
fyvo Jun 29, 2022
443a755
Small change to the translate-* family of prompts - now Translate thi…
fyvo Jun 29, 2022
2608a38
fix names
VictorSanh Jun 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file modified promptsource/templates/cc_news/templates.yaml
100755 → 100644
View file
Open in desktop
Empty file.
254 changes: 254 additions & 0 deletions promptsource/templates/wmt14/de-en/templates.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,254 @@
dataset: wmt14
subset: de-en
templates:
08ddea55-1710-4615-bbfa-fe5803e21e43: !Template
answer_choices: null
id: 08ddea55-1710-4615-bbfa-fe5803e21e43
jinja: 'If the German version says: {{translation["de"]}}; then the English version
should say:

||| {{translation["en"]}}'
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: version-de-en-source+target
reference: ''
0bc0e46c-d1fe-4bc9-99d1-9b61aa42cd02: !Template
answer_choices: null
id: 0bc0e46c-d1fe-4bc9-99d1-9b61aa42cd02
jinja: 'If the English version says: {{translation["en"]}}; then the German version
should say:

||| {{translation["de"]}}'
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: version-en-de-source+target
reference: ''
1fbf99d9-980a-4bf8-b30e-dcf9e3ad5feb: !Template
answer_choices: null
id: 1fbf99d9-980a-4bf8-b30e-dcf9e3ad5feb
jinja: "Translate German into English: {{translation[\"de\"]}} \n||| {{translation[\"\
en\"]}}"
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: translate-de-en-source+target
reference: https://arxiv.org/pdf/1910.10683.pdf
2033cc05-3ef7-4c93-9684-90a618390f4b: !Template
answer_choices: null
id: 2033cc05-3ef7-4c93-9684-90a618390f4b
jinja: "What is the English translation of the German sentence: {{translation[\"\
fr\"]}} \n||| {{translation[\"en\"]}}"
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: gpt-3-de-en-src+target
reference: ''
280bc983-9f20-4e80-ab02-5f333df90a42: !Template
answer_choices: null
id: 280bc983-9f20-4e80-ab02-5f333df90a42
jinja: "What is the English translation of : {{translation[\"de\"]}} \n||| {{translation[\"\
en\"]}}"
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: gpt-3-de-en-target
reference: ''
3abec84a-592d-4282-b605-406fe4a6cc33: !Template
answer_choices: null
id: 3abec84a-592d-4282-b605-406fe4a6cc33
jinja: 'Comment dit-on {{translation["de"]}} en anglais ?

||| {{translation["en"]}}'
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: comment_dit_on-de-en-target
reference: ''
41e7647e-2b9f-4d86-8987-5abe70000362: !Template
answer_choices: null
id: 41e7647e-2b9f-4d86-8987-5abe70000362
jinja: "Translate into English: {{translation[\"de\"]}} \n||| {{translation[\"\
en\"]}}"
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: translate-de-en-target
reference: ''
43dc1b77-e8ea-4dc8-8a12-0abc3b0dbba0: !Template
answer_choices: null
id: 43dc1b77-e8ea-4dc8-8a12-0abc3b0dbba0
jinja: "Given the following passage in German: {{translation[\"de\"]}} , a good\
\ English translation is: \n||| {{translation[\"en\"]}}"
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: a_good_translation-de-en-source+target
reference: ''
474c20a1-a2ea-4ff4-b4c8-7f9c6466ff20: !Template
answer_choices: null
id: 474c20a1-a2ea-4ff4-b4c8-7f9c6466ff20
jinja: "Given the following passage: {{translation[\"en\"]}} , a good German translation\
\ is: \n||| {{translation[\"de\"]}}"
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: a_good_translation-en-de-target
reference: ''
4933dcae-3a66-4506-a479-a1387b287232: !Template
answer_choices: null
id: 4933dcae-3a66-4506-a479-a1387b287232
jinja: "What is the German translation of : {{translation[\"en\"]}} \n||| {{translation[\"\
fr\"]}}"
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: gpt-3-en-de-target
reference: ''
51d41a04-1d90-4bae-b6e5-be7598cdbfb0: !Template
answer_choices: null
id: 51d41a04-1d90-4bae-b6e5-be7598cdbfb0
jinja: 'How do you say {{translation["de"]}} in English?

||| {{translation["en"]}}'
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: how_to_say-de-en-target
reference: ''
527d2074-74fa-47ec-802e-7373728230df: !Template
answer_choices: null
id: 527d2074-74fa-47ec-802e-7373728230df
jinja: "Translate into German: {{translation[\"en\"]}} \n||| {{translation[\"\
fr\"]}}"
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: translate-en-de-target
reference: ''
5af8c873-7176-46b8-a31a-2b2d393c6c68: !Template
answer_choices: null
id: 5af8c873-7176-46b8-a31a-2b2d393c6c68
jinja: "Translate English into German: {{translation[\"en\"]}} \n||| {{translation[\"\
fr\"]}}"
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: translate-en-de-src+starget
reference: ''
762c0878-c8fc-43ec-839f-d5d8435a94f6: !Template
answer_choices: null
id: 762c0878-c8fc-43ec-839f-d5d8435a94f6
jinja: "Given the following passage: {{translation[\"de\"]}} , a good English\
\ translation is: \n||| {{translation[\"en\"]}}"
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: a_good_translation-de-en-target
reference: ''
9fe6b44b-2dc6-4557-8201-14d6ea7668ff: !Template
answer_choices: null
id: 9fe6b44b-2dc6-4557-8201-14d6ea7668ff
jinja: 'If the original version says: {{translation["de"]}}; then the English
version should say:

||| {{translation["en"]}}'
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: version-de-en-target
reference: ''
a3a87505-e423-4c03-9a22-a3da4ccbeae5: !Template
answer_choices: null
id: a3a87505-e423-4c03-9a22-a3da4ccbeae5
jinja: "Given the following passage in English: {{translation[\"en\"]}} , a good\
\ German translation is: \n||| {{translation[\"de\"]}}"
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: a_good_translation-en-de-source+target
reference: ''
b38093aa-3d76-433f-ade9-cbdd7c6b76de: !Template
answer_choices: null
id: b38093aa-3d76-433f-ade9-cbdd7c6b76de
jinja: "Comment dit-on {{translation[\"en\"]}} en allemand ?\n||| {{translation[\"\
de\"]}}"
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: comment_dit_on-en-de-target
reference: ''
c50ca84e-3ca7-4c3c-89e1-3c95351f6ed8: !Template
answer_choices: null
id: c50ca84e-3ca7-4c3c-89e1-3c95351f6ed8
jinja: 'How to you say {{translation["en"]}} in German?

||| {{translation["de"]}}'
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: how_to_say-en-de-target
reference: ''
c80e443a-0ba4-4c5d-be98-998e050a202d: !Template
answer_choices: null
id: c80e443a-0ba4-4c5d-be98-998e050a202d
jinja: 'If the original version says: {{translation["en"]}}; then the German version
should say:

||| {{translation["de"]}}'
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: version-en-de-target
reference: ''
ec3b96b0-de5e-4ff4-b7bb-cda348ff7fcf: !Template
answer_choices: null
id: ec3b96b0-de5e-4ff4-b7bb-cda348ff7fcf
jinja: "What is the German translation of the English sentence: {{translation[\"\
en\"]}} \n||| {{translation[\"de\"]}}"
metadata: !TemplateMetadata
choices_in_prompt: false
metrics:
- BLEU
original_task: true
name: gpt-3-en-de-src+target
reference: ''
Loading