The number of LLM evaluated examples

[kkk-an]

I just run below code and find that the examples need to be evaluated by LLM are not equivalent to your papers.
`rule_based_source = ["E2E", "WIKIEVENTS", "CONLL2003",
"text_editing", "cnn_dailymail", "xsum", "samsum", "gigaword", "arxiv",
"BBH_logical", "BBH_time", "self_made_space", "gsm_8k"]

for type in ["content", "situation", "format", "example", "mixed"]:
data = json.load(open(f"./data/{type}_constraints.json"))
rule, llm = 0, 0
for d in data:
level = d["level"]
if level == 0:
continue
source = d["source"]
if source in rule_based_source:
rule += 1
else:
llm += 1
print(f"type: {type}, rule: {rule}, llm:{llm}")`

Is there any misunderstanding in your paper of code?

Thanks for your reply.

[kkk-an]

I have checked my gpt4_discriminative_eval_input and find that the number of examples that need to be evaluated by LLMs are:
content: 65 | mixed: 45 | format: 140 | situation: 70
but your paper just reports:
content: 50 | mixed: 10 | format: 120 | situation: 55

I am very confused and kindly request your help, thank you so much.