Bad test results

[liziming5353]

I download llama1 and use apply_delta.py to generate ckpt. Then I use the original run_valley.py without any modifications to test on a test dataset, but get terrible results. May I ask if there is anything special that needs to be set up? Like prompt or something?

[RupertLuo]

Make sure the “<video&gt;” token in your query.&nbsp;Can I see the results you got?

…

------------------&nbsp;原始邮件&nbsp;------------------ 发件人: "RupertLuo/Valley" ***@***.***&gt;; 发送时间:&nbsp;2023年11月13日(星期一) 下午2:55 ***@***.***&gt;; ***@***.***&gt;; 主题:&nbsp;[RupertLuo/Valley] Bad test results (Issue #23) I download llama1 and use apply_delta.py to generate ckpt. Then I use the original run_valley.py without any modifications to test on a test dataset, but get terrible results. May I ask if there is anything special that needs to be set up? Like prompt or something? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***&gt;

[RupertLuo]

Try this query, 'Describe this video concisely\<video\&gt;.\n'

…

------------------&nbsp;原始邮件&nbsp;------------------ 发件人: "RupertLuo/Valley" ***@***.***&gt;; 发送时间:&nbsp;2023年11月13日(星期一) 下午3:11 ***@***.***&gt;; ***@***.******@***.***&gt;; 主题:&nbsp;Re: [RupertLuo/Valley] Bad test results (Issue #23) Make sure the “<video&gt;” token in your query.&nbsp;Can I see the results you got? … ------------------&nbsp;原始邮件&nbsp;------------------ 发件人: "RupertLuo/Valley" @.&gt;; 发送时间:&nbsp;2023年11月13日(星期一) 下午2:55 @.&gt;; @.&gt;; 主题:&nbsp;[RupertLuo/Valley] Bad test results (Issue #23) I download llama1 and use apply_delta.py to generate ckpt. Then I use the original run_valley.py without any modifications to test on a test dataset, but get terrible results. May I ask if there is anything special that needs to be set up? Like prompt or something? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.&gt; query is "Describe this video concisely.\n — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***&gt;

[liziming5353]

The query is:

The response is:

I also print the outputs before self.process_response, and the result is:

No matter which video I input, it's always this result.

[RupertLuo]

Are your transformers version 4.28.0.dev?

[pratikkayal]

Hi I'm getting the same issue, I do have transformers version 4.28.0.dev Getting the following output: [',,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,....,,,,,,,,,,,,,,,,,,,,,,,,....................................................................................................,.,.........,.,,,,,,.,................................................................,,,,,,,.,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,................................................................................,.,.,...............................................................................................................................................................................,.,.,.,.....,.,.,.,.,................................................................................................................................................................................................,.,....,.,.,.,............................................................ (G..................................................................................-------']

[annopackage]

same issue here, offline demo seems normal.

The query is: The response is: I also print the outputs before self.process_response, and the result is: No matter which video I input, it's always this result.