New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mixtral evaluation PR #8989

Merged

ericharper merged 4 commits into NVIDIA:main from Slyne:mixtral-as-judge-eval-pipeline2

May 6, 2024

Contributor

Slyne commented Apr 19, 2024

Please refer to the previous PR:
#8988


          add mixtral evaluation

43f0f39

Co-authored-by: PannuMuthu <[email protected]>

Signed-off-by: Slyne Deng <[email protected]>

github-actions bot added the Multi Modal label

Contributor Author

Slyne commented Apr 19, 2024

@ethanhe42
too messy to fix the signature issue for the previous pr: #8988
So restart a new one. LOL..


          fix format & remove unused import

6fd6fab

Signed-off-by: Slyne Deng <[email protected]>

ethanhe42 approved these changes

View reviewed changes

Member

ethanhe42 commented May 3, 2024

@Slyne could you move doc to docs/source/multimodal/mllm/video_neva.rst ?

Slyne Deng added 2 commits

May 3, 2024 12:06


          Merge branch 'main' of https://github.com/NVIDIA/NeMo into mixtral-as…

0a11c8e

…-judge-eval-pipeline2


          move mixtral as judge evaluation document

ef7fd3a

Signed-off-by: Slyne Deng <[email protected]>

Contributor Author

Slyne commented May 3, 2024

@ethanhe42 Done.

yaoyu-33 added the Run CICD label

github-advanced-security bot found potential problems

View reviewed changes

examples/multimodal/multimodal_llm/neva/eval/mixtral_eval.py

+                      if line:
+                          try:
+                              res = json.loads(line.decode("utf-8").split("data: ")[1])
+                          except:

Check notice

Code scanning / CodeQL

Except block handles 'BaseException' Note

Except block directly handles BaseException.

examples/multimodal/multimodal_llm/neva/eval/mixtral_eval.py

+              def generate_prompt(args, answer_list):
+                  f_q = open(os.path.expanduser(args.question_file))
+                  f_ans1 = open(os.path.expanduser(answer_list[0]))

Check warning

Code scanning / CodeQL

File is not always closed Warning

File is opened but is not closed.

examples/multimodal/multimodal_llm/neva/eval/mixtral_eval.py

+              def generate_prompt(args, answer_list):
+                  f_q = open(os.path.expanduser(args.question_file))
+                  f_ans1 = open(os.path.expanduser(answer_list[0]))
+                  f_ans2 = open(os.path.expanduser(answer_list[1]))

Check warning

Code scanning / CodeQL

File is not always closed Warning

File is opened but is not closed.

examples/multimodal/multimodal_llm/neva/eval/mixtral_eval.py

+                  f_q = open(os.path.expanduser(args.question_file))
+                  f_ans1 = open(os.path.expanduser(answer_list[0]))
+                  f_ans2 = open(os.path.expanduser(answer_list[1]))
+                  rule_dict = json.load(open(os.path.expanduser(args.rule_file), 'r'))

Check warning

Code scanning / CodeQL

File is not always closed Warning

File is opened but is not closed.

examples/multimodal/multimodal_llm/neva/eval/mixtral_eval.py

+                  rule_dict = json.load(open(os.path.expanduser(args.rule_file), 'r'))
+                  if os.path.isfile(os.path.expanduser(args.output)):
+                      cur_reviews = [json.loads(line) for line in open(os.path.expanduser(args.output))]

Check warning

Code scanning / CodeQL

File is not always closed Warning

File is opened but is not closed.

examples/multimodal/multimodal_llm/neva/eval/mixtral_eval.py

+                  else:
+                      cur_reviews = []
+                  review_file = open(f'{args.output}', 'a')

Check warning

Code scanning / CodeQL

File is not always closed Warning

File may not be closed if an exception is raised.

examples/multimodal/multimodal_llm/neva/eval/mixtral_eval.py


		review_file = open(f'{args.output}', 'a')

		context_list = [json.loads(line) for line in open(os.path.expanduser(args.context_file))]

Check warning

Code scanning / CodeQL

File is not always closed Warning

File is opened but is not closed.

examples/multimodal/multimodal_llm/neva/eval/mixtral_eval.py



		def preprocess(args, response_file, model_name):
		questions = [json.loads(q) for q in open(os.path.expanduser(args.question_file), "r")]

Check warning

Code scanning / CodeQL

File is not always closed Warning

File is opened but is not closed.

examples/multimodal/multimodal_llm/neva/eval/mixtral_eval.py

+              def preprocess(args, response_file, model_name):
+                  questions = [json.loads(q) for q in open(os.path.expanduser(args.question_file), "r")]
+                  responses = [json.loads(r) for r in open(os.path.expanduser(response_file), "r")]

Check warning

Code scanning / CodeQL

File is not always closed Warning

File is opened but is not closed.

examples/multimodal/multimodal_llm/neva/eval/mixtral_eval.py

+                  answer_file = os.path.join(args.answers_dir, f'{base}_answer{ext}')
+                  answers_file = os.path.expanduser(answer_file)
+                  os.makedirs(os.path.dirname(answers_file), exist_ok=True)
+                  ans_file = open(answers_file, "w")

Check warning

Code scanning / CodeQL

File is not always closed Warning

File may not be closed if an exception is raised.

ericharper merged commit f609553 into NVIDIA:main

130 of 131 checks passed

rohitrango pushed a commit to rohitrango/NeMo that referenced this pull request


          mixtral evaluation PR (NVIDIA#8989)

* add mixtral evaluation
Co-authored-by: PannuMuthu <[email protected]>

Signed-off-by: Slyne Deng <[email protected]>

* fix format & remove unused import

Signed-off-by: Slyne Deng <[email protected]>

* move mixtral as judge evaluation document

Signed-off-by: Slyne Deng <[email protected]>

---------

Signed-off-by: Slyne Deng <[email protected]>
Co-authored-by: Slyne Deng <[email protected]>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Multi Modal Run CICD