Skip to content

add image-to-text and visual question answering example#738

Merged
regisss merged 5 commits into
mainfrom
blip_example
Mar 7, 2024
Merged

add image-to-text and visual question answering example#738
regisss merged 5 commits into
mainfrom
blip_example

Conversation

@sywangyi
Copy link
Copy Markdown
Collaborator

@sywangyi sywangyi commented Feb 26, 2024

What does this PR do?

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

@sywangyi sywangyi requested a review from regisss as a code owner February 26, 2024 09:58
@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

HuggingFaceDocBuilderDev commented Feb 26, 2024

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@sywangyi sywangyi force-pushed the blip_example branch 3 times, most recently from 5dca78f to 2a3c6e9 Compare February 26, 2024 10:34
@regisss
Copy link
Copy Markdown
Collaborator

regisss commented Feb 27, 2024 •
edited
Loading

@sywangyi I'll review this PR shortly. FYI this PR was merged so that generation kwargs can be passed when calling the pipeline (if we want to disable HPU graphs for instance).

@sywangyi
Copy link
Copy Markdown
Collaborator Author

sywangyi commented Feb 27, 2024

yes, later we could pass generation kwargs in VQA example when transformers in optimum-habana is upgraded to contain the fix

libinta
libinta reviewed Feb 27, 2024
View reviewed changes
Comment thread README.md
libinta
libinta reviewed Feb 27, 2024
View reviewed changes
Comment thread examples/image-to-text/run_pipeline.py Outdated
libinta
libinta reviewed Feb 27, 2024
View reviewed changes
Comment thread examples/image-to-text/run_pipeline.py Outdated
libinta
libinta reviewed Feb 27, 2024
View reviewed changes
Comment thread examples/image-to-text/run_pipeline.py Outdated
libinta
libinta reviewed Feb 27, 2024
View reviewed changes
Comment thread examples/image-to-text/run_pipeline.py Outdated
@sywangyi
Copy link
Copy Markdown
Collaborator Author

sywangyi commented Feb 28, 2024

huggingface/transformers#29342 is upload to fix multi-batch incorrect output for image-to-text.

sywangyi added 3 commits March 3, 2024 20:46
@sywangyi
add image-to-text and visual question answering example
d59f88f 
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
@sywangyi
update example to contain fp32 and batch_size
17f6c99 
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
@sywangyi
no need to autocast for image to text
381dbf5 
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
@sywangyi
add prompt input in the example, llava needs it
97a5c1b 
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
libinta
libinta reviewed Mar 4, 2024
View reviewed changes
Comment thread examples/image-to-text/run_pipeline.py
libinta
libinta reviewed Mar 4, 2024
View reviewed changes
Comment thread examples/visual-question-answering/README.md
libinta
libinta reviewed Mar 4, 2024
View reviewed changes
Copy link
Copy Markdown
Collaborator

@libinta libinta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@regisss @sywangyi should we also have a ci test with it?

@lkk12014402 lkk12014402 mentioned this pull request Mar 5, 2024
Merged
libinta
libinta reviewed Mar 6, 2024
View reviewed changes
Copy link
Copy Markdown
Collaborator

@libinta libinta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@regisss I don't have other comments about this PR except adding test case. WDYT?

regisss
regisss reviewed Mar 7, 2024
View reviewed changes
Comment thread examples/image-to-text/README.md Outdated
Comment thread examples/image-to-text/README.md Outdated
Comment thread examples/image-to-text/run_pipeline.py Outdated
Comment thread examples/image-to-text/run_pipeline.py Outdated
Comment thread examples/image-to-text/run_pipeline.py Outdated
Comment thread examples/visual-question-answering/run_pipeline.py Outdated
Comment thread examples/visual-question-answering/run_pipeline.py Outdated
Comment thread examples/visual-question-answering/run_pipeline.py Outdated
Comment thread examples/visual-question-answering/run_pipeline.py Outdated
Comment thread examples/visual-question-answering/run_pipeline.py Outdated
@sywangyi
address the review comment
caf309b 
Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
@regisss regisss merged commit 2f55de3 into main Mar 7, 2024
@regisss regisss deleted the blip_example branch March 7, 2024 06:20
puneeshkhanna pushed a commit to puneeshkhanna/optimum-habana-fork that referenced this pull request Mar 11, 2024
@sywangyi
Add image-to-text and visual question answering example (huggingface#738
1defbc6 
)

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
HolyFalafel pushed a commit to HabanaAI/optimum-habana-fork that referenced this pull request Mar 11, 2024
@sywangyi @HolyFalafel
Add image-to-text and visual question answering example (huggingface#738
125a6b5 
)

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@libinta libinta libinta left review comments

@regisss regisss regisss approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sywangyi @HuggingFaceDocBuilderDev @regisss @libinta