Add new community pipeline for 'Adaptive Mask Inpainting', introduced in [ECCV2024] ComA #9228
Conversation
… in [ECCV2024] Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
|
Hi @jellyheadandrew, this seems really interesting, can you please post some example generations of this? |
|
@asomoza Yeah sure! Completely forgot updating the changes. tmp_cache.3.mp4
|
adaptive_mask_inpainting_example.py
Outdated
|
|
||
| if __name__ == "__main__": | ||
| """ | ||
| Download Necessary Files |
There was a problem hiding this comment.
all this example part should go inside the community pipelines README. This file can't go in the root of the project and in general, we don't use a separate example file for each community pipeline
adaptive_mask_inpainting_example.py
Outdated
| beta_start=0.00085, | ||
| beta_end=0.012, | ||
| beta_schedule="scaled_linear", | ||
| clip_sample=False, | ||
| set_alpha_to_one=False |
There was a problem hiding this comment.
you can simplify this by just changing the scheduler, no need to add the same config manually.
There was a problem hiding this comment.
Sorry, I'm kinda new to contributing to huggingface. What do you mean by changing the scheduler? Do you mean changing the config?
There was a problem hiding this comment.
oh, no problem, normally we can just change the scheduler with any other scheduler just using it like this:
from diffusers import DDIMScheduler
....
pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)which is cleaner and simpler.
There was a problem hiding this comment.
and if it has a different config (which doesn't seem the case here) you can just add the changed param as an additional kwarg
adaptive_mask_inpainting_example.py
Outdated
| adaptive_mask_model = PointRendPredictor( | ||
| pointrend_thres=0.2, | ||
| device="cuda" if torch.cuda.is_available() else "cpu", | ||
| use_visualizer=use_visualizer, | ||
| config_pth="pointrend_rcnn_R_50_FPN_3x_coco.yaml", | ||
| weights_pth="model_final_edd263.pkl", | ||
| ) | ||
| pipeline.register_adaptive_mask_model(adaptive_mask_model) |
There was a problem hiding this comment.
you can probably remove this from here and add it as code inside the pipeline to make it easier for the user. But it's your decision, if you do, you should probably download the files inside the pipeline too then.
adaptive_mask_inpainting_example.py
Outdated
| adaptive_mask_settings = EasyDict( | ||
| dict( | ||
| dilate_scheduler=MaskDilateScheduler( | ||
| max_dilate_num=20, | ||
| num_inference_steps=num_steps, | ||
| schedule=[20] * step_num + [10] * step_num + [5] * step_num + [4] * step_num + [3] * step_num + [2] * step_num + [1] * step_num + [0] * final_step_num | ||
| ), | ||
| dilate_kernel=np.ones((3, 3), dtype=np.uint8), | ||
| provoke_scheduler=ProvokeScheduler( | ||
| num_inference_steps=num_steps, | ||
| schedule=list(range(2, 10 + 1, 2)) + list(range(12, 40 + 1, 2)) + [45], | ||
| is_zero_indexing=False, | ||
| ), | ||
| ) | ||
| ) | ||
| pipeline.register_adaptive_mask_settings(adaptive_mask_settings) |
There was a problem hiding this comment.
Thanks for the thorough feedback! I've applied them, and will PR again.
(Please check minor questions above)
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
Hi @asomoza , I have updated the files based on your feedbacks. Waiting for more feedbacks, or a merge! |
|
Hi @jellyheadandrew sorry it took me so long to get back to you. Let's finish this so we can merge it. First can you please merge it to the main branch, I can see some couple of outdated imports that will probably fail. Also can you run |
|
@asomoza Hi, I've updated my branch! Could you check |
|
@jellyheadandrew thanks, you merged main but we still need the |
|
Thanks @asomoza , I've finished |
|
@jellyheadandrew merged! thanks for your contribution! |
… in [ECCV2024] ComA (#9228) * Add new community pipeline for 'Adaptive Mask Inpainting', introduced in [ECCV2024] Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models
Add new community pipeline for 'Adaptive Mask Inpainting' introduced in [ECCV2024] Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models.
What does this PR do?
This PR implements the 'Adaptive Mask Inpainting' algorithm introduced in [ECCV2024] Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models.
The code is borrowed from the author's repository (https://github.com/snuvclab/coma), changed slightly to represent the demo usage.
In a nutshell, this pipeline provides a way to insert human inside the scene image without altering the background, by inpainting with adapting mask.
Fixes # (issue)
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.