Issues with inpainting image input

[abidlabs]

Describe the bug

This demo uses the new image input (gr.Image(tool="sketch", label="Input",type="numpy")): https://huggingface.co/spaces/akhaliq/lama

However, when examples are added to the demo, clicking on the examples does not populate the inpainting image input. Instead, you get a blank gray rectangle there:

Is there an existing issue for this?

• I have searched the existing issues

Reproduction

https://huggingface.co/spaces/akhaliq/lama

Screenshot

No response

Logs

No errors

System Info

3.0.22

Severity

serious, but I can work around it

[drscotthawley]

Finding that manual mask drawing is not supported for almost any uploaded image for the LAMA model https://huggingface.co/spaces/akhaliq/lama

Only the default image works. This is using Brave Browser (Chromium... Version 1.37.116 Chromium: 100.0.4896.127 (Official Build) (64-bit) ).

For square uploaded images, I notice that the white circle showing the mask drawing location appears about 150 pixels down and to the right of the mouse pointer.

For most images, however, be they square or rectangular, simply no mask drawing is possible.

[abidlabs]

Thanks @drscotthawley, we'll take a look!

[drscotthawley]

Examples of images for which manual mask painting does not work (on Chrome, Linux):

[AK391]

this might be related to setting a default image using gr.Image(value="image.png"), after removing that works on the images shown above

[pngwn]

This is a scaling issue with very large images, as I understand it. The image and UI is downscaled when the image is very large. We need to scale the image independent of the UI and scale the painted region to the size of the image in order for it to appear correctly.

[pngwn]

Pretty sure I'm wrong, these images aren't even that big. Investigating.

[pngwn]

The default image issue also affected using the sketch as an output, the simple reason for this is we simple didn't support that use case, which is a huge oversight and a bug. This is easily addressed.

The issue of the drawing tool not being available is actually slightly trickier. The scaling is wrong, per my previous comments, but there is an additional bug in that the canvas is set up only once and not refreshed when the base image changes. This is fine for the old sketch (with no bg image) but not for the new one. The brush size is also only set up once and never changes, however, the brush position does get updated (once per frame when the mouse moves). This combination of some things not getting updated + rescaled (the canvas is actually very large and gets rescaled, thus the brush position has to be scaled accordingly) and others refreshing every frame leads to the brush appearing in strange locations. If the two images aren't too different in size then the brush position is just slightly offset from the cursor, however, when the difference is greater the brush can appear offscreen. If you hard reload the page every time then all of these images work roughly as expected (with some weird brush scaling), you also get different behaviour depending on which image you use first.

I'll actually just change the way we scale everything to make this problem go away.

[abidlabs]

Not sure if this is the best place, but putting some thoughts down around the various modes of the gr.Image component and how to support them all:

---

What use cases does the Image component need to serve?

On the Python side, users want…

• A standalone uploadable image (image classification, image segmentation, etc.)
• A standalone black-and-white sketch (handwriting recognition)
• A standalone color sketch (sketch2image)
• An uploadable image + a binary mask (inpainting)
• An uploadable image + a color sketch (paint2pix)

Note: I do not think we'll need image + binary mask + sketch

On the front end,

• The black-white sketch / binary mask should support:
  • Variations in stroke width
  • Variations in stroke opacity
• The color sketch should support
  • Variations in stroke width
  • Variations in stroke color
  • Variations in stroke opacity
• The size of the uploaded image should not affect the stroke size (that’s led to a lot of bugs)

Gradio API:

• gr.Image(source=”upload”) gives you 1: standalone uploadable image
  • Since source=”upload” by default, gr.Image() also works
• gr.Image(source=”canvas”, tool=”sketch”) gives you 2: standalone black-and-white sketch
  • We change the default value of tool parameter to be “sketch” if source=”canvas” so that so gr.Image(source=”canvas”) works too (preserving backwards compatibility)
  • We keep the gr.Sketchpad() template (preserving backwards compatibility)
• gr.Image(source=”canvas”, took=”color-sketch”) gives you 3
  • We add gr.Paint() template for ease of use -- this one does not invert colors and does not have a default size
• gr.Image(source=”upload”, tool=”sketch”) gives you 4
  • We add gr.ImageMask() template for ease of use
• gr.Image(source=”upload”, tool=”color-sketch”) gives you 5
  • We add gr.ImagePaint() template for ease of use

What does frontend send to the backend? (Or if the Space is used as an API, what is returned?)
Note: Ideally, we always send a dictionary back, but this would break the API usage, as well as gr.Interface.load(), etc, so for now:

• In cases 1, 2, 3, we send a single base64 image back
• In cases 4, 5, we send a dict with keys “image” and “mask” back, each have a base64 image