Sketching + Inpainting Capabilities to Gradio

[abidlabs]

What use cases does the Image component need to serve?

On the Python side, users want…

• A standalone uploadable image (image classification, image segmentation, etc.)
• A standalone black-and-white sketch (handwriting recognition)
• A standalone color sketch (sketch2image)
• An uploadable image + a binary mask (inpainting)
• An uploadable image + a color sketch (paint2pix)

This PR adds support for all of these (plus a few minor ones like webcam + mask/sketch). To see all of the different ways the Image component can now be used, go to: https://huggingface.co/spaces/gradio-pr-deploys/pr-2144-all-demos and click on blocks_mask or use the beta release gradio==3.4.0b to test

The blocks_mask demo has the code snippets for all of the different modes. For example, for image upload + color sketching, you would something like:

import gradio as gr

gr.Interface(lambda x: x, gr.Image(source='upload', tool='color-sketch'), gr.Image())

Fixes: #1721
Fixes: #2060
Fixes: #2124
Fixes: #2030
Fixes: #1174
Fixes: #2312
Fixes: #2224
Fixes #2295

[github-actions]

All the demos for this PR have been deployed at https://huggingface.co/spaces/gradio-pr-deploys/pr-2144-all-demos

[abidlabs]

As discussed with @pngwn, gr.Sketchpad() is currently broken. If we are not able to fix it by tomorrow, I think we should release 3.3 without inpainting / sketching support.

[abidlabs]

@pngwn -- copying over from Slack for visibility:

I was testing the sketch/painting PR, and it seems like our old Sketchpad isn't working anymore. Specifically, it used to be that something like this:

gr.Interface(lambda x:x, gr.Image(source="canvas"), gr.Image()).launch()

Or using string shortcuts, something like:

gr.Interface(lambda x:x, gr.Sketchpad(), gr.Image()).launch()

would create a black and white sketchpad (e.g. for handwritten recognition)

However, if I run this code now, I get nothing showing up and a bunch of JS errors in the console:

We should confirm that all 5 of the use cases mentioned in the issue work before merging this in.

[pngwn]

@abidlabs This should be ready for review now. I made to make some changes to components.py other than those we discussed (because i wanted tool="color-sketch" to always have the same preprocess return regardless on the source) but other than that the changes are mostly frontend.

I've added more demos to blocks_mask which I think should cover all usecases.

There is currently a discrepancy between the gr. Sketchpad() and manually typing the kwargs as I mentioned.

Do Paint, ImagePaint and ImageMask helper components exist yet?

[abidlabs]

Awesome! Will go through these.

The shortcuts do exist, but right now as gr.templates.Paint, gr.templates.ImagePaint etc. because I haven't exposed them as top-level classes yet. But will do!

[abidlabs]

This looks really fantastic @pngwn! I updated the blocks_mask demo with a few more use cases. A few notes:

(1) There are two cases in which the behavior is weird, and that is if (source, tool) is ("webcam", "sketch") or ("webcam", "color-sketch"). In these cases, the behavior I observe is that when you start sketching, the image flips after you draw the first stroke, and then the image disappears altogether after you do the second stroke. I've added these cases to the blocks_demo
() so you can see the behavior here: https://huggingface.co/spaces/gradio-pr-deploys/pr-2144-all-demos

(2) As you noted, gr.Sketchpad() behaves differently than gr.Image(source="canvas", tool="sketch"). This is expected, because Sketchpad()` contains a few more parameters that are specifically designed to make it useful for MNIST demos (size, inversion of colors).

I made some other changes:

• If someone uses the gr.Webcam(), gr.Sketchpad(), gr.Paint(), etc templates, I've added interactive=True, since I don't see why these components would ever be used if interactivity is not desired, and it makes them a little easier to try out in Blocks.
• Changed the return type of color-sketch so it returns a string instead of a dict with an empty "mask" key
• I was thinking about the point you made @pngwn about how their might be churn if we change the API signature of the Image component from returning "a string in most cases but a dictionary in one case" to "a dictionary in all cases", so I added a parameter in the component called "force_dict". If set to True, it preprocesses the data in all cases to be a dictionary with keys image and mask. I hope this is not more confusing to users.
• Released gradio==3.4.0b to test

If we can fix point (1), then I think this is good to merge!

[pngwn]

I added a parameter in the component called "force_dict". If set to True, it preprocesses the data in all cases to be a dictionary with keys image and mask. I hope this is not more confusing to users.

I'm not sure anyone will actually use this. Although it doesn't do much harm, it does add another option which makes the docs more overwhelming. I think it would be okay to leave it as is and potentially add a flag when we do make any changes. We can make a hard break from some of these APIs in 4.0, likewise we could just add the flag and remove in 4.0 when we clean up some of the API.

I'll take a look at the webcam thing, it is a little strange but I think I know what is causing it, however, I'm not sure how easy it will be to fix because we don't have a flipped version of the image on the frontend, and if we do flip on the frontend we will need to make we don't flip on the backend as well. Will take a look at this today.

[abidlabs]

Sounds good, I actually felt similarly (that it was adding too many options) and didn’t end up pushing the force_dict (forgot to edit the GitHub comment).

…

On Fri, Sep 16, 2022 at 12:24 AM pngwn ***@***.***> wrote: I added a parameter in the component called "force_dict". If set to True, it preprocesses the data in all cases to be a dictionary with keys image and mask. I hope this is not more confusing to users. I'm not sure anyone will actually use this. Although it doesn't do much harm, it does add another option which makes the docs more overwhelming. I think it would be okay to leave it as is and potentially add a flag when we do make any changes. We can make a hard break from some of these APIs in 4.0, likewise we could just add the flag and remove in 4.0 when we clean up some of the API. I'll take a look at the webcam thing, it is a little strange but I think I know what is causing it, however, I'm not sure how easy it will be to fix because we don't have a flipped version of the image on the frontend, and if we do flip on the frontend we will need to make we *don't* flip on the backend as well. Will take a look at this today. — Reply to this email directly, view it on GitHub <#2144 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANSE6OWNIPRRVUOJ3IKDELV6QOBDANCNFSM6AAAAAAQBTLZXU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[freddyaboulton]

The blocks_mask demo makes it really easy to see all the proposed changes! Thank you for preparing it.

@abidlabs What do you mean by this? Why do we need a class just for MNIST demos (toy examples)

Sketchpad() contains a few more parameters that are specifically designed to make it useful for MNIST demos (size, inversion of colors).

The one thing that tripped me up is that you have to click on the dropper icon to change color. I would think that you can just change color by clicking anywhere on the color spectrum.

[abidlabs]

Sketchpad() contains a few more parameters that are specifically designed to make it useful for MNIST demos (size, inversion of colors).

Yeah this is a remnant from the early days of Gradio, in the early days we wanted to show how easy it was to get started with Gradio, so we had a shortcut ("sketchpad") specifically for MNIST models. We should have dropped in Gradio 3.0, but forgot. I think we are going to need to keep this now for backwards compatibility reasons, but we can file an issue to drop it in 4.0.

The one thing that tripped me up is that you have to click on the dropper icon to change color. I would think that you can just change color by clicking anywhere on the color spectrum.

Nice catch @freddyaboulton!

[abidlabs]

Noticed a bug --> if you use the ImagePaint mode (i.e. source="upload", tool="color-sketch"), then after you upload an image and start painting, you have no way of clearing and uploading a new image. If you clear the image, it becomes like a regular color-sketch component and you cannot upload an image.

Also @pngwn do you know if this PR fixes #1961? It might be worth including that fix in this PR while we're working on the sktechpad component

[abidlabs]

In addition, I find the brush behavior unintuitive in one regard: that you have to move your cursor outside of the entire brush circle in order to get the brush to move. Instead, the center of the brush should just track the cursor.

[abidlabs]

Additional feedback from users:

The only buggy behavior I've spotted so far is the canvas resetting when changing the scale of my browser window. Otherwise, it works very well! 👌 Maybe a rubber would be a good idea to implement soon :) - https://twitter.com/fffiloni/status/1570891355433611266

+1 on the eraser suggestion

[fffiloni]

Hello ! I got some new observations to report, i'll go straight to the points:

1. sketch-color BUG : canvas background resetting to none (or transparent) after browser window scale change (Chrome)

2. uploaded image + sketch-color BUG: when cleaning the canvas for some reason (missed the drawing, want to start again), it cleans the whole canvas, even the uploaded image —› we have to refresh the page to be able to re-upload
   @abidlabs reported it sooner (Sketching + Inpainting Capabilities to Gradio #2144 (comment))

Solution: add a clean canvas button on the right to only clean drawings layer, not background layer containing the uploaded img

3. Following the 2nd point, we could benefit from a "undo last line" button ;)

4. Yes, users will ask for an eraser soon :)

5. For every sketch related blocks, i suggest to add a parameter in the method to let the user specify the canvas width and height he could need to control. For the moment, i use css to "hack" the canvas size, but i think it should be more easy to directly set the canvas dimensions as a parameter.

That's it, for the moment ;)

[fffiloni]

For the eraser feature, i made a good working one for my animation app, using p5js
where r is the eraser radius, and target is the graphics (canvas instance) targeted to update pixels to transparent

trueErase(r, target){
    // called in a p5 mouseDragged function, when the "E" key is pressed
    // target is the graphics you want to erase on | e.g: ani.frameGraphics
    
    target.loadPixels();

    for (let x = mouseX - r; x < mouseX + r; x++) {
      for (let y = mouseY - r; y < mouseY + r; y++) {
        if ((dist(x,y, mouseX, mouseY) < r) && x > 0 && x <= width) {

          target.set(x,y,color(0,0));

        }
      }
    }

    target.updatePixels();
}

You can take a look on my work here : https://editor.p5js.org/fffiloni/sketches/LPSfkKlQ6
Methods at play here are stored in the DrawHandler.js class file

And if you plan to add an animation gradio block in the future, i would be glad to help ! :)

[fffiloni]

Sketch-Tool BUG: Fast move reveals spikes on the end of each chunk of the line. See screenshot below:

[abidlabs]

Additional feedback from @hysts (from Discord)

• It would be nice to have features typical drawing tools have, like an eraser, bucket, undo/redo button, changing transparency, etc.
• It would be better to have a larger canvas for painting. For example, it would be nice to open up a new drawing window with an edit button.
• It would be great if app creators could specify a predefined list of colors and users could select colors only from the list. Some apps that take a semantic segmentation mask as an input require the input image with only the predefined colors. I'm thinking of a use case like this Space: https://huggingface.co/spaces/CVPR/drawings-to-human.
  Also, I'm wondering if it's possible to get a color mask as the binary mask demo 3a. The ImagePaint seems to return painted image, but being able to return the original image and the mask separately would be nice.
  Here are something I noticed when I tried the blocks_masks demo (https://huggingface.co/spaces/gradio-pr-deploys/pr-2144-all-demos/blob/main/demos/blocks_mask/run.py):
• It's not possible to Clear or Submit without painting when using io5a and io5b.
• io5c doesn't seem to be working properly. The image flips horizontally when one draws the first stroke. And the previous image disappears after each stroke.
• Clearing image doesn't work properly. After clearing image, we cannot add a new image.

[pngwn]

I've updated this issue to gather feedback on any changes that are out of scope for this PR, would be good to get everyone's thoughts: #466

[pngwn]

Pushed some changes.

• Webcam should work as expected now. Webcam with colour-sketch returns a single image, webcam with sketch returns an image + mask layer separately. Required some tweaks in the processor.
• Clearing (using the "X" button) now works as expected (clearing both the image + sketch, and going back to the default 'upload' screen). I can make clear image + clear sketch separate but needs some design work. Better to add in another PR.
• You can now submit when source=upload and tool=color-sketch without sketching first.
• You can now change the brush size for masks as well as colour sketches.
• "Undo" is back! When using the sketch tool you can undo lines 1 by 1.

The gradio "Clear" button at the bottom doesn't work as intended for all sketches, i'm looking into this as well as the issue with Tabs, I think they are related.

[eduardocarvp]

Hi, thanks for working on this!

To add to @abidlabs' comment on the brush behavior: it is indeed a bit unintuitive that the cursor does not track the center of the brush. It makes it particularly hard to go over the borders or even the icons/buttons from the tool itself (i.e. the color or brush size picker), since once the cursor is off the canvas the painting stops and we need a new click inside the canvas to continue.

Looking forward to testing the evolutions on this PR :)

[pngwn]

@eduardocarvp I agree, the 'lazy brush' behaviour was originally designed to help draw smoother curves with a mouse but it might be too lazy but should probably be toggleable from the gradio API or GUI.

I have listed the ability to disable/ toggle this feature in #466 as it is out of scope for this PR.

[pngwn]

This PR should be ready for final testing now.

[pngwn]

Although this PR doesn't address every single feature requested in the linked issues above, I've created a new issue to track feature requests for the Image component, so they can all be closed with this PR (which addresses most of the requests anyway).

[abidlabs]

This is really nice @pngwn! I tested the blocks_mask demo on the Spaces deploy in different conditions, and I noticed a couple of things:

---

A) the first time you use a sketchpad or a color-sketchpad, it tends to add a gray background to your sketch:

When I would click the submit button, I found that the output would be all black:

If you clear the sketchpad and start drawing again, the sketchpads work fine.

--

B) 5(a) and 5(b) have the opposite problem: the first time, they work fine with an image. But after you clear the first image, they no longer correctly upload an image. If you try to upload an image again, it will just flicker and then become all white:

--

C) The example in 3(b) doesn't seem to work (nothing happens when you click Submit). 3(a) works fine though.

Haven't noticed any other bugs while testing!

[pngwn]

I can't reproduce the issues with 2a locally:

3b was broken because there was another instance of it on the page (#2320) i've fixed the demo.

Undo was buggy, fixed that now i think. Will push to see if there was an issue with the spaces deploy becaus egetting different results locally + in spaces isn't ideal.

Looking into 5a + 5b.

[pngwn]

I'm getting different results locally vs the spaces deploy, and I'm not sure what is going on. It could be the tabs the deployed demos are embedded in or it could be something related to the environment. Not sure.

[pngwn]

5a + 5b should work okay now. Still not sure how to repro the issue with transparent canvases.

[pngwn]

I've removed the lazy brush as well but kept some degree of path smoothing. Should be a little more intuitive now.

[abidlabs]

Confirming that, besides the aforementioned issue with 2(a)/2(b)/4(a)/4(b), all other issues look good!

[abidlabs]

🥳 LGTM

[franchesoni]

Hello,

All of this is going in a great direction. Do you plan to add full interactivity support? What I mean is interacting not only with the input image, but also with an output image that can be resubmitted. It could be useful for interactive image segmentation demos (between other research fields that have "interactive" in their name). The hardest to fulfill requirement is to maintain a state from one user interaction to the next. Should I open a new issue to discuss this?