PApplet.cursor(PImage img) broken for JOGL and AWT since PImage.getNative() always returns null

[neonfabrik]

Description

PImage.getNative() always returns null since 4.0a3 (commit: e46ecea).

This produces a NullPointerException if PApplet.cursor(PImage img) or PApplet.cursor(PImage img, int x, int y) is invoked while the renderer is set to JAVA2D/P2D/P3D.

The recently adopted null value return as opposed to an java.awt.Image seems to stem from contributors practicing exorcism to evict the evil AWT (a billion thanks for making Processing available and updated, by the way).

I think I've found a decent way to keep the cursor(PImage) functionality for the JOGL renderers (P2D/P3D), while at the same time getting rid of the two awt.image dependencies in PSurfaceJOGL.

---

Sidenote # 1 (JavaFX problems):
For FX2D the behaviour is almost as expected, but also have some problems:

1. The custom cursor remains after mouse leaving the sketch window (might be intentional?).
2. The custom cursor disappears when switching to another application (good) but doesn't reappear when switching back to the sketch (bad).
   However, this seems to be unrelated to the issues with other renderers and is not discussed more in this submission.

Sidenote # 2 (quick fix for AWT/JOGL):
As a temporary quick-fix – for those in desperate need of cursor functionality – using PImageAWT (as proposed in the wiki) works fine as far as I can tell.
For example, the following code will result in the expecting behaviour (a square little cursor):

import processing.awt.PImageAWT;
import java.awt.image.BufferedImage;

size(100,100,P2D);

PGraphics cursor = createGraphics(16,16);
println(getSurface());
cursor.beginDraw();
cursor.rect(0,0,10,10);
cursor.endDraw();

BufferedImage buff = new BufferedImage(16, 16, BufferedImage.TYPE_INT_ARGB);
PImageAWT cursorAWT = new PImageAWT(buff);
cursorAWT.set(0, 0, cursor);

cursor(cursorAWT);

Expected Behavior

The mouse cursor should change to the specified PImage when invoking cursor(PImage someImage).

Current Behavior

A NullPointerException is always thrown for P2D/P3D/JAVA2D renderers.

Steps to Reproduce

1. Run the following code:

size(100, 100, P2D);  // i.e. getSurface() will return a PSurfaceJOGL.
PImage cursor = createImage(16, 16, ARGB); // an empty, transparent PImage.
cursor(cursor); // should produce an empty (invisible) cursor.

2. A NullPointerException is thrown, if the specified renderer is P2D, P3D, JAVA2D or unspecified (= JAVA2D).

Your Environment

• Processing version: 4.0a3
• Operating System and OS version: MacOS Catalina 10.15.7

Possible Causes / Solutions

Cause:

The problem boils down to getNative() of PImage.java always returning null:

public Object getNative() {  // ignore
    return null;
}

Backtrack:
In PApplet.java (4.0a3) cursor(PImage img) forwards to setCursor(PImage img, int x, int y) of the current surface, i.e. either PSurfaceAWT or PSurfaceJOGL for default JAVA2D or P2D/P3D renderers, respectively:

public void cursor(PImage img) {
    cursor(img, img.width/2, img.height/2);
}
// ...
public void cursor(PImage img, int x, int y) {
    surface.setCursor(img, x, y);
}

In PSurfaceAWT.java (4.0a3) we have ...

public void setCursor(PImage img, int x, int y) {
// ...
    Cursor cursor =
      canvas.getToolkit().createCustomCursor((Image) img.getNative(), // <<<<<<< Bad boy getNative() <<<<<<<
                                             new Point(x, y),
                                             "custom");
    canvas.setCursor(cursor);
// ...

and in PSurfaceJOGL.java ...

public void setCursor(PImage image, int hotspotX, int hotspotY) {
    Display disp = window.getScreen().getDisplay();
    BufferedImage bimg = (BufferedImage)image.getNative(); // <<<<<<< Bad boy getNative() <<<<<<<
// ...

Solution for PSurfaceJOGL:

For setCursor(PImage image, int hotspotX, int hotspotY) in PSurfaceJOGL, the dependency on AWT's BufferedImage seems redundant, but I may have misunderstood something. Also, I think it's unnessecary to declare a new disp variable since this should refer to the class variable display?

The code as of now, with my comments:

  public void setCursor(PImage image, int hotspotX, int hotspotY) {
    Display disp = window.getScreen().getDisplay();                     //<< Should be same as display variable of the class?
    BufferedImage bimg = (BufferedImage)image.getNative();              //<< REDUNDANT ???
    DataBufferInt dbuf = (DataBufferInt)bimg.getData().getDataBuffer(); //<< REDUNDANT ???
    int[] ipix = dbuf.getData();                                        //<< Should be exactly equal to image.pixels ?
    ByteBuffer pixels = ByteBuffer.allocate(ipix.length * 4);
    pixels.asIntBuffer().put(ipix);
    PixelFormat format = PixelFormat.ARGB8888;
    final Dimension size = new Dimension(bimg.getWidth(), bimg.getHeight());
    PixelRectangle pixelrect = new PixelRectangle.GenericPixelRect(format, size, 0, false, pixels);
    final PointerIcon pi = disp.createPointerIcon(pixelrect, hotspotX, hotspotY);
    display.getEDTUtil().invoke(false, new Runnable() {
      @Override
      public void run() {
        window.setPointerVisible(true);
        window.setPointerIcon(pi);
      }
    });
  }

As I understand it, the PImage.pixels array (int[]) already stores each pixel as a 32bit unsigned integer as specified by the already implemented PixelFormat.ARGB8888 (i.e. #AARRGGBB in hex notation). As such, casting from PImage -> Buffered Image -> DataBufferInt -> int[] should be redundant as we simply can use the PImage.pixels int array as it is.
If we also make use of loadPixels(), we make sure pixels is up to date (for example, I noticed pixels otherwise returned null for a newly created PGraphics).

Proposed change (in PSurfaceJOGL.java):

 public void setCursor(PImage image, int hotspotX, int hotspotY) {
-   Display disp = window.getScreen().getDisplay();
-   BufferedImage bimg = (BufferedImage)image.getNative();
-   DataBufferInt dbuf = (DataBufferInt)bimg.getData().getDataBuffer();
-   int[] ipix = dbuf.getData();
+   image.loadPixels();   //Make sure pixels are loaded.
+   int[] ipix = image.pixels;
    ByteBuffer pixels = ByteBuffer.allocate(ipix.length * 4);
    pixels.asIntBuffer().put(ipix);
    PixelFormat format = PixelFormat.ARGB8888;
-   final Dimension size = new Dimension(bimg.getWidth(), bimg.getHeight());
+   final Dimension size = new Dimension(image.width, image.height);
    PixelRectangle pixelrect = new PixelRectangle.GenericPixelRect(format, size, 0, false, pixels);
-   final PointerIcon pi = disp.createPointerIcon(pixelrect, hotspotX, hotspotY);
+   final PointerIcon pi = display.createPointerIcon(pixelrect, hotspotX, hotspotY);
    display.getEDTUtil().invoke(false, new Runnable() {
      @Override
      public void run() {
        window.setPointerVisible(true);
        window.setPointerIcon(pi);
      }
    });
  }

The two awt.image dependencies can then also be removed:

-   import java.awt.image.BufferedImage;
-   import java.awt.image.DataBufferInt;

Proof of concept:

I have tested the following code in a processing sketch and it works as expected for P2D & P3D (& OPENGL).

import com.jogamp.nativewindow.util.Dimension;
import com.jogamp.nativewindow.util.PixelFormat;
import com.jogamp.nativewindow.util.PixelRectangle;
import java.nio.ByteBuffer;
import com.jogamp.newt.Display;
import com.jogamp.newt.Display.PointerIcon;
import com.jogamp.newt.opengl.GLWindow;

GLWindow window;  // window and display variables should already
Display display;  // be availible in PSurfaceJOGL (same var names).

void setup() {
  size(200, 200, P2D);
  
  window = (GLWindow) getSurface().getNative(); // window and display variables should already
  display = window.getScreen().getDisplay();    // be availible in PSurfaceJOGL (same var names).
  
  getGraphics().loadPixels();
  println(Integer.toHexString(getGraphics().pixels[0]));
  
  PGraphics cur = createGraphics(16,16);
  cur.beginDraw();
  cur.rect(0,0,10,10);
  cur.endDraw();
  setCursor(cur, 0,0);
}

public void setCursor(PImage image, int hotspotX, int hotspotY) {
  image.loadPixels();   //Make sure pixels are loaded.
  int[] ipix = image.pixels;
  ByteBuffer pixels = ByteBuffer.allocate(ipix.length * 4);
  pixels.asIntBuffer().put(ipix);
  PixelFormat format = PixelFormat.ARGB8888;
  final Dimension size = new Dimension(image.width, image.height);
  PixelRectangle pixelrect = new PixelRectangle.GenericPixelRect(format, size, 0, false, pixels);
  final PointerIcon pi = display.createPointerIcon(pixelrect, hotspotX, hotspotY);
  display.getEDTUtil().invoke(false, new Runnable() {
    @Override
    public void run() {
      window.setPointerVisible(true);
      window.setPointerIcon(pi);
    }
  });
}

[benfry]

Thanks for looking into it. For at least the next alpha (but probably the 4.0 release as well), I've decided to go a less ambitious direction with the AWT changes for better backwards compatibility.

[github-actions]

This issue has been automatically locked. To avoid confusion with reports that have already been resolved, closed issues are automatically locked 30 days after the last comment. Please open a new issue for related bugs.