Improved clipboard functionality with support for multiple formats

[lostminds]

Describe the project you are working on

An editor app

Describe the problem or limitation you are having in your project

Currently the built in clipboard functionality in DisplayServer is limited to getting, setting and checking for a text string (EDIT: get/set text string or get image now with godotengine/godot#63826 merged!) on the system clipboard. This prevents getting and setting data in different formats (like images and other specific data formats) along with setting/getting data in multiple formats for redundancy/support. In other words, OS clipboards these days often supply multiple different formats of the data placed on them, like an graphics in PNG and PDF, or text in plain text and formatted rtf. Based on context the application can then pick the format that best suits the situation when the user pastes something.

This is something that I think would be useful for godot projects where the user will want to exchange data with other applications, but also for a lot of editor-specific cases where the current clipboard implementation seems to be presenting limitations. For example in #6318 with better format-specified support we could paste in other resource types apart from text, and the multiple data formats could be used to have different data sets to paste based on context in cases like #5623 and #955

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Add support for getting, setting and checking for data on the clipboard for multiple data types via a dedicated Clipboard class.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Add a new Clipboard class, accessible as a Clipboard singleton for the system clipboard.

To this class add methods

• accessible
  Returns if we have access to a system clipboard (might not be the case on all platforms Add a way to detect whether the clipboard can be used in HTML5 #5430 )

• clear()
  Clears all clipboard content

• types()
  Returns array of currently available data types on the clipboard.

• has_data_for_type(type)
  Returns if the clipboard has data of the specified type

• set_data_for_type(data,type)
  Sets clipboard data for the specified type

• get_data_for_type(type)
  Returns data for the specified type

• [has/get/set]_string()
  Convenience shorthand for accessing strings on the clipboard, returns and uses String vars

• [has/get/set]_image()
  Convenience shorthand for accessing images on the clipboard, returns and uses Image vars, but could be based on checking for and storing multiple clipboard image formats

This move to a new Clipboard class would I think also make the features easier to find. The current place in DisplayServer is a little odd to me since the clipboard is system wide and not linked to windows or displays. The existing DisplayServer.clipboard_get() etc could be kept as aliases for the new Clipboard.get_string() etc for compatibility but be supplied with a warning when used and eventually deprecated.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No, I do not think so.

Is there a reason why this should be core and not an add-on in the asset library?

Primarily since I think it would be potentially useful to editor development as well. But I think that since we already have some clipboard access functionality in the core this could be added to the core for the same reason. Godot 4 and the following updates have improved UX a lot by better OS integration, and this proposed improved OS clipboard integration could be seen as a step in the same direction.

EDIT: Added mention of godotengine/godot#63826 that adds a not yet documented clipboard_get_image() and clipboard_has_image() to DisplayServer to allow access to images on the clipboard.

[RedMser]

Since it's not commonly used, how about instead of another new singleton that's part of global scope, it could use this pattern like how EditorInterface works:
Have e.g. DisplayServer.get_clipboard_interface() which returns a Clipboard API object like you propose.

[lostminds]

You're probably right that it's not commonly used, but I don't know if adding another singleton would be that much of an overhead. Adding a new Clipboard singleton was mostly to make it easier to find, as to me it makes no sense that it's currently accessed via DisplayServer.

Getting access to an interface for the clipboard functionality would work fine. But to me the natural point to place that would be in OS then, instead of DisplayServer. For example: OS.clipboard.get_image()

However, from a pure functionality perspective all the functionality I propose above (like get/set/has_data_for_type) could just be placed on DisplayServer like they are now, with some clipboard_ prefix or something to show they're clipboard functions.

[Setadokalo]

I started looking into implementing this, and the main design concern on my mind is how generic the clipboard system should be. If we want the clipboard to be able to copy/paste any type (i.e. be easily extensible), the solution that comes to mind to me would be to make ResourceFormatLoaders responsible for loading data from clipboards. That feels somewhat gross to me though, so I'd like to hear other people's takes on it.

[lostminds]

@Setadokalo great to hear you're working on this! I'm not familiar enough with ResourceFormatLoader to know if this is a good or bad idea in general. But as I understand it this will convert data to a Resource (like an Image, Mesh, AudioStream or whatever), and for these data types it might be a good idea to use them to get/set data to the clipboard (like the set_image(image) or get_image() methods). However, apart from images I think most other data formats the clipboard would be useful for do not have builtin Resource types.

The two main use-cases I'm seeing for in extending the clipboard would be introducing custom internal data formats (basically to let you use the clipboard to pass custom data structures around only your apps can parse, correctly identified as this datatype on the clipboard) and to be able to export/import data types via the clipboard that other apps support but do not have built-in support in Godot (like some special audio/video format, svg vector graphics etc).

So for a more general solution I think it's important that it will be possible to access the raw data (like get_data_for_type(type_identifier) and set_data_for_type(data, type_identifier)) without any Resource conversion, but for the common get/set image using a ResourceFormatLoader and ResourceFormatSaver as a middle layer to convert data to/from the clipboard might be a good idea.

Another thing to consider is that unfortunately the type_identifier that are used to identify data types do not seem to be standardized across plattforms. MacOS of uses Uniform Type Identifiers while on Windows it seems there a is small set of system formats but with many other named formats that don't seem to be very standardized, I'm not sure what it's like on Linux. But in both cases it's just a string name, so I think we can leave the platform specific format name the developer to supply, and just make sure to document that these will be platform specific. For the custom data format case it doesn't matter, since you can just use your own custom name.

[Riteo]

@Setadokalo using ResourceLoader feels a bit risky, as it may allow ACE, IIRC.

I wholly agree with the approach of @lostminds, which I also advocated (and still advocate) a lot.

@lostminds regarding identifying types, there is actually a pretty cross platform solution: media (aka MIME) types. The respective DisplayServers can translate the "standard" version for each platform into a MIME without much issue AFAIK.

So, taking your example, has_data_for_type would accept a media type string (like image/png), and if that's available it would return true.

This would give a lot of flexibility to clients. For the common stuff, like images, we can always add helper methods like in the OP.

This is the approach done internally in the Wayland backend and it works pretty well, although I might be a bit biased as Wayland has native support for media types, with multiple different typed data streams too.

[Setadokalo]

X11 also uses MIME for all but a very small set of types, and as far as I can tell (I haven't looked into it in practice yet) on windows most applications support both the extension and the mime as clipboard targets. Mac definitely doesn't, but it looks like there's an API to translate from MIME to the native representation there.

[santosh]

Not sure if this is relevant, but is definitely related.

I'm using Wayland instead of X11. When I copy even text, I see this:

Has this been reported before? And can/should be support for multiple backend be added?

[Riteo]

@santosh Hi, wayland support is WIP and opt-in. Since the error is referring the X11 DisplayServer you're using XWayland. If you're using a DE with an easy way to switch backend, you could try that and see if the error persists.