[iOS] Dictation is not working ("new line" breaks the editor)

[miskotam]

Is this a bug report or feature request? (choose one)

🐞 Bug report

📋 Steps to reproduce

1. Open the classic editor on https://ckeditor.com/ckeditor-5/demo/ in Safari on an iPhone/iPad.
2. Activate dictation
3. Say "test text new line test text"

✅ Expected result

The following text in the editor:
Test text
Test text

❎ Actual result

Test text

📃 Other details that might be useful

It seems like after I say "new line" the dictation is not working anymore.

[Reinmar]

Hi! Thanks for the bug report.

Unfortunately, it's extremely tricky to support normal typing due to lack of reasonable APIs in the browser, so I expect that fixing this issue may be hard if not impossible. In fact, the only option that could work is beforeInput but IDK how well Safari supports it.

[wimleers]

Is this related to IME support, which is being worked on this iteration (see #11787) in #11438 ?

[mlewand]

There was a lot of work recently done in IME realm. It's not done yet but there has been major progress. We should check if this problem is still reproducible there.

Otherwise this sounds indeed like a very challenging issue, we should do a research on how the dictation feature works (what events it fires) and why is this issue happening.

[oleq]

The branch to test new IME support ck/11438-beforeinput-ime-research-vol1.2-android. The status of things is that the first "Test text" shows up, the new line is ignored, and then the rest of the text shows up after dictation.

[wimleers]

FYI: tracking this at https://www.drupal.org/project/drupal/issues/3283802 😊

[mgifford]

Is this an existing Siri/Dictation bug in the iOS operating system or is this something that can be replicated on a Mac?

I don't have an iPhone or iPad to test this with.

[Reinmar]

I wanted to check again this issue again before closing it (it worked on the IME branch some time ago). It turns out it crashes again:

Good thing we have the IME branch on master. Bad thing – we'll need to dive into this bit again 😭

[niegowski]

This happens because of a \n char in the middle of the inserted text.

[niegowski]

While dictating "new paragraph" a double new-line char is provided in the insertText data \n\n and it might be surrounded by other text. I was checking it to understand whether a single new line should be treated as a soft break or a new paragraph.

[niegowski]

While trying to resolve this issue (by splitting a single insertText beforeInput event into a series of smaller text inserts and paragraph inserts) I noticed that it still locks after a new paragraph is inserted. After checking in the plain contenteditable I can see the same broken behavior. In the textarea it does work correctly.

[Reinmar]

Does it lock after we do something or on a plain contenteditable with no additional JS too?

[niegowski]

Does it lock after we do something or on a plain contenteditable with no additional JS too?

Plain contenteditable. After you stop dictation (by touching anywhere or by disabling dictation, the content is inserted but it duplicates the first line.

[niegowski]

And the above locking is not happening every time. If I start with "new-line" and then dictate text it works (but there are some text replications still to resolve).

[wimleers]

So this missed the #12565 deadline, right? Is it still on track for the next iteration/release? 🤞

[niegowski]

Unfortunately, this looks like an issue with Safari or iOS because it fails the same way in CKEditor4 and Tiny:

RPReplay_Final1666969724.MP4

RPReplay_Final1666969202.MP4

It stops updating content (and firing beforeInput events) just after I say "new line". After turning the dictation mode off it fires a single event with the whole dictated content and the provided targetRanges are not correct (should replace the original previously dictated and already inserted text).

[bbenjamin]

The prior comment is not consistent with my experience. The one issue I've encountered with "new line" is Voice Control + Chrome + CKEditor 4. In this situation "new paragraph" still works.

Ck4 + Safari + Voice Control (New Line|Paragraph) work fine for me on OSX and iOS Safari.

CK5 + Voice Control (New Line|Paragraph) does not currently work in any browser/OS I try it in

I just re-evaluated this with the editors from https://ckeditor.com/ckeditor-4/demo/ https://ckeditor.com/ckeditor-5/demo/

[Reinmar]

I tested dictation in iOS 16 on my iPhone. I have similar results to what @niegowski got in CKEditor 4 in TinyMCE. After saying "insert new line" the new text stops being inserted. I need click somewhere for it to appear. But parts of it are duplicated (like Safari loses a track of what was already inserted and where and keeps inserting it). I can record some screencasts later on, but I guess it makes little sense here.

[Reinmar]

CK5 + Voice Control (New Line|Paragraph) does not currently work in any browser/OS I try it in

We retested this on macOS+Voce Control on https://ckeditor5.github.io/docs/nightly/ckeditor5/latest/examples/builds/classic-editor.html and ck/2045 branch.

It works reasonably well on the nightly docs already, so that's good.

On the ck/2045 which brings some more specific handling for macOS's quirks it works even better.

We'll record some screencasts later on.

[Reinmar]

We did a bit more testing together with @bbenjamin and @wimleers. 

On macOS + Voice Control everything works fine on ck/2045. Even stuff like "delete word" and "backspace".

We confirmed that on iOS dictation doesn't work very well – CKE4, Tiny, CKE5. It's an iOS issue. We could report it to Webkit issue tracker.

However, @bbenjamin pointed out that he's using Voice Control on iOS (different thing than the dictation mode available on the keyboard). It has to be enabled in the Settings and it works for English only (or at least it's not available in some languages). But from my tests, based on ck/2045 all works fine as well ❤️.