keyboard not always minimizing correctly on boot

[danielkvam]

If other layouts than 'us' is selected, or multiple layous, an empty keyboard is stuck on the screen after booting.

I tried
layout=us (only this works)
layout=us,de
layout=de
layout=us,no
layout=us,fr

This happened on a wide screen. using wpe-webkit-mir-kiosk, on ubuntu server 22.04.3.
It does not seem to make a difference what url is in use.

[Saviq]

Hi @danielkvam, thanks for reporting. I can't reproduce here, though.

I used snap set wpe-webkit-mir-kiosk url=https://google.com/ to have an input box, and when things started, the OSK was off screen, when I clicked on the text field, it popped up, and when clicked outside, it's gone again.

I tried layout=us,no and some other combinations.

Can you report more? Show a screenshot? Can you interact with the web view when this happens? Maybe you've a specific resolution in use?

[danielkvam]

Hi, retried with a fresh Ubuntu 22.04 LTS install using the default image in MAAS.
Thereafter manually using the steps in install procedure.txt
Snapped an image of the pc, and how it looks after booting.

Interacting with the website works fine, and selecting an input field will bring up the osk. Deselecting it will remove the osk, as normal.

ubuntu-frame-log.txt
ubuntu-frame-osk-log.txt

I initially believed I had reduced the error to being related to the layout, since it worked fine at one point using the us layout only, but now I am honestly not sure what is going on.

[Saviq]

Thanks @danielkvam, this definitely looks as if the keyboard went away in a wrong way.

So this is only a boot issue, it resolves itself afterwards? Or does the "invisible" OSK remain on screen all the time?

Can you please show your snap list ubuntu-frame ubuntu-frame-osk so we know we're working on the same revisions?

[danielkvam]

This is only a boot issue yes. The keyboard appears after boot for a split second, but disappears, leaving the black bar. The bar will stay until an input field is selected. At which point the keyboard appears and from then will show/hide correctly when using input fields.

Name              Version                 Rev   Tracking   Publisher   Notes
ubuntu-frame      109-mir2.15.0           6626  22/stable  canonical✓  -
ubuntu-frame-osk  47-squeekboard-v1.17.1  326   22/stable  canonical✓  -

[AlanGriffiths]

Looking at the screenshot, this looks like a window management issue. Specifically, it looks like the exclusion zone set by the keyboard is being honoured after the keyboard disappears.

I can think of two possible reasons:

1. the keyboard doesn't release the exclusion zone
2. there is a race in libmiral that sometimes doesn't update the layout after the exclusion zone is released

Given that this isn't seen consistently I favour the second option. It should be possible to confirm this happens by adding window-management-trace to the Frame config:

$ snap set ubuntu-frame config="window-management-trace="

That will add messages to the ubuntu-frame log describing the window management (in excruciating detail).

[Saviq]

Thanks @danielkvam, that sounds like a race and at least this line in the log could point at the problem:

2023-09-12T16:57:51Z ubuntu-frame.daemon[1101]: [2023-09-12 16:57:51.371150] < -warning- > mirserver: Ignoring layer shell protocol violation: wl_surface destroyed before associated zwlr_layer_surface_v1@28

And Alan just beat me to it!

[danielkvam]

I can give you the trace logs. Keep in mind, the logs from last time was on a fresh install of ubuntu server, while this is from a custom image set up by packer-maas. It really did not make a difference when installed on the physical machine.
@AlanGriffiths you got it right, it happens most of the time, but not consistently. I would say it breaks 9/10 times. And it worked fine for a while after changing layouts, hence the title of this post.
Thank you both so much for looking into this!

ubuntu-frame-log-verbose.txt

[AlanGriffiths]

Here is the interesting bit:

The application zone changes from (1920, 696) to (1920, 1080) - which seems reasonable for the keyboard releasing the exclusion zone:

<information> miral::Window Management: advise_application_zone_update updated=(0, 0, (1920, 1080)), original=(0, 0, (1920, 696))

We then loop over the apps:

<information> miral::Window Management: for_each_application

The background window is resized from (1920, 696) to (1920, 1080)

<information> miral::Window Management: info_for ->
<information> miral::Window Management: place_and_size_for_state modifications={state=maximized} window_info={name=, type=freestyle, state=fullscreen, top_left=0, 0, size=(1920, 696), restore_rect=(480, 270, (960, 540)), children={}, depth_layer=0, exclusive_rect=0, preferred_orientation=0xf, confine_pointer=0}
<information> miral::Window Management: confirm_placement_on_display window_info={name=, type=freestyle, state=fullscreen, top_left=0, 0, size=(1920, 696), restore_rect=(480, 270, (960, 540)), children={}, depth_layer=0, exclusive_rect=0, preferred_orientation=0xf, confine_pointer=0}, new_state= maximized, new_placement= (0, 0, (1920, 1080)) -> (0, 0, (1920, 1080))
<information> miral::Window Management: modify_window window_info={name=, type=freestyle, state=fullscreen, top_left=0, 0, size=(1920, 696), restore_rect=(480, 270, (960, 540)), children={}, depth_layer=0, exclusive_rect=0, preferred_orientation=0xf, confine_pointer=0}, modifications={top_left=0, 0, size=(1920, 1080), state=fullscreen}
<information> miral::Window Management: advise_resize window_info={name=, type=freestyle, state=fullscreen, top_left=0, 0, size=(1920, 696), restore_rect=(480, 270, (960, 540)), children={}, depth_layer=0, exclusive_rect=0, preferred_orientation=0xf, confine_pointer=0}, new_size=(1920, 1080)

Then the Cog window is resized from (1920, 696) to (1920, 1080)

<information> miral::Window Management: info_for -> Cog
<information> miral::Window Management: place_and_size_for_state modifications={state=maximized} window_info={name=Cog, type=freestyle, state=fullscreen, top_left=0, 0, size=(1920, 696), restore_rect=(480, 270, (960, 540)), children={}, exclusive_rect=0, preferred_orientation=0xf, confine_pointer=0}
<information> miral::Window Management: confirm_placement_on_display window_info={name=Cog, type=freestyle, state=fullscreen, top_left=0, 0, size=(1920, 696), restore_rect=(480, 270, (960, 540)), children={}, exclusive_rect=0, preferred_orientation=0xf, confine_pointer=0}, new_state= maximized, new_placement= (0, 0, (1920, 1080)) -> (0, 0, (1920, 1080))
<information> miral::Window Management: modify_window window_info={name=Cog, type=freestyle, state=fullscreen, top_left=0, 0, size=(1920, 696), restore_rect=(480, 270, (960, 540)), children={}, exclusive_rect=0, preferred_orientation=0xf, confine_pointer=0}, modifications={top_left=0, 0, size=(1920, 1080), state=fullscreen}
<information> miral::Window Management: advise_resize window_info={name=Cog, type=freestyle, state=fullscreen, top_left=0, 0, size=(1920, 696), restore_rect=(480, 270, (960, 540)), children={}, exclusive_rect=0, preferred_orientation=0xf, confine_pointer=0}, new_size=(1920, 1080)

All that looks good and what I'd expect.

But, the point of this issue, a (1920, 1080) Cog window is not what can be seen on screen.

Then three seconds later we get a request to modify the Cog window to (1920, 696)! We don't do that we make it (1920, 1080) which has no effect.

<information> miral::Window Management: handle_modify_window window_info={name=Cog, type=freestyle, state=fullscreen, top_left=0, 0, size=(1920, 1080), restore_rect=(480, 270, (960, 540)), children={}, exclusive_rect=0, preferred_orientation=0xf, confine_pointer=0}, modifications={size=(1920, 696)}
<information> miral::Window Management: place_and_size_for_state modifications={state=maximized} window_info={name=Cog, type=freestyle, state=fullscreen, top_left=0, 0, size=(1920, 1080), restore_rect=(480, 270, (960, 540)), children={}, exclusive_rect=0, preferred_orientation=0xf, confine_pointer=0}
<information> miral::Window Management: confirm_placement_on_display window_info={name=Cog, type=freestyle, state=fullscreen, top_left=0, 0, size=(1920, 1080), restore_rect=(480, 270, (960, 540)), children={}, exclusive_rect=0, preferred_orientation=0xf, confine_pointer=0}, new_state= maximized, new_placement= (0, 0, (1920, 1080)) -> (0, 0, (1920, 1080))
<information> miral::Window Management: modify_window window_info={name=Cog, type=freestyle, state=fullscreen, top_left=0, 0, size=(1920, 1080), restore_rect=(480, 270, (960, 540)), children={}, exclusive_rect=0, preferred_orientation=0xf, confine_pointer=0}, modifications={top_left=0, 0, size=(1920, 1080), state=fullscreen}

So, my new theory is that the window management logic is correct. But the apps are unaware of the resize. Consequently, the background doesn't repaint and Cog continues to submit (1920, 696) buffers, (Which would be why we see a handle_modify_window message.)

[AlanGriffiths]

There's another report on discourse

[AlanGriffiths]

I'm able to reproduce with the following steps (assuming Frame, OSK, WPE installed etc):

snap set frame-it osk=enable
frame-it wpe-webkit-mir-kiosk.cog

I also see that the background is correctly painted for the fullscreen, while the WPE browser is not.

[AlanGriffiths]

I've not been able to reproduce with other apps: chromium, gedit, glmark2-wayland, kate, gnome-chess, gnome-todo, ...

Maybe this is specific to wpe-webkit-mir-kiosk?

Edit: Also checked our IOT examples

[AlanGriffiths]

Looking at the logs from WPE though, it looks as though we are sizing incorrectly, the question is why.

This is from a run that "works":

$ frame-it ./wayland-debug-wrapper wpe-webkit-mir-kiosk.cog 2>&1 | grep -e 'xdg_toplevel@[0-9]*.configure' -e 'zwp_text_input[^@]*@'
[1123619.126]  -> [email protected](14, "zwp_text_input_manager_v1", 1, new id [unknown]@9)
[1123619.152]  -> [email protected](16, "zwp_text_input_manager_v3", 1, new id [unknown]@10)
[1123651.893]  -> [email protected]_text_input(new id zwp_text_input_v3@18, wl_seat@6)
[1123670.872] [email protected](0, 0, array)
[1123670.971] [email protected](0, 0, array)
[1123670.987] [email protected](1268, 741, array)
[1123764.700] [email protected](1268, 741, array)
[1123764.746] [email protected](wl_surface@13)
[1123803.478] [email protected](1268, 994, array)

Note the final "configure" is with the larger height (994).

Here's a run that "fails"

$ frame-it ./wayland-debug-wrapper wpe-webkit-mir-kiosk.cog 2>&1 | grep -e 'xdg_toplevel@[0-9]*.configure' -e 'zwp_text_input[^@]*@'
[1181972.328]  -> [email protected](14, "zwp_text_input_manager_v1", 1, new id [unknown]@9)
[1181972.351]  -> [email protected](16, "zwp_text_input_manager_v3", 1, new id [unknown]@10)
[1182002.984]  -> [email protected]_text_input(new id zwp_text_input_v3@18, wl_seat@6)
[1182019.583] [email protected](0, 0, array)
[1182019.691] [email protected](0, 0, array)
[1182019.725] [email protected](1268, 994, array)
[1182020.828] [email protected](1268, 741, array)
[1182125.820] [email protected](1268, 741, array)
[1182125.864] [email protected](wl_surface@13)

Note the final "configure" is with the smaller height (741).

From an app that immediately request text input:

$ frame-it ./wayland-debug-wrapper kate 2>&1 | grep -e 'xdg_toplevel@[0-9]*.configure' -e 'zwp_text_input[^@]*@'
[1276816.695]  -> [email protected](15, "zwp_text_input_manager_v2", 1, new id [unknown]@11)
[1276816.706]  -> [email protected]_text_input(new id zwp_text_input_v2@12, wl_seat@6)
[1279179.322] [email protected](0, 0, array[0])
[1279179.528] [email protected](0, 0, array[0])
[1279179.556] [email protected](1268, 994, array[4])
[1279875.322] [email protected](1268, 994, array[8])
[1279875.374] [email protected](12, wl_surface@21)
[1279879.403]  -> [email protected](wl_surface@21)
[1279879.434]  -> [email protected]_surrounding_text("", 0, 0)
[1279879.443]  -> [email protected]_content_type(7, 0)
[1279879.462]  -> [email protected]_cursor_rectangle(0, 0, 1, 18)
[1279879.474]  -> [email protected]_preferred_language("")
[1279879.479]  -> [email protected]_state(12, 3)
[1279879.858]  -> [email protected]_state(12, 0)
[1279879.901]  -> [email protected]_surrounding_text("", 0, 0)
[1279879.908]  -> [email protected]_content_type(7, 0)
[1279879.915]  -> [email protected]_cursor_rectangle(69, 85, 1, 18)
[1279879.952]  -> [email protected]_preferred_language("")
[1279879.956]  -> [email protected]_state(12, 3)
[1279907.453] [email protected](1268, 741, array[8])
[1279908.239]  -> [email protected]_state(12, 0)
[1279908.315]  -> [email protected]_state(12, 0)

[AlanGriffiths]

OK, I'm testing a solution to this (I need to check for unintended side-effects).

It requires changes to Mir, so this won't affect Frame on stable until the next Mir release, but should be available on edge once it lands.

Meanwhile, you can test the proposed solution with:

snap refresh --channel edge/mir-pr3042 ubuntu-frame

[jrmcpeek]

I've not been able to reproduce with other apps: chromium, gedit, glmark2-wayland, kate, gnome-chess, gnome-todo, ...

We've encountered this issue with a custom flutter-based application as well.

I'll pull the test branch and report back!

[jrmcpeek]

Testing was performed using a machine running Ubuntu Core, core22 basis.

Original setup:

ubuntu-frame       99-mir2.13.0            5787   latest/stable  canonical✓  -
ubuntu-frame-osk   44-squeekboard-v1.17.1  316    latest/stable  canonical✓  -

Keyboard space redraw/resize issue would happen on a power cycle about 3 out of every 5 boots.

Running snap refresh --channel edge/mir-pr3042 ubuntu-frame failed for me:

$ snap refresh --channel edge/mir-pr3042 ubuntu-frame

error: requested a non-existing branch on latest/edge for snap "ubuntu-frame": mir-pr3042

I needed to instead run:

snap refresh --channel 22/edge/mir-pr3042 ubuntu-frame

Minor surprise, since I figured latest/edge would track 22/edge, but I guess not 🙂.

Which brings me to:

ubuntu-frame       113-mir2.15.0+dev122    6877   22/edge/…      canonical✓  -
ubuntu-frame-osk   44-squeekboard-v1.17.1  316    latest/stable  canonical✓  -

When using this build, ubuntu-frame starts, followed by the ubuntu-frame-osk (which shows visibly, then dismisses), despite a lack of inputs that should trigger it.

However, our application never loads - we're presented with a greyscale placeholder background.

Checking the logs, I see:

Sep 19 16:34:09 ubuntu systemd[1]: snap.onboard.onboard.service: Scheduled restart job, restart counter is at 5.
Sep 19 16:34:09 ubuntu systemd[1]: Stopped Service for snap application onboard.onboard.
Sep 19 16:34:09 ubuntu systemd[1]: snap.onboard.onboard.service: Start request repeated too quickly.
Sep 19 16:34:09 ubuntu systemd[1]: snap.onboard.onboard.service: Failed with result 'exit-code'.
Sep 19 16:34:09 ubuntu systemd[1]: Failed to start Service for snap application onboard.onboard.

If I ssh in and restart the app, it loads, but does log:

Sep 19 16:35:41 ubuntu systemd[1]: Started Service for snap application onboard.onboard.
Sep 19 16:35:41 ubuntu onboard[4320]: Settings portal not found: Cannot autolaunch D-Bus without X11 $DISPLAY
Sep 19 16:35:41 ubuntu onboard[4320]: gdk_window_get_state: assertion 'GDK_IS_WINDOW (window)' failed

If I instead use the latest edge version of ubuntu-frame snap :

ubuntu-frame       113-mir2.15.0+dev122    6877   22/edge        canonical✓  -
ubuntu-frame-osk   44-squeekboard-v1.17.1  316    latest/stable  canonical✓  -

The app loads as expected when power cycling.

Through 10 power cycles, I was not able to get the keyboard redraw/resize issue to occur when using 22/edge, compared to fairly consistently with 20/stable.

But, 22/edge/mir-pr3042 appears to break our application loading (unless I've unintentionally caused an issue with the snap refreshing).

[AlanGriffiths]

Minor surprise, since I figured latest/edge would track 22/edge, but I guess not 🙂.

Here's the explanation: https://discourse.ubuntu.com/t/psa-retiring-the-latest-tracks-for-ubuntu-frame-graphics-and-mir-test-tools-action-required/37531

[jrmcpeek]

Here's the explanation: https://discourse.ubuntu.com/t/psa-retiring-the-latest-tracks-for-ubuntu-frame-graphics-and-mir-test-tools-action-required/37531

Ah, makes sense. Thanks for the explanation. I'll update our model accordingly.

If there's any additional information I can get you related to the fail-to-start issue using your pr branch, I'm happy to help!

[AlanGriffiths]

@jrmcpeek that PR has landed and is now on edge, but we notices it introduced problems we hadn't seen in initial testing.

There's a follow up 22/edge/mir-pr3048 fixes these (and, hopefully, fixes things for your app too).

[jrmcpeek]

I should be able to test this new branch later today and will report back!

[jrmcpeek]

@AlanGriffiths

Using the following:

ubuntu-frame       113-mir2.15.0+dev124    6905   22/edge/…      canonical✓  -
ubuntu-frame-osk   44-squeekboard-v1.17.1  316    latest/stable  canonical✓  -

This is maybe a little bit better, but still the same issue.

It's about a 50% success rate of the app loading and going full screen or attempting to load, failing a bunch, and the init servicing saying "no more".

100% success rate of the app loading as expected if I simply run a snap restart onboard.onboard or systemctl retsart snap.onboard.onboard after it fails and I ssh into the machine.

[jrmcpeek]

I also performed some testing with 22/stable and confirmed:

• This keyboard redraw issue returns (expected)
• The app fails to start 6 out of 10 attempts

I also re-ran power cycle tests with 20/stable and:

• This keyboard redraw issue returns (expected)
• The app never failed to start (success for 10 of 10 attempts).

[AlanGriffiths]

We have established that this is fixed by canonical/mir#3042. Other issues raised have been split off