Repository navigation
Wayland behavior different from X11? #186
Description
I'm trying to get a keymapper app to work with Wayland the same way it works with X11, to access the window attributes of the focused window and provide app-specific shortcut remappings.
I should note that I didn't have anything to do with creating the keymapper (keyszer) or the app it was forked from (xkeysnail), and I barely understand how they work. I'm just trying to do some minor adaptation of keyszer to get it to work with Wayland. Specifically, with Wayland/GNOME, via communicating with a GNOME shell extension over DBus.
The good news is that I was able to obtain the window attributes and the per-application matching is working.
The bad news is that I'm getting behavior from the virtual keyboard (which I understand is created via evdev) that I've never seen in X11. The logic of the keymapper doesn't seem to be at fault. The keystroke events that show up in the log are the same as when the app is running in X11.
What's happening is odd. This is just an example: Keyszer has the ability to process strings into keystroke output, to "type" macros out with the virtual keyboard. So you can have it type a string like this:
You have double-tapped the Grave key!
It will get translated into things like Shift-Y, O, U, Space, etc.
But when I attempt this kind of output, triggered by an input shortcut, in Wayland, I get this:
you have double-tapped the grave key1
The keystrokes for pressing Shift, typing and releasing keys like "y" and "g" and "1", and releasing Shift, are all there as they are supposed to be, in the log. Adding debugging output shows the key events getting triggered just as they would normally be in X11. But things just aren't coming out right in Wayland. Toggling CapsLock has no effect on the output, unlike X11 where it will reverse the shifted state of normal keys in the string.
This issue seems to be affecting the modifier keys, while leaving the regular part of the keyboard alone. Thus the result being all lowercase, instead of mixed-case. Other shortcuts also don't work, as the app seems to see different modifiers than it should.
The physical device and virtual device seem to be showing up just fine in Wayland, in libinput list-devices:
Device: AT Translated Set 2 keyboard
Kernel: /dev/input/event1
Group: 3
Seat: seat0, default
Capabilities: keyboard
Tap-to-click: n/a
Tap-and-drag: n/a
Tap drag lock: n/a
Left-handed: n/a
Nat.scrolling: n/a
Middle emulation: n/a
Calibration: n/a
Scroll methods: none
Click methods: none
Disable-w-typing: n/a
Accel profiles: n/a
Rotation: n/a
Device: Keyszer (virtual) Keyboard
Kernel: /dev/input/event6
Group: 5
Seat: seat0, default
Capabilities: keyboard pointer
Tap-to-click: n/a
Tap-and-drag: n/a
Tap drag lock: n/a
Left-handed: disabled
Nat.scrolling: disabled
Middle emulation: disabled
Calibration: n/a
Scroll methods: button
Click methods: none
Disable-w-typing: n/a
Accel profiles: flat *adaptive
Rotation: n/a
I'm just wondering if anyone here might have insight into what's happening here. As far as we know, the logic of the keymapper when catching input combos and transforming them into output combos isn't doing anything wrong. The keyboard device is being grabbed like it normally is, and the virtual keyboard is being created and used. But the results are different than expected.
Keymapper app:
https://github.com/joshgoebel/keyszer
Activity
sezanzeb commentedon Feb 12, 2023
As far as I remember keyboard handling in Wayland is individual to each desktop environment. Have you tried it in both gnome and kde on wayland?
RedBearAK commentedon Feb 12, 2023
@sezanzeb
No, and I don't know that there is a method yet to obtain the window information in KDE/Wayland, so that might be slightly pointless. Although maybe I could test it with a completely generic keymap that doesn't care about the window context. For now I've been concentrating on getting this DBus to GNOME shell extension thing working. My config is from Kinto.sh, which does a lot of app-specific shortcut remapping to make everything work like a Mac.
https://github.com/rbreaves/kinto/pull/750/files
So Wayland may be somehow interfering with the output from the virtual keyboard? Is that what you're saying? What can I do to troubleshoot this further?
sezanzeb commentedon Feb 12, 2023
My understanding is very limited. But from what I know wayland is just a protocol, and gnome and kde have to implement all of the stuff themselves.
Is it possible to add some delay between the injected keystrokes? I once had problems when keycodes were injected too fast.
sezanzeb commentedon Feb 12, 2023
The output form
sudo evtestmight be interesting when reproducing the problem in waylandRedBearAK commentedon Feb 12, 2023
Yes, there are a couple of places to do that. I've actually had quite a few problems with "macros" (sets of multiple keystrokes all emitted at once) for quite a long time. Do you have any references for what causes that problem? Is it a defect in
libinputor some kind of device buffer? We've had a really hard time figuring out why that happens.I did try that, but the output in the same environment seems... normal? Although I'm not exactly sure what to look for.
RedBearAK commentedon Feb 12, 2023
Wait, adding a 0.05s delay to a part of the output module does have a significant effect. As in, it makes the intended transformed set of keystrokes "work" as they should.
But there is still the very strange problem where the input combo, that the app window is never supposed to "see", is still being "seen" in addition to the transformed keystrokes. I've been working with this keymapper for a couple of years in X11 environments and have never seen this phenomenon. Which is why I came over here to see if there is a chance the input is not being successfully isolated from Wayland when the keymapper "grabs" the device, which has always worked fine in X11.
RedBearAK commentedon Feb 12, 2023
Alright, I slowed things WAAAY down with a 0.5s delay, and got this:
The macro string is supposed to be:
The CapsLock key that triggers the macro is "seen" by the app immediately (I see an on-screen notification for CapsLock and NumLock keys) and then this macro string is slowly typed out, with each character typed twice. Maybe because the long delay is before sending a sync, and something is interpreting as the key being tapped again? But there are two capital "W", which would require Shift-W in both cases.
I'm not completely familiar with how the event code sequences work, and why this long delay causes this doubling up of the characters.
RedBearAK commentedon Feb 13, 2023
Issue with CapsLock was a red herring, produced by the Boxes application not doing a true "grab" of my real keyboard to isolate CapsLock key presses from the host OS.
So what I'm left with is that in two cases (Kubuntu/X11/KDE in a VM, and now Ubuntu/Wayland/GNOME in a VM) I have extreme unreliability of transformed keyboard shortcuts without adding a significant time delay to the key events coming from the virtual keyboard.
I think I'll close this because the premise was a mistake. The problem is different.
But that doubling of the characters, that' s a clue about something...
sezanzeb commentedon Feb 13, 2023
the output of the macro would have been interesting. The sequence of inputs that causes "you have double-tapped the grave key1" to be written.