11

Fixing Wayland: Xwayland screen casting

 2 years ago
source link: http://blog.davidedmundson.co.uk/blog/xwaylandvideobridge/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

This blog introduces a new tool to make it easy to stream wayland windows and screens to existing applications running under Xwayland that don't have native pipewire support in an easy-to-use manner than retains full user control throughout.

Intro

On my Plasma wayland session right now if I use the screen share function in Discord, I'm presented with the following.

discord_bad-1024x576.png

It doesn't work ?

I have no windows listed, even though I clearly have windows open. The full screen shares are blank.
The same will be true in MS Teams, Slack, Zoom as well as any other Xwayland apps that involve screencasts.

Linux enthusiasts - and by reading developer blogs you're definitely in this camp - will understand why this is and probably know some technique to avoid this involving changing my setup or workflow in some way to work round things on a per-app basis.

For our wider userbase this isn't enough. Wayland is a technical detail and we want any switch has to be as transparent as possible for as many apps as possible for all cases - including cases we haven't thought of.

Introducing XwaylandVideoBridge

With our new tool, written by Aleix Pol and myself, running the workflow is as follows: I click on the share button. Immediately I'm presented with an option to choose what windows or screens I wish to make available to X.

This is now selectable in my existing application. The stream continues until I stop sharing within the application.

discord_final-1024x360.png

Left: a prompt to choose windows to stream.
Right: How it appears after selection

Security

The bridge works via the same mechanisms as any "native wayland" video streaming tool would work, through the XDG Desktop Portals. Getting data through PipeWire as requested it through the portal with explicit user consent.
Whilst the bridge does mean that any X application could eavesdrop the shared window the user still remains completely at the helm of which windows are shared to X applications and most importantly when.

We could go the route of trying to be completely seamless to the X client with N fake windows all forwarding content on demand, but I like the demonstration that it's possible to not break existing user applications without having to compromise on our lofty wayland goals.

Performance

Technically there's an overhead, pragmatically it uses no CPU and any latency is negligible compared to the cost of streaming.

Installation

Grab a flatpak from: Our gitlab builder or from source.

Note it requires a non-released libkpipewire, something the flatpak resolves.

Whilst only tested on a Plasma wayland session, this should be usable by any Wayland desktop with a working xdg-desktop-portal, screencasting and standard system tray.

Usage

Ensure our proxy is running in the background flatpak run org.kde.xwaylandvideobridge optionally setting it to autostart. After that everything should kick in automatically the next time an Xwayland application starts streaming.

How it works under the hood

The inspiration for this came from the debug tool to show pipewire streams in a window whilst we were working on Plasma remote desktop support. If we force that debug tool to run as an Xwayland client, it becomes visible to other Xwayland chat / streaming programs. We had 90% of the code already.

The only remaining step was some sneaky tricks to hide this X11 window from the user's view making it unfocussable, invisible and out of view. We then added detection for when we're being streamed by using the XRecord extension to monitor all clients redirecting the window we own.

It's an excellent example of X11 allowing you to do really, really stupid things, for novel and creative puposes.

Future Plans

This is only an initial Alpha release. How we take it in the future is not completely decided; it might remain a standalone tool moving to flathub or distros, we might proposed it into Plasma 6 by default. There's a possibility the Linux desktop might be at a point where this is redundant.

There's definitely some minor tweaks still to do on the exact workflow.

Please do give feedback and let us know!

http://disuss.kde.org thread

</div


report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK