Exploring X11 windows and devices and emulating mouse movements and keystrokes
source link: https://relational-pipes.globalcode.info/v_0/examples-x11-basics.xhtml
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Exploring X11 windows and devices and emulating mouse movements and keystrokes
The X Window System or X11 comes from 1984 and is still widely used (despite we have some other options like Wayland). This protocol and set of libraries and interfaces gives us GUI (graphical user interface), manages our displays, windows, keyboards and mice.
Since Relational pipes
v0.18 we can interact with this wonderful technology through the relpipe-in-x11 and relpipe-out-x11 tools.
In this example we will show how to acquire information about the input devices, screens and windows or capture and emit X11 events (key presses and mouse movements).
Listing windows
In X11 almost everything is a window. Especially classic applications like XCalc are composed from many windows. We list all windows of the current display this way:
relpipe-in-x11 --list-windows true | relpipe-out-tabular | less -RSi relpipe-in-x11 --list-windows true | relpipe-out-recfile | less
and realize that there are so many windows (hundreds on a common desktop) with so much metadata (long names, titles etc.).
%rec: x11_window %type: id int %type: root int %type: parent int %type: level int %type: process int %type: x int %type: y int %type: width int %type: height int id: 1254 root: 1254 parent: 0 level: 0 process: -1 name: res_class: res_name: x: 0 y: 0 width: 3840 height: 2160 … id: 125829135 root: 1254 parent: 155189297 level: 3 process: 10046 name: Exploring X11 windows and devices and emulating mouse movements and keystrokes – Relational pipes - Falkon res_class: Falkon res_name: Falkon Browser x: 0 y: 0 width: 3840 height: 2044 …
Windows are identified by their id and organized hierarchically – have parent attribute that contains ID of their ancestor in the tree.
There is also the process attribute that contains the PID (process ID) of the running program that created this window.
However we should be aware that this information is voluntarily provided by the X11 clients, not managed by the X11 server, and thus might be missing (quite often) or even misleading (rarely).
In order to not get lost in so many windows, we can run a separate X11 server and put only some programs/windows on its display:
# Run a nested X11 server: Xephyr :8 -screen 640x480 # Run an application on its display: DISPLAY=:8 xcalc # List the windows: DISPLAY=:8 relpipe-in-x11 --list-windows true | relpipe-out-tabular
This classic calculator consist of just 51 windows:
x11_window: ╭──────────────┬────────────────┬──────────────────┬─────────────────┬───────────────────┬───────────────┬────────────────────┬───────────────────┬─────────────┬─────────────┬─────────────────┬──────────────────╮ │ id (integer) │ root (integer) │ parent (integer) │ level (integer) │ process (integer) │ name (string) │ res_class (string) │ res_name (string) │ x (integer) │ y (integer) │ width (integer) │ height (integer) │ ├──────────────┼────────────────┼──────────────────┼─────────────────┼───────────────────┼───────────────┼────────────────────┼───────────────────┼─────────────┼─────────────┼─────────────────┼──────────────────┤ │ 681 │ 681 │ 0 │ 0 │ -1 │ │ │ │ 0 │ 0 │ 640 │ 480 │ │ 2097174 │ 681 │ 681 │ 1 │ -1 │ Calculator │ XCalc │ xcalc │ 0 │ 0 │ 226 │ 304 │ │ 2097175 │ 681 │ 2097174 │ 2 │ -1 │ │ │ │ 0 │ 0 │ 226 │ 304 │ │ 2097180 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 272 │ 40 │ 26 │ │ 2097181 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 272 │ 40 │ 26 │ │ 2097182 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 272 │ 40 │ 26 │ │ 2097183 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 272 │ 40 │ 26 │ │ 2097184 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 272 │ 40 │ 26 │ │ 2097185 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 242 │ 40 │ 26 │ │ 2097186 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 242 │ 40 │ 26 │ │ 2097187 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 242 │ 40 │ 26 │ │ 2097188 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 242 │ 40 │ 26 │ │ 2097189 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 242 │ 40 │ 26 │ │ 2097190 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 212 │ 40 │ 26 │ │ 2097191 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 212 │ 40 │ 26 │ │ 2097192 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 212 │ 40 │ 26 │ │ 2097193 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 212 │ 40 │ 26 │ │ 2097194 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 212 │ 40 │ 26 │ │ 2097195 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 182 │ 40 │ 26 │ │ 2097196 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 182 │ 40 │ 26 │ │ 2097197 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 182 │ 40 │ 26 │ │ 2097198 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 182 │ 40 │ 26 │ │ 2097199 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 182 │ 40 │ 26 │ │ 2097200 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 152 │ 40 │ 26 │ │ 2097201 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 152 │ 40 │ 26 │ │ 2097202 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 152 │ 40 │ 26 │ │ 2097203 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 152 │ 40 │ 26 │ │ 2097204 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 152 │ 40 │ 26 │ │ 2097205 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 122 │ 40 │ 26 │ │ 2097206 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 122 │ 40 │ 26 │ │ 2097207 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 122 │ 40 │ 26 │ │ 2097208 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 122 │ 40 │ 26 │ │ 2097209 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 122 │ 40 │ 26 │ │ 2097210 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 92 │ 40 │ 26 │ │ 2097211 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 92 │ 40 │ 26 │ │ 2097212 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 92 │ 40 │ 26 │ │ 2097213 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 92 │ 40 │ 26 │ │ 2097214 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 92 │ 40 │ 26 │ │ 2097215 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 180 │ 62 │ 40 │ 26 │ │ 2097216 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 136 │ 62 │ 40 │ 26 │ │ 2097217 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 92 │ 62 │ 40 │ 26 │ │ 2097218 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 48 │ 62 │ 40 │ 26 │ │ 2097219 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 62 │ 40 │ 26 │ │ 2097220 │ 681 │ 2097175 │ 3 │ -1 │ │ │ │ 4 │ 2 │ 216 │ 46 │ │ 2097221 │ 681 │ 2097220 │ 4 │ -1 │ │ │ │ 6 │ 2 │ 204 │ 38 │ │ 2097222 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 127 │ 21 │ 18 │ 15 │ │ 2097223 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 91 │ 21 │ 34 │ 15 │ │ 2097224 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 61 │ 21 │ 26 │ 15 │ │ 2097225 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 31 │ 21 │ 26 │ 15 │ │ 2097226 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 4 │ 23 │ 26 │ 15 │ │ 2097227 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 18 │ 2 │ 186 │ 17 │ │ 2097228 │ 681 │ 2097221 │ 5 │ -1 │ │ │ │ 4 │ 2 │ 10 │ 15 │ ╰──────────────┴────────────────┴──────────────────┴─────────────────┴───────────────────┴───────────────┴────────────────────┴───────────────────┴─────────────┴─────────────┴─────────────────┴──────────────────╯ Record count: 52
Filtering windows with SQL and recursive CTE
Another way to deal with the abundance of windows is SQL, especially the recursive CTE (common table expressions). This kind of SQL expressions allows us to work with tree structures and filter all descendants of given parent window.
#!/bin/bash
get_x11_child_windows() {
SQL="
WITH
RECURSIVE w AS (
SELECT * FROM x11_window WHERE name = ?
UNION ALL
SELECT ch.* FROM x11_window ch JOIN w ON (w.id = ch.parent)
)
SELECT * FROM w
ORDER BY x, y
";
relpipe-in-x11 --list-windows true \
| relpipe-tr-sql \
--relation 'window' "$SQL" \
--parameter "$1"
}
format_result() { [[ -t 1 ]] && relpipe-out-tabular || cat; }
get_x11_child_windows "Calculator" | format_result
When we run this script on a desktop with many windows and running XCalc, we will get a listing of the only 51 windows – similar to the listing above.
The root window (screen) is not included in this case, but we can also add it using another UNION ALL.
Thanks to Relational pipes modular design we can use also other transformation tools and languages – e.g. Scheme (relpipe-tr-scheme), AWK (relpipe-tr-awk) or XPath (relpipe-tr-xpath).
Embedding and composing windows
Once we know the ID of a particular window, we can have some fun and run another application inside it:
/usr/lib/xscreensaver/euphoria --regular -window-id 148897953 mpv --wid=148897935 "film.mkv"
So we can e.g. watch a film or enjoy a screen saver while calculating:
And we can setup everything automatically whithout typing the IDs by hand:
read_nullbyte() { local IFS=; for v in "$@"; do export "$v"; read -r -d '' "$v"; done }
./the-x11-script-listed-above.sh \
| relpipe-tr-xpath \
--relation '.*' \
--where 'level >= 5' \
--output-attribute 'wid' integer 'id' \
| relpipe-out-nullbyte | while read_nullbyte wid; do
sleep 1;
/usr/lib/xscreensaver/euphoria --regular -window-id "$wid" &
done
and experience pure euphoria:
This X11 feature can be used also in much more useful way for composing GUI from several separate programs that might be even written in different programming languages.
Listing input devices
In order to list our keyboards, mice and similar devices we will invoke:
relpipe-in-x11 | relpipe-out-tabular
or we can explicitly say --list-input-devices true; in both cases we will get a listing similar to this one:
x11_input_device: ╭──────────────┬───────────────────────────────────────────────────────────┬───────────────╮ │ id (integer) │ name (string) │ type (string) │ ├──────────────┼───────────────────────────────────────────────────────────┼───────────────┤ │ 2 │ Virtual core pointer │ │ │ 3 │ Virtual core keyboard │ │ │ 4 │ Virtual core XTEST pointer │ │ │ 5 │ Virtual core XTEST keyboard │ │ │ 6 │ Power Button │ keyboard │ │ 7 │ Video Bus │ keyboard │ │ 8 │ Power Button │ keyboard │ │ 10 │ Logitech USB Trackball │ mouse │ │ 11 │ ZSA Technology Labs Inc ErgoDox EZ Shine │ keyboard │ │ 12 │ ZSA Technology Labs Inc ErgoDox EZ Shine │ mouse │ │ 13 │ ZSA Technology Labs Inc ErgoDox EZ Shine System Control │ keyboard │ │ 14 │ ZSA Technology Labs Inc ErgoDox EZ Shine Consumer Control │ mouse │ │ 15 │ ZSA Technology Labs Inc ErgoDox EZ Shine Keyboard │ keyboard │ │ 16 │ AT Translated Set 2 keyboard │ keyboard │ │ 17 │ ZSA Technology Labs Inc ErgoDox EZ Shine Consumer Control │ keyboard │ │ 9 │ 3Dconnexion 3Dconnexion Universal Receiver Mouse │ mouse │ ╰──────────────┴───────────────────────────────────────────────────────────┴───────────────╯ Record count: 16
Device IDs can be used to identify the source of input events and their filtering.
Capturing input events
Once we connect to an X11 server (through the relpipe-in-x11 X11 client in our case)
we can listen to the input events and monitor the key codes and mouse movements.
relpipe-in-x11 --list-input-events true | relpipe-out-csv
To see the events immediatelly, it is important to use an output filter that does no buffering (e.g. relpipe-out-csv or relpipe-out-gui).
"device","type","state","key","button","x","y" "11","key","released","36","-1","1405","1416" "11","key","pressed","38","-1","1405","1416" "11","key","released","38","-1","1405","1416" "11","key","pressed","43","-1","1405","1416" "11","key","released","43","-1","1405","1416" "11","key","pressed","32","-1","1405","1416" "11","key","released","32","-1","1405","1416" "11","key","pressed","44","-1","1405","1416" "11","key","released","44","-1","1405","1416" "9","motion",,"-1","-1","1405","1416" "9","motion",,"-1","-1","1405","1417" "9","motion",,"-1","-1","1406","1418" "9","motion",,"-1","-1","1406","1419" "9","button","pressed","-1","1","1406","1420" "9","button","released","-1","1","1406","1420" "9","motion",,"-1","-1","1406","1420" "9","motion",,"-1","-1","1405","1420"
In order to capture events from a different X11 server, we set the DISPLAY environment variable.
This way, we can also capture events remotely over SSH.
ssh example.com DISPLAY=:0 relpipe-in-x11 --list-input-events true | relpipe-out-csv
We can log the events to a file or forward them to another X11 server through relpipe-out-x11.
Or we can multiplex the events and forward them to several X11 servers e.g. to run simultaneous tests of different versions or configurations or certain GUI application.
Simulating input events
With the relpipe-out-x11 client we can connect to an X11 server and emit input events like they come from a keyboard or mouse
(actually they would come from a virtual/simulated one).
This command reads relational data with same structure as produced by its counterpart relpipe-in-x11.
We can use data captured earlier or create some new one using any tool like
relpipe-in-cli or other relpipe-in-* optionally transformed through a relpipe-tr-* filter.
We may also translate other signals like MIDI events to X11 ones (relpipe-in-jack).
For example, the following command moves the cursor to the coordinates 100,200 and then to 640,480.
Such events might be produced also in a shell loop and passed through the pipe to a single relpipe-out-x11 process
(i.e. without connecting and disconnecting the X11 server on each event).
relpipe-in-cli \ --relation "x11_input_event" \ --attribute "type" string \ --attribute "x" integer \ --attribute "y" integer \ --records \ "motion" "100" "200" \ "motion" "640" "480" \ | relpipe-out-x11
The relpipe-out-x11 tool has two options:
--dry-run true that suppresses the events and just tests validity of the input; and
--debug true that prints the events in a text form (XML) on STDERR (so we can see that something comes through).
relpipe-in-x11 --list-input-events true | relpipe-out-x11 --dry-run true --debug true
Using SSH, DDC, relpipe-in-x11 and relpipe-out-x11,
we can build a software replacement of a KVM switch and control several computers from one seat (keyboard, video, mouse).
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK