Interact with virtual display with adb #42

New Issue

2025-06-02T16:22:17+02:00

Benjamin_Loison commented

2025-06-02 16:22:17 +02:00

Related to Benjamin_Loison/Voice_assistant/issues/30.

Related to [Benjamin_Loison/Voice_assistant/issues/30](https://codeberg.org/Benjamin_Loison/Voice_assistant/issues/30).

Benjamin_Loison commented

2025-06-02 16:24:53 +02:00

adb shell uiautomator dump, tap and sending text.

Pointer location may help.

Would allow using the device at the same time as app interaction in the background, to send a Signal message while staying on Termux for instance.

It would allow working on multiple apps at the same time, for instance to backup app by app in parallel.

Would help the issue:

and the issue Benjamin-Loison/android/issues/46.

`adb shell uiautomator dump`, tap and sending text. Pointer location may help. Would allow using the device at the same time as app interaction in the background, to send a Signal message while staying on Termux for instance. It would allow working on multiple apps at the same time, for instance to backup app by app in parallel. <details> <summary>Would help the issue:</summary> ``` -----BEGIN PGP MESSAGE----- hF4DTQa9Wom5MBgSAQdAFB92x+n46Hj2AucAYZAIUcj8WrDwltfwaU8E7v3SWQIw RCuyb94C62wlBpCJ8g99pWEUaw3IOwlfAEBPUass/eJruPKwGZP63CRTPu8HHi3G 0p0BnnOV6jNJ0kVM38AzUOT8L6BLECmqBLPQixSTCufTKfpV4ijXZa+q5Tlt140Y RG+wyEdcd3zD+GV6I1899mEdKnWpfQfqIoZy7MEjG77ZP1m+QkenYsKVmsHwMwAq yaHeAwrgXGMveNYHba8RWp2VYmCVEMx6jf2kUr8YlWlui5NRtRUDhZJmg1MmiIqI w1NWz8eDZTH5aRmOqdGm =Horx -----END PGP MESSAGE----- ``` </details> and the issue [Benjamin-Loison/android/issues/46](https://github.com/Benjamin-Loison/android/issues/46).

Benjamin_Loison commented

2025-06-02 17:06:59 +02:00

Note that here I assume that there is no ambiguity about an app being on multiple displays.

Bash script:

adb shell uiautomator dump
FILE=window_dump.xml
adb pull /sdcard/$FILE
xmllint --format $FILE | sponge $FILE
xmlstarlet sel -t -v '/hierarchy/node/@package' $FILE

Source: the Unix Stack answer 529680

On my Debian 12 GNOME laptop Virtual Machine Manager LineageOS 22.2 virtual machine adb shell uiautomator dump returns the app content that I clicked last, no matter the last Debian 12 GNOME window clicked.

adb shell monkey -p org.fdroid.fdroid 1

Output:

  bash arg: -p
  bash arg: org.fdroid.fdroid
  bash arg: 1
args: [-p, org.fdroid.fdroid, 1]
 arg: "-p"
 arg: "org.fdroid.fdroid"
 arg: "1"
data="org.fdroid.fdroid"
Events injected: 1
## Network stats: elapsed time=3ms (0ms mobile, 0ms wifi, 3ms not connected)

allows switching the considered app to dump its UI without modifying on which display the app is.

The question is what if during long UI capture, the app changes.

I confirm that both apps are not part of the returned UI.

window_dump.xml

so can also parse the launcher.

adb shell rm /sdcard/window_dump.xml
rm window_dump.xml

go on F-Droid on a virtual display, Termux on another and on the launcher on the actual screen (at the end).

Bash script:

adb shell monkey -p org.fdroid.fdroid 1
adb shell uiautomator dump &
adb shell monkey -p com.termux 1
sleep 5
FILE=window_dump.xml
adb pull /sdcard/$FILE
xmllint --format $FILE | sponge $FILE
xmlstarlet sel -t -v '/hierarchy/node/@package' $FILE

Output:

  bash arg: -p
  bash arg: org.fdroid.fdroid
  bash arg: 1
args: [-p, org.fdroid.fdroid, 1]
 arg: "-p"
 arg: "org.fdroid.fdroid"
 arg: "1"
data="org.fdroid.fdroid"
Events injected: 1
## Network stats: elapsed time=4ms (0ms mobile, 0ms wifi, 4ms not connected)
[1] 215677
  bash arg: -p
  bash arg: com.termux
  bash arg: 1
args: [-p, com.termux, 1]
 arg: "-p"
 arg: "com.termux"
 arg: "1"
data="com.termux"
Events injected: 1
## Network stats: elapsed time=7ms (0ms mobile, 0ms wifi, 7ms not connected)
UI hierchary dumped to: /sdcard/window_dump.xml
[1]+  Done                    adb shell uiautomator dump
/sdcard/window_dump.xml: 1 file pulled, 0 skipped. 15.1 MB/s (17688 bytes in 0.001s)
org.fdroid.fdroid

so it seems to work as wanted.

Like Minecraft screenshot it may post-process after having blocked retrieving the raw data.

Note that here I assume that there is no ambiguity about an app being on multiple displays. <details> <summary>Bash script:</summary> ```bash adb shell uiautomator dump FILE=window_dump.xml adb pull /sdcard/$FILE xmllint --format $FILE | sponge $FILE xmlstarlet sel -t -v '/hierarchy/node/@package' $FILE ``` </details> Source: [the Unix Stack answer 529680](https://unix.stackexchange.com/a/529680) On my Debian 12 GNOME laptop Virtual Machine Manager LineageOS 22.2 virtual machine `adb shell uiautomator dump` returns the app content that I clicked last, no matter the last Debian 12 GNOME window clicked. ```bash adb shell monkey -p org.fdroid.fdroid 1 ``` <details> <summary>Output:</summary> ``` bash arg: -p bash arg: org.fdroid.fdroid bash arg: 1 args: [-p, org.fdroid.fdroid, 1] arg: "-p" arg: "org.fdroid.fdroid" arg: "1" data="org.fdroid.fdroid" Events injected: 1 ## Network stats: elapsed time=3ms (0ms mobile, 0ms wifi, 3ms not connected) ``` </details> allows switching the considered app to dump its UI without modifying on which display the app is. The question is what if during long UI capture, the app changes. I confirm that both apps are not part of the returned UI. ![Screenshot from 2025-06-02 17-11-30.png](/attachments/8dd98872-7886-4518-9923-01da710803b2) [window_dump.xml](/attachments/3da2b68e-4977-4e79-9ef4-d9e22b9314a8) so can also parse the launcher. ```bash adb shell rm /sdcard/window_dump.xml rm window_dump.xml ``` go on F-Droid on a virtual display, Termux on another and on the launcher on the actual screen (at the end). <details> <summary>Bash script:</summary> ```bash adb shell monkey -p org.fdroid.fdroid 1 adb shell uiautomator dump & adb shell monkey -p com.termux 1 sleep 5 FILE=window_dump.xml adb pull /sdcard/$FILE xmllint --format $FILE | sponge $FILE xmlstarlet sel -t -v '/hierarchy/node/@package' $FILE ``` </details> <details> <summary>Output:</summary> ``` bash arg: -p bash arg: org.fdroid.fdroid bash arg: 1 args: [-p, org.fdroid.fdroid, 1] arg: "-p" arg: "org.fdroid.fdroid" arg: "1" data="org.fdroid.fdroid" Events injected: 1 ## Network stats: elapsed time=4ms (0ms mobile, 0ms wifi, 4ms not connected) [1] 215677 bash arg: -p bash arg: com.termux bash arg: 1 args: [-p, com.termux, 1] arg: "-p" arg: "com.termux" arg: "1" data="com.termux" Events injected: 1 ## Network stats: elapsed time=7ms (0ms mobile, 0ms wifi, 7ms not connected) UI hierchary dumped to: /sdcard/window_dump.xml [1]+ Done adb shell uiautomator dump /sdcard/window_dump.xml: 1 file pulled, 0 skipped. 15.1 MB/s (17688 bytes in 0.001s) org.fdroid.fdroid ``` </details> so it seems to work as wanted. Like Minecraft screenshot it may post-process after having blocked retrieving the raw data.

Screenshot from 2025-06-02 17-11-30.png

531 KiB

window_dump.xml

11 KiB

Benjamin_Loison commented

2025-06-02 17:17:00 +02:00

adb shell input text test

works as wanted after using adb shell monkey.

```bash adb shell input text test ``` works as wanted after using `adb shell monkey`.

Benjamin_Loison commented

2025-06-02 17:20:44 +02:00

adb shell input tap 977 707

uses actual display even if manually used last an app on the virtual one.

```bash adb shell input tap 977 707 ``` uses actual display even if manually used last an app on the virtual one.

Benjamin_Loison commented

2025-06-02 17:22:26 +02:00

adb shell input -h

Output:

Usage: input [<source>] [-d DISPLAY_ID] <command> [<arg>...]

The sources are: 
      touchnavigation
      touchscreen
      joystick
      stylus
      touchpad
      gamepad
      dpad
      mouse
      keyboard
      trackball
      rotaryencoder
[axis_value] represents an option specifying the value of a given axis 
      The syntax is as follows: --axis <axis_name>,<axis_value>
            where <axis_name> is the name of the axis as defined in 
            MotionEvent without the AXIS_ prefix (e.g. SCROLL, X)
      Sample [axis_values] entry: `--axis Y,3`, `--axis SCROLL,-2`

-d: specify the display ID.
      (Default: -1 for key event, 0 for motion event if not specified.)
The commands and default sources are:
      text <string> (Default: keyboard)
      keyevent [--longpress|--duration <duration to hold key down in ms>] [--doubletap] [--async] [--delay <duration between keycodes in ms>] <key code number or name> ... (Default: keyboard)
      tap <x> <y> (Default: touchscreen)
      swipe <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
      draganddrop <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
      press (Default: trackball)
      roll <dx> <dy> (Default: trackball)
      motionevent <DOWN|UP|MOVE|CANCEL> <x> <y> (Default: touchscreen)
      scroll (Default: rotaryencoder). Has the following syntax:
            scroll <x> <y> [axis_value] (for pointer-based sources)
            scroll [axis_value] (for non-pointer-based sources)
            Axis options: SCROLL, HSCROLL, VSCROLL
            None or one or multiple axis value options can be specified.
            To specify multiple axes, use one axis option for per axis.
            Example: `scroll --axis VSCROLL,2 --axis SCROLL,-2.4`
      keycombination [-t duration(ms)] <key code 1> <key code 2> ... (Default: keyboard, the key order is important here.)

adb shell input tap -h

Output:


Exception occurred while executing 'tap':
java.lang.IllegalArgumentException: Error: Invalid arguments for command: tap
	at com.android.server.input.InputShellCommand.onCommand(InputShellCommand.java:313)
	at com.android.modules.utils.BasicShellCommandHandler.exec(BasicShellCommandHandler.java:97)
	at android.os.ShellCommand.exec(ShellCommand.java:38)
	at com.android.server.input.InputManagerService.onShellCommand(InputManagerService.java:3665)
	at android.os.Binder.shellCommand(Binder.java:1143)
	at android.os.Binder.onTransact(Binder.java:945)
	at android.hardware.input.IInputManager$Stub.onTransact(IInputManager.java:1500)
	at android.os.Binder.execTransactInternal(Binder.java:1411)
	at android.os.Binder.execTransact(Binder.java:1350)

adb shell input --help

Unknown command: --help

adb shell input tap --help

Output:


Exception occurred while executing 'tap':
java.lang.IllegalArgumentException: Error: Invalid arguments for command: tap
	at com.android.server.input.InputShellCommand.onCommand(InputShellCommand.java:313)
	at com.android.modules.utils.BasicShellCommandHandler.exec(BasicShellCommandHandler.java:97)
	at android.os.ShellCommand.exec(ShellCommand.java:38)
	at com.android.server.input.InputManagerService.onShellCommand(InputManagerService.java:3665)
	at android.os.Binder.shellCommand(Binder.java:1143)
	at android.os.Binder.onTransact(Binder.java:945)
	at android.hardware.input.IInputManager$Stub.onTransact(IInputManager.java:1500)
	at android.os.Binder.execTransactInternal(Binder.java:1411)
	at android.os.Binder.execTransact(Binder.java:1350)

``` adb shell input -h ``` <details> <summary>Output:</summary> ``` Usage: input [<source>] [-d DISPLAY_ID] <command> [<arg>...] The sources are: touchnavigation touchscreen joystick stylus touchpad gamepad dpad mouse keyboard trackball rotaryencoder [axis_value] represents an option specifying the value of a given axis The syntax is as follows: --axis <axis_name>,<axis_value> where <axis_name> is the name of the axis as defined in MotionEvent without the AXIS_ prefix (e.g. SCROLL, X) Sample [axis_values] entry: `--axis Y,3`, `--axis SCROLL,-2` -d: specify the display ID. (Default: -1 for key event, 0 for motion event if not specified.) The commands and default sources are: text <string> (Default: keyboard) keyevent [--longpress|--duration <duration to hold key down in ms>] [--doubletap] [--async] [--delay <duration between keycodes in ms>] <key code number or name> ... (Default: keyboard) tap <x> <y> (Default: touchscreen) swipe <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen) draganddrop <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen) press (Default: trackball) roll <dx> <dy> (Default: trackball) motionevent <DOWN|UP|MOVE|CANCEL> <x> <y> (Default: touchscreen) scroll (Default: rotaryencoder). Has the following syntax: scroll <x> <y> [axis_value] (for pointer-based sources) scroll [axis_value] (for non-pointer-based sources) Axis options: SCROLL, HSCROLL, VSCROLL None or one or multiple axis value options can be specified. To specify multiple axes, use one axis option for per axis. Example: `scroll --axis VSCROLL,2 --axis SCROLL,-2.4` keycombination [-t duration(ms)] <key code 1> <key code 2> ... (Default: keyboard, the key order is important here.) ``` </details> ``` adb shell input tap -h ``` <details> <summary>Output:</summary> ``` Exception occurred while executing 'tap': java.lang.IllegalArgumentException: Error: Invalid arguments for command: tap at com.android.server.input.InputShellCommand.onCommand(InputShellCommand.java:313) at com.android.modules.utils.BasicShellCommandHandler.exec(BasicShellCommandHandler.java:97) at android.os.ShellCommand.exec(ShellCommand.java:38) at com.android.server.input.InputManagerService.onShellCommand(InputManagerService.java:3665) at android.os.Binder.shellCommand(Binder.java:1143) at android.os.Binder.onTransact(Binder.java:945) at android.hardware.input.IInputManager$Stub.onTransact(IInputManager.java:1500) at android.os.Binder.execTransactInternal(Binder.java:1411) at android.os.Binder.execTransact(Binder.java:1350) ``` </details> ``` adb shell input --help ``` ``` Unknown command: --help ``` ``` adb shell input tap --help ``` <details> <summary>Output:</summary> ``` Exception occurred while executing 'tap': java.lang.IllegalArgumentException: Error: Invalid arguments for command: tap at com.android.server.input.InputShellCommand.onCommand(InputShellCommand.java:313) at com.android.modules.utils.BasicShellCommandHandler.exec(BasicShellCommandHandler.java:97) at android.os.ShellCommand.exec(ShellCommand.java:38) at com.android.server.input.InputManagerService.onShellCommand(InputManagerService.java:3665) at android.os.Binder.shellCommand(Binder.java:1143) at android.os.Binder.onTransact(Binder.java:945) at android.hardware.input.IInputManager$Stub.onTransact(IInputManager.java:1500) at android.os.Binder.execTransactInternal(Binder.java:1411) at android.os.Binder.execTransact(Binder.java:1350) ``` </details>

Benjamin_Loison commented

2025-06-02 17:24:17 +02:00

adb shell input keyboard tap 977 707

does not seem helpful (but does not tap anymore on actual display) based on Default: touchscreen for tap vs Default: keyboard for text.

```bash adb shell input keyboard tap 977 707 ``` does not seem helpful (but does not tap anymore on actual display) based on *Default: touchscreen* for `tap` vs *Default: keyboard* for `text`.

Benjamin_Loison commented

2025-06-02 17:25:09 +02:00

adb shell input -d keyboard tap 977 707

Output:


Exception occurred while executing '-d':
java.lang.IllegalArgumentException: Error: Invalid arguments for display ID.
	at com.android.server.input.InputShellCommand.getDisplayId(InputShellCommand.java:178)
	at com.android.server.input.InputShellCommand.onCommand(InputShellCommand.java:284)
	at com.android.modules.utils.BasicShellCommandHandler.exec(BasicShellCommandHandler.java:97)
	at android.os.ShellCommand.exec(ShellCommand.java:38)
	at com.android.server.input.InputManagerService.onShellCommand(InputManagerService.java:3665)
	at android.os.Binder.shellCommand(Binder.java:1143)
	at android.os.Binder.onTransact(Binder.java:945)
	at android.hardware.input.IInputManager$Stub.onTransact(IInputManager.java:1500)
	at android.os.Binder.execTransactInternal(Binder.java:1411)
	at android.os.Binder.execTransact(Binder.java:1350)

``` adb shell input -d keyboard tap 977 707 ``` <details> <summary>Output:</summary> ``` Exception occurred while executing '-d': java.lang.IllegalArgumentException: Error: Invalid arguments for display ID. at com.android.server.input.InputShellCommand.getDisplayId(InputShellCommand.java:178) at com.android.server.input.InputShellCommand.onCommand(InputShellCommand.java:284) at com.android.modules.utils.BasicShellCommandHandler.exec(BasicShellCommandHandler.java:97) at android.os.ShellCommand.exec(ShellCommand.java:38) at com.android.server.input.InputManagerService.onShellCommand(InputManagerService.java:3665) at android.os.Binder.shellCommand(Binder.java:1143) at android.os.Binder.onTransact(Binder.java:945) at android.hardware.input.IInputManager$Stub.onTransact(IInputManager.java:1500) at android.os.Binder.execTransactInternal(Binder.java:1411) at android.os.Binder.execTransact(Binder.java:1350) ``` </details>

Benjamin_Loison commented

2025-06-02 17:26:55 +02:00

adb shell input -d 4 tap 977 707

does not need adb shell monkey.

Same for adb shell input -d 4 text test it may solve possible keyboard display issue.

```bash adb shell input -d 4 tap 977 707 ``` does not need `adb shell monkey`. Same for `adb shell input -d 4 text test` it may solve possible keyboard display issue.

Benjamin_Loison commented

2025-06-02 17:33:49 +02:00

adb shell uiautomator dump --help

UI hierchary dumped to: /sdcard/window_dump.xml

adb shell uiautomator --help

Output:

Usage: uiautomator <subcommand> [options]

Available subcommands:

help: displays help message

runtest: executes UI automation tests
    runtest <class spec> [options]
    <class spec>: <JARS> < -c <CLASSES> | -e class <CLASSES> >
      <JARS>: a list of jar files containing test classes and dependencies. If
        the path is relative, it's assumed to be under /data/local/tmp. Use
        absolute path if the file is elsewhere. Multiple files can be
        specified, separated by space.
      <CLASSES>: a list of test class names to run, separated by comma. To
        a single method, use TestClass#testMethod format. The -e or -c option
        may be repeated. This option is not required and if not provided then
        all the tests in provided jars will be run automatically.
    options:
      --nohup: trap SIG_HUP, so test won't terminate even if parent process
               is terminated, e.g. USB is disconnected.
      -e debug [true|false]: wait for debugger to connect before starting.
      -e runner [CLASS]: use specified test runner class instead. If
        unspecified, framework default runner will be used.
      -e <NAME> <VALUE>: other name-value pairs to be passed to test classes.
        May be repeated.
      -e outputFormat simple | -s: enabled less verbose JUnit style output.

dump: creates an XML dump of current UI hierarchy
    dump [--verbose][file]
      [--compressed]: dumps compressed layout information.
      [file]: the location where the dumped XML should be stored, default is
      /sdcard/window_dump.xml

events: prints out accessibility events until terminated

so there is no workaround to dump UI.

``` adb shell uiautomator dump --help ``` ``` UI hierchary dumped to: /sdcard/window_dump.xml ``` ``` adb shell uiautomator --help ``` <details> <summary>Output:</summary> ``` Usage: uiautomator <subcommand> [options] Available subcommands: help: displays help message runtest: executes UI automation tests runtest <class spec> [options] <class spec>: <JARS> < -c <CLASSES> | -e class <CLASSES> > <JARS>: a list of jar files containing test classes and dependencies. If the path is relative, it's assumed to be under /data/local/tmp. Use absolute path if the file is elsewhere. Multiple files can be specified, separated by space. <CLASSES>: a list of test class names to run, separated by comma. To a single method, use TestClass#testMethod format. The -e or -c option may be repeated. This option is not required and if not provided then all the tests in provided jars will be run automatically. options: --nohup: trap SIG_HUP, so test won't terminate even if parent process is terminated, e.g. USB is disconnected. -e debug [true|false]: wait for debugger to connect before starting. -e runner [CLASS]: use specified test runner class instead. If unspecified, framework default runner will be used. -e <NAME> <VALUE>: other name-value pairs to be passed to test classes. May be repeated. -e outputFormat simple | -s: enabled less verbose JUnit style output. dump: creates an XML dump of current UI hierarchy dump [--verbose][file] [--compressed]: dumps compressed layout information. [file]: the location where the dumped XML should be stored, default is /sdcard/window_dump.xml events: prints out accessibility events until terminated ``` </details> so there is no workaround to dump UI.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Benjamin_Loison/adb#42

Interact with virtual display with adb #42

Interact with virtual display with `adb` #42