Benjamin_Loison/adb
1
0
Code Issues 43 Packages Projects Wiki

Interact with virtual display with adb #42

New Issue
Open
opened 2025-06-02 16:22:17 +02:00 by Benjamin_Loison · 9 comments
Benjamin_Loison commented 2025-06-02 16:22:17 +02:00
Owner
Copy Link

Related to Benjamin_Loison/Voice_assistant/issues/30.

Related to [Benjamin_Loison/Voice_assistant/issues/30](https://codeberg.org/Benjamin_Loison/Voice_assistant/issues/30).
Benjamin_Loison commented 2025-06-02 16:24:53 +02:00
Author
Owner
Copy Link

adb shell uiautomator dump, tap and sending text.

Pointer location may help.

Would allow using the device at the same time as app interaction in the background, to send a Signal message while staying on Termux for instance.

It would allow working on multiple apps at the same time, for instance to backup app by app in parallel.

Would help the issue: 
-----BEGIN PGP MESSAGE-----

hF4DTQa9Wom5MBgSAQdAFB92x+n46Hj2AucAYZAIUcj8WrDwltfwaU8E7v3SWQIw
RCuyb94C62wlBpCJ8g99pWEUaw3IOwlfAEBPUass/eJruPKwGZP63CRTPu8HHi3G
0p0BnnOV6jNJ0kVM38AzUOT8L6BLECmqBLPQixSTCufTKfpV4ijXZa+q5Tlt140Y
RG+wyEdcd3zD+GV6I1899mEdKnWpfQfqIoZy7MEjG77ZP1m+QkenYsKVmsHwMwAq
yaHeAwrgXGMveNYHba8RWp2VYmCVEMx6jf2kUr8YlWlui5NRtRUDhZJmg1MmiIqI
w1NWz8eDZTH5aRmOqdGm
=Horx
-----END PGP MESSAGE-----

and the issue Benjamin-Loison/android/issues/46.

`adb shell uiautomator dump`, tap and sending text. Pointer location may help. Would allow using the device at the same time as app interaction in the background, to send a Signal message while staying on Termux for instance. It would allow working on multiple apps at the same time, for instance to backup app by app in parallel. <details> <summary>Would help the issue:</summary> ``` -----BEGIN PGP MESSAGE----- hF4DTQa9Wom5MBgSAQdAFB92x+n46Hj2AucAYZAIUcj8WrDwltfwaU8E7v3SWQIw RCuyb94C62wlBpCJ8g99pWEUaw3IOwlfAEBPUass/eJruPKwGZP63CRTPu8HHi3G 0p0BnnOV6jNJ0kVM38AzUOT8L6BLECmqBLPQixSTCufTKfpV4ijXZa+q5Tlt140Y RG+wyEdcd3zD+GV6I1899mEdKnWpfQfqIoZy7MEjG77ZP1m+QkenYsKVmsHwMwAq yaHeAwrgXGMveNYHba8RWp2VYmCVEMx6jf2kUr8YlWlui5NRtRUDhZJmg1MmiIqI w1NWz8eDZTH5aRmOqdGm =Horx -----END PGP MESSAGE----- ``` </details> and the issue [Benjamin-Loison/android/issues/46](https://github.com/Benjamin-Loison/android/issues/46).
Benjamin_Loison commented 2025-06-02 17:06:59 +02:00
Author
Owner
Copy Link

Note that here I assume that there is no ambiguity about an app being on multiple displays.

Bash script: 
adb shell uiautomator dump
FILE=window_dump.xml
adb pull /sdcard/$FILE
xmllint --format $FILE | sponge $FILE
xmlstarlet sel -t -v '/hierarchy/node/@package' $FILE

Source: the Unix Stack answer 529680

On my Debian 12 GNOME laptop Virtual Machine Manager LineageOS 22.2 virtual machine adb shell uiautomator dump returns the app content that I clicked last, no matter the last Debian 12 GNOME window clicked.

adb shell monkey -p org.fdroid.fdroid 1
Output: 
  bash arg: -p
  bash arg: org.fdroid.fdroid
  bash arg: 1
args: [-p, org.fdroid.fdroid, 1]
 arg: "-p"
 arg: "org.fdroid.fdroid"
 arg: "1"
data="org.fdroid.fdroid"
Events injected: 1
## Network stats: elapsed time=3ms (0ms mobile, 0ms wifi, 3ms not connected)

allows switching the considered app to dump its UI without modifying on which display the app is.

The question is what if during long UI capture, the app changes.

I confirm that both apps are not part of the returned UI.

Screenshot from 2025-06-02 17-11-30.png

window_dump.xml

so can also parse the launcher.

adb shell rm /sdcard/window_dump.xml
rm window_dump.xml

go on F-Droid on a virtual display, Termux on another and on the launcher on the actual screen (at the end).

Bash script: 
adb shell monkey -p org.fdroid.fdroid 1
adb shell uiautomator dump &
adb shell monkey -p com.termux 1
sleep 5
FILE=window_dump.xml
adb pull /sdcard/$FILE
xmllint --format $FILE | sponge $FILE
xmlstarlet sel -t -v '/hierarchy/node/@package' $FILE
Output: 
  bash arg: -p
  bash arg: org.fdroid.fdroid
  bash arg: 1
args: [-p, org.fdroid.fdroid, 1]
 arg: "-p"
 arg: "org.fdroid.fdroid"
 arg: "1"
data="org.fdroid.fdroid"
Events injected: 1
## Network stats: elapsed time=4ms (0ms mobile, 0ms wifi, 4ms not connected)
[1] 215677
  bash arg: -p
  bash arg: com.termux
  bash arg: 1
args: [-p, com.termux, 1]
 arg: "-p"
 arg: "com.termux"
 arg: "1"
data="com.termux"
Events injected: 1
## Network stats: elapsed time=7ms (0ms mobile, 0ms wifi, 7ms not connected)
UI hierchary dumped to: /sdcard/window_dump.xml
[1]+  Done                    adb shell uiautomator dump
/sdcard/window_dump.xml: 1 file pulled, 0 skipped. 15.1 MB/s (17688 bytes in 0.001s)
org.fdroid.fdroid

so it seems to work as wanted.

Like Minecraft screenshot it may post-process after having blocked retrieving the raw data.

Note that here I assume that there is no ambiguity about an app being on multiple displays. <details> <summary>Bash script:</summary> ```bash adb shell uiautomator dump FILE=window_dump.xml adb pull /sdcard/$FILE xmllint --format $FILE | sponge $FILE xmlstarlet sel -t -v '/hierarchy/node/@package' $FILE ``` </details> Source: [the Unix Stack answer 529680](https://unix.stackexchange.com/a/529680) On my Debian 12 GNOME laptop Virtual Machine Manager LineageOS 22.2 virtual machine `adb shell uiautomator dump` returns the app content that I clicked last, no matter the last Debian 12 GNOME window clicked. ```bash adb shell monkey -p org.fdroid.fdroid 1 ``` <details> <summary>Output:</summary> ``` bash arg: -p bash arg: org.fdroid.fdroid bash arg: 1 args: [-p, org.fdroid.fdroid, 1] arg: "-p" arg: "org.fdroid.fdroid" arg: "1" data="org.fdroid.fdroid" Events injected: 1 ## Network stats: elapsed time=3ms (0ms mobile, 0ms wifi, 3ms not connected) ``` </details> allows switching the considered app to dump its UI without modifying on which display the app is. The question is what if during long UI capture, the app changes. I confirm that both apps are not part of the returned UI. ![Screenshot from 2025-06-02 17-11-30.png](/attachments/8dd98872-7886-4518-9923-01da710803b2) [window_dump.xml](/attachments/3da2b68e-4977-4e79-9ef4-d9e22b9314a8) so can also parse the launcher. ```bash adb shell rm /sdcard/window_dump.xml rm window_dump.xml ``` go on F-Droid on a virtual display, Termux on another and on the launcher on the actual screen (at the end). <details> <summary>Bash script:</summary> ```bash adb shell monkey -p org.fdroid.fdroid 1 adb shell uiautomator dump & adb shell monkey -p com.termux 1 sleep 5 FILE=window_dump.xml adb pull /sdcard/$FILE xmllint --format $FILE | sponge $FILE xmlstarlet sel -t -v '/hierarchy/node/@package' $FILE ``` </details> <details> <summary>Output:</summary> ``` bash arg: -p bash arg: org.fdroid.fdroid bash arg: 1 args: [-p, org.fdroid.fdroid, 1] arg: "-p" arg: "org.fdroid.fdroid" arg: "1" data="org.fdroid.fdroid" Events injected: 1 ## Network stats: elapsed time=4ms (0ms mobile, 0ms wifi, 4ms not connected) [1] 215677 bash arg: -p bash arg: com.termux bash arg: 1 args: [-p, com.termux, 1] arg: "-p" arg: "com.termux" arg: "1" data="com.termux" Events injected: 1 ## Network stats: elapsed time=7ms (0ms mobile, 0ms wifi, 7ms not connected) UI hierchary dumped to: /sdcard/window_dump.xml [1]+ Done adb shell uiautomator dump /sdcard/window_dump.xml: 1 file pulled, 0 skipped. 15.1 MB/s (17688 bytes in 0.001s) org.fdroid.fdroid ``` </details> so it seems to work as wanted. Like Minecraft screenshot it may post-process after having blocked retrieving the raw data.
Screenshot from 2025-06-02 17-11-30.png
531 KiB
window_dump.xml
11 KiB
Benjamin_Loison commented 2025-06-02 17:17:00 +02:00
Author
Owner
Copy Link
adb shell input text test

works as wanted after using adb shell monkey.

```bash adb shell input text test ``` works as wanted after using `adb shell monkey`.
Benjamin_Loison commented 2025-06-02 17:20:44 +02:00
Author
Owner
Copy Link
adb shell input tap 977 707

uses actual display even if manually used last an app on the virtual one.

```bash adb shell input tap 977 707 ``` uses actual display even if manually used last an app on the virtual one.
Benjamin_Loison commented 2025-06-02 17:22:26 +02:00
Author
Owner
Copy Link
adb shell input -h
Output: 
Usage: input [<source>] [-d DISPLAY_ID] <command> [<arg>...]

The sources are: 
      touchnavigation
      touchscreen
      joystick
      stylus
      touchpad
      gamepad
      dpad
      mouse
      keyboard
      trackball
      rotaryencoder
[axis_value] represents an option specifying the value of a given axis 
      The syntax is as follows: --axis <axis_name>,<axis_value>
            where <axis_name> is the name of the axis as defined in 
            MotionEvent without the AXIS_ prefix (e.g. SCROLL, X)
      Sample [axis_values] entry: `--axis Y,3`, `--axis SCROLL,-2`

-d: specify the display ID.
      (Default: -1 for key event, 0 for motion event if not specified.)
The commands and default sources are:
      text <string> (Default: keyboard)
      keyevent [--longpress|--duration <duration to hold key down in ms>] [--doubletap] [--async] [--delay <duration between keycodes in ms>] <key code number or name> ... (Default: keyboard)
      tap <x> <y> (Default: touchscreen)
      swipe <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
      draganddrop <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
      press (Default: trackball)
      roll <dx> <dy> (Default: trackball)
      motionevent <DOWN|UP|MOVE|CANCEL> <x> <y> (Default: touchscreen)
      scroll (Default: rotaryencoder). Has the following syntax:
            scroll <x> <y> [axis_value] (for pointer-based sources)
            scroll [axis_value] (for non-pointer-based sources)
            Axis options: SCROLL, HSCROLL, VSCROLL
            None or one or multiple axis value options can be specified.
            To specify multiple axes, use one axis option for per axis.
            Example: `scroll --axis VSCROLL,2 --axis SCROLL,-2.4`
      keycombination [-t duration(ms)] <key code 1> <key code 2> ... (Default: keyboard, the key order is important here.)

adb shell input tap -h
Output: 

Exception occurred while executing 'tap':
java.lang.IllegalArgumentException: Error: Invalid arguments for command: tap
	at com.android.server.input.InputShellCommand.onCommand(InputShellCommand.java:313)
	at com.android.modules.utils.BasicShellCommandHandler.exec(BasicShellCommandHandler.java:97)
	at android.os.ShellCommand.exec(ShellCommand.java:38)
	at com.android.server.input.InputManagerService.onShellCommand(InputManagerService.java:3665)
	at android.os.Binder.shellCommand(Binder.java:1143)
	at android.os.Binder.onTransact(Binder.java:945)
	at android.hardware.input.IInputManager$Stub.onTransact(IInputManager.java:1500)
	at android.os.Binder.execTransactInternal(Binder.java:1411)
	at android.os.Binder.execTransact(Binder.java:1350)

adb shell input --help

Unknown command: --help

adb shell input tap --help
Output: 

Exception occurred while executing 'tap':
java.lang.IllegalArgumentException: Error: Invalid arguments for command: tap
	at com.android.server.input.InputShellCommand.onCommand(InputShellCommand.java:313)
	at com.android.modules.utils.BasicShellCommandHandler.exec(BasicShellCommandHandler.java:97)
	at android.os.ShellCommand.exec(ShellCommand.java:38)
	at com.android.server.input.InputManagerService.onShellCommand(InputManagerService.java:3665)
	at android.os.Binder.shellCommand(Binder.java:1143)
	at android.os.Binder.onTransact(Binder.java:945)
	at android.hardware.input.IInputManager$Stub.onTransact(IInputManager.java:1500)
	at android.os.Binder.execTransactInternal(Binder.java:1411)
	at android.os.Binder.execTransact(Binder.java:1350)
``` adb shell input -h ``` <details> <summary>Output:</summary> ``` Usage: input [<source>] [-d DISPLAY_ID] <command> [<arg>...] The sources are: touchnavigation touchscreen joystick stylus touchpad gamepad dpad mouse keyboard trackball rotaryencoder [axis_value] represents an option specifying the value of a given axis The syntax is as follows: --axis <axis_name>,<axis_value> where <axis_name> is the name of the axis as defined in MotionEvent without the AXIS_ prefix (e.g. SCROLL, X) Sample [axis_values] entry: `--axis Y,3`, `--axis SCROLL,-2` -d: specify the display ID. (Default: -1 for key event, 0 for motion event if not specified.) The commands and default sources are: text <string> (Default: keyboard) keyevent [--longpress|--duration <duration to hold key down in ms>] [--doubletap] [--async] [--delay <duration between keycodes in ms>] <key code number or name> ... (Default: keyboard) tap <x> <y> (Default: touchscreen) swipe <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen) draganddrop <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen) press (Default: trackball) roll <dx> <dy> (Default: trackball) motionevent <DOWN|UP|MOVE|CANCEL> <x> <y> (Default: touchscreen) scroll (Default: rotaryencoder). Has the following syntax: scroll <x> <y> [axis_value] (for pointer-based sources) scroll [axis_value] (for non-pointer-based sources) Axis options: SCROLL, HSCROLL, VSCROLL None or one or multiple axis value options can be specified. To specify multiple axes, use one axis option for per axis. Example: `scroll --axis VSCROLL,2 --axis SCROLL,-2.4` keycombination [-t duration(ms)] <key code 1> <key code 2> ... (Default: keyboard, the key order is important here.) ``` </details> ``` adb shell input tap -h ``` <details> <summary>Output:</summary> ``` Exception occurred while executing 'tap': java.lang.IllegalArgumentException: Error: Invalid arguments for command: tap at com.android.server.input.InputShellCommand.onCommand(InputShellCommand.java:313) at com.android.modules.utils.BasicShellCommandHandler.exec(BasicShellCommandHandler.java:97) at android.os.ShellCommand.exec(ShellCommand.java:38) at com.android.server.input.InputManagerService.onShellCommand(InputManagerService.java:3665) at android.os.Binder.shellCommand(Binder.java:1143) at android.os.Binder.onTransact(Binder.java:945) at android.hardware.input.IInputManager$Stub.onTransact(IInputManager.java:1500) at android.os.Binder.execTransactInternal(Binder.java:1411) at android.os.Binder.execTransact(Binder.java:1350) ``` </details> ``` adb shell input --help ``` ``` Unknown command: --help ``` ``` adb shell input tap --help ``` <details> <summary>Output:</summary> ``` Exception occurred while executing 'tap': java.lang.IllegalArgumentException: Error: Invalid arguments for command: tap at com.android.server.input.InputShellCommand.onCommand(InputShellCommand.java:313) at com.android.modules.utils.BasicShellCommandHandler.exec(BasicShellCommandHandler.java:97) at android.os.ShellCommand.exec(ShellCommand.java:38) at com.android.server.input.InputManagerService.onShellCommand(InputManagerService.java:3665) at android.os.Binder.shellCommand(Binder.java:1143) at android.os.Binder.onTransact(Binder.java:945) at android.hardware.input.IInputManager$Stub.onTransact(IInputManager.java:1500) at android.os.Binder.execTransactInternal(Binder.java:1411) at android.os.Binder.execTransact(Binder.java:1350) ``` </details>
Benjamin_Loison commented 2025-06-02 17:24:17 +02:00
Author
Owner
Copy Link
adb shell input keyboard tap 977 707

does not seem helpful (but does not tap anymore on actual display) based on Default: touchscreen for tap vs Default: keyboard for text.

```bash adb shell input keyboard tap 977 707 ``` does not seem helpful (but does not tap anymore on actual display) based on *Default: touchscreen* for `tap` vs *Default: keyboard* for `text`.
Benjamin_Loison commented 2025-06-02 17:25:09 +02:00
Author
Owner
Copy Link
adb shell input -d keyboard tap 977 707
Output: 

Exception occurred while executing '-d':
java.lang.IllegalArgumentException: Error: Invalid arguments for display ID.
	at com.android.server.input.InputShellCommand.getDisplayId(InputShellCommand.java:178)
	at com.android.server.input.InputShellCommand.onCommand(InputShellCommand.java:284)
	at com.android.modules.utils.BasicShellCommandHandler.exec(BasicShellCommandHandler.java:97)
	at android.os.ShellCommand.exec(ShellCommand.java:38)
	at com.android.server.input.InputManagerService.onShellCommand(InputManagerService.java:3665)
	at android.os.Binder.shellCommand(Binder.java:1143)
	at android.os.Binder.onTransact(Binder.java:945)
	at android.hardware.input.IInputManager$Stub.onTransact(IInputManager.java:1500)
	at android.os.Binder.execTransactInternal(Binder.java:1411)
	at android.os.Binder.execTransact(Binder.java:1350)
``` adb shell input -d keyboard tap 977 707 ``` <details> <summary>Output:</summary> ``` Exception occurred while executing '-d': java.lang.IllegalArgumentException: Error: Invalid arguments for display ID. at com.android.server.input.InputShellCommand.getDisplayId(InputShellCommand.java:178) at com.android.server.input.InputShellCommand.onCommand(InputShellCommand.java:284) at com.android.modules.utils.BasicShellCommandHandler.exec(BasicShellCommandHandler.java:97) at android.os.ShellCommand.exec(ShellCommand.java:38) at com.android.server.input.InputManagerService.onShellCommand(InputManagerService.java:3665) at android.os.Binder.shellCommand(Binder.java:1143) at android.os.Binder.onTransact(Binder.java:945) at android.hardware.input.IInputManager$Stub.onTransact(IInputManager.java:1500) at android.os.Binder.execTransactInternal(Binder.java:1411) at android.os.Binder.execTransact(Binder.java:1350) ``` </details>
Benjamin_Loison commented 2025-06-02 17:26:55 +02:00
Author
Owner
Copy Link
adb shell input -d 4 tap 977 707

does not need adb shell monkey.

Same for adb shell input -d 4 text test it may solve possible keyboard display issue.

```bash adb shell input -d 4 tap 977 707 ``` does not need `adb shell monkey`. Same for `adb shell input -d 4 text test` it may solve possible keyboard display issue.
Benjamin_Loison commented 2025-06-02 17:33:49 +02:00
Author
Owner
Copy Link
adb shell uiautomator dump --help

UI hierchary dumped to: /sdcard/window_dump.xml

adb shell uiautomator --help
Output: 
Usage: uiautomator <subcommand> [options]

Available subcommands:

help: displays help message

runtest: executes UI automation tests
    runtest <class spec> [options]
    <class spec>: <JARS> < -c <CLASSES> | -e class <CLASSES> >
      <JARS>: a list of jar files containing test classes and dependencies. If
        the path is relative, it's assumed to be under /data/local/tmp. Use
        absolute path if the file is elsewhere. Multiple files can be
        specified, separated by space.
      <CLASSES>: a list of test class names to run, separated by comma. To
        a single method, use TestClass#testMethod format. The -e or -c option
        may be repeated. This option is not required and if not provided then
        all the tests in provided jars will be run automatically.
    options:
      --nohup: trap SIG_HUP, so test won't terminate even if parent process
               is terminated, e.g. USB is disconnected.
      -e debug [true|false]: wait for debugger to connect before starting.
      -e runner [CLASS]: use specified test runner class instead. If
        unspecified, framework default runner will be used.
      -e <NAME> <VALUE>: other name-value pairs to be passed to test classes.
        May be repeated.
      -e outputFormat simple | -s: enabled less verbose JUnit style output.

dump: creates an XML dump of current UI hierarchy
    dump [--verbose][file]
      [--compressed]: dumps compressed layout information.
      [file]: the location where the dumped XML should be stored, default is
      /sdcard/window_dump.xml

events: prints out accessibility events until terminated

so there is no workaround to dump UI.

``` adb shell uiautomator dump --help ``` ``` UI hierchary dumped to: /sdcard/window_dump.xml ``` ``` adb shell uiautomator --help ``` <details> <summary>Output:</summary> ``` Usage: uiautomator <subcommand> [options] Available subcommands: help: displays help message runtest: executes UI automation tests runtest <class spec> [options] <class spec>: <JARS> < -c <CLASSES> | -e class <CLASSES> > <JARS>: a list of jar files containing test classes and dependencies. If the path is relative, it's assumed to be under /data/local/tmp. Use absolute path if the file is elsewhere. Multiple files can be specified, separated by space. <CLASSES>: a list of test class names to run, separated by comma. To a single method, use TestClass#testMethod format. The -e or -c option may be repeated. This option is not required and if not provided then all the tests in provided jars will be run automatically. options: --nohup: trap SIG_HUP, so test won't terminate even if parent process is terminated, e.g. USB is disconnected. -e debug [true|false]: wait for debugger to connect before starting. -e runner [CLASS]: use specified test runner class instead. If unspecified, framework default runner will be used. -e <NAME> <VALUE>: other name-value pairs to be passed to test classes. May be repeated. -e outputFormat simple | -s: enabled less verbose JUnit style output. dump: creates an XML dump of current UI hierarchy dump [--verbose][file] [--compressed]: dumps compressed layout information. [file]: the location where the dumped XML should be stored, default is /sdcard/window_dump.xml events: prints out accessibility events until terminated ``` </details> so there is no workaround to dump UI.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Benjamin_Loison
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Benjamin_Loison/adb#42
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?