adb push folder seems to consume as much memory as we try to push #11

New Issue

Benjamin_Loison · 2024-08-27T22:17:08+02:00

Benjamin_Loison commented

2024-08-27 22:17:08 +02:00

Hence, adb is Killed if we try to push more than available memory (including swap).

See Benjamin-Loison/android/issues/46#issuecomment-2282301165.

Hence, `adb` is *Killed* if we try to push more than available memory (including swap). See [Benjamin-Loison/android/issues/46#issuecomment-2282301165](https://github.com/Benjamin-Loison/android/issues/46#issuecomment-2282301165).

Benjamin_Loison changed title from ~~`adb push` seems to consume as much memory as we try to push~~ to `adb push` folder seems to consume as much memory as we try to push

2024-08-27 22:17:49 +02:00

Benjamin_Loison commented

2024-08-27 22:30:49 +02:00

import subprocess
import psutil
import os
from tqdm import tqdm

FOLDER_TO_PUSH = '.SeedVaultAndroidBackup'
FOLDER_TO_PUSH_TO = '/sdcard'
# Source: [the Stack Overflow answer 22103295](https://stackoverflow.com/a/22103295)
MAXIMUM_MEMORY_TO_USE = psutil.virtual_memory().total // 2

# Source: [the Stack Overflow answer 1392549](https://stackoverflow.com/a/1392549)
def get_size(start_path):
    total_size = 0
    for dirpath, dirnames, filenames in os.walk(start_path):
        for f in filenames:
            fp = os.path.join(dirpath, f)
            # skip if it is symbolic link
            if not os.path.islink(fp):
                total_size += os.path.getsize(fp)

    return total_size

def executeCommand(command):
    # Should verify the absence of command injection.
    # It seems fine as SeedVault file/folder names do not seem data name dependent (otherwise it is a hash of it).
    #print(command)
    return subprocess.check_output(command, shell = True).decode('utf-8')[:-1]

def pushFolder(folderToPush):
    folderToPushSize = get_size(folderToPush)
    folderToPushTo = f'{FOLDER_TO_PUSH_TO}/{folderToPush}'
    if folderToPushSize > MAXIMUM_MEMORY_TO_USE:
        print(f"{folderToPush}'s size ({folderToPushSize} bytes) is greater than `MAXIMUM_MEMORY_TO_USE` ({MAXIMUM_MEMORY_TO_USE} bytes), so pushing entry by entry...")
        executeCommand(f'adb shell mkdir {folderToPushTo}')
        for entry in os.listdir(folderToPush):
            entryPath = f'{folderToPush}/{entry}'
            if os.path.isfile(entry):
                # Assume that pushing a file does not require a linear amount of memory.
                # This assumption maybe wrong but it should not matter as no file size should be greater than `MAXIMUM_MEMORY_TO_USE`.
                print(f'Pushing file {entry}...')
                executeCommand(f'adb push {entryPath} {folderToPushTo}')
                progressBar.update(os.path.getsize(entryPath))
            else: # os.path.isdir(entry)
                print(f'Pushing folder {entry}...')
                pushFolder(f'{entryPath}')
    else:
        print(f"Pushing folder {folderToPush}, as its size ({folderToPushSize} bytes) is inferior to `MAXIMUM_MEMORY_TO_USE` ({MAXIMUM_MEMORY_TO_USE} bytes)...")
        executeCommand(f'adb push {folderToPush} {folderToPushTo}')

with tqdm(total = get_size(FOLDER_TO_PUSH)) as progressBar:
    pushFolder(FOLDER_TO_PUSH)

```python import subprocess import psutil import os from tqdm import tqdm FOLDER_TO_PUSH = '.SeedVaultAndroidBackup' FOLDER_TO_PUSH_TO = '/sdcard' # Source: [the Stack Overflow answer 22103295](https://stackoverflow.com/a/22103295) MAXIMUM_MEMORY_TO_USE = psutil.virtual_memory().total // 2 # Source: [the Stack Overflow answer 1392549](https://stackoverflow.com/a/1392549) def get_size(start_path): total_size = 0 for dirpath, dirnames, filenames in os.walk(start_path): for f in filenames: fp = os.path.join(dirpath, f) # skip if it is symbolic link if not os.path.islink(fp): total_size += os.path.getsize(fp) return total_size def executeCommand(command): # Should verify the absence of command injection. # It seems fine as SeedVault file/folder names do not seem data name dependent (otherwise it is a hash of it). #print(command) return subprocess.check_output(command, shell = True).decode('utf-8')[:-1] def pushFolder(folderToPush): folderToPushSize = get_size(folderToPush) folderToPushTo = f'{FOLDER_TO_PUSH_TO}/{folderToPush}' if folderToPushSize > MAXIMUM_MEMORY_TO_USE: print(f"{folderToPush}'s size ({folderToPushSize} bytes) is greater than `MAXIMUM_MEMORY_TO_USE` ({MAXIMUM_MEMORY_TO_USE} bytes), so pushing entry by entry...") executeCommand(f'adb shell mkdir {folderToPushTo}') for entry in os.listdir(folderToPush): entryPath = f'{folderToPush}/{entry}' if os.path.isfile(entry): # Assume that pushing a file does not require a linear amount of memory. # This assumption maybe wrong but it should not matter as no file size should be greater than `MAXIMUM_MEMORY_TO_USE`. print(f'Pushing file {entry}...') executeCommand(f'adb push {entryPath} {folderToPushTo}') progressBar.update(os.path.getsize(entryPath)) else: # os.path.isdir(entry) print(f'Pushing folder {entry}...') pushFolder(f'{entryPath}') else: print(f"Pushing folder {folderToPush}, as its size ({folderToPushSize} bytes) is inferior to `MAXIMUM_MEMORY_TO_USE` ({MAXIMUM_MEMORY_TO_USE} bytes)...") executeCommand(f'adb push {folderToPush} {folderToPushTo}') with tqdm(total = get_size(FOLDER_TO_PUSH)) as progressBar: pushFolder(FOLDER_TO_PUSH) ```

Benjamin_Loison commented

2024-08-27 23:03:18 +02:00

Showing a single progress bar based on the total transferred number of bytes would be nice.

Benjamin_Loison commented

2024-08-27 23:49:43 +02:00

Above algorithm looks correct to me only a single GB differs according to du -sh but the implementation differ:

du --version

du (GNU coreutils) 9.4
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Torbjörn Granlund, David MacKenzie, Paul Eggert,
and Jim Meyering.

adb shell du --version

toybox 0.8.10-android

and checking files and folders quickly everything look correct.

I guess that if it is incorrectly implemented, then the restoration will fail or some data will be missing. It seems that the restoration went through successfully and I do not notice any data missing, except those explicitly mentioned in Benjamin-Loison/android/issues/46#issuecomment-2282301165. I will try to remember to come back to this issue if I notice later on some other data missing while they should not according to the mentioned Git issue comment.

Above algorithm looks correct to me only a single GB differs according to `du -sh` but the implementation differ: ```bash du --version ``` ``` du (GNU coreutils) 9.4 Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Written by Torbjörn Granlund, David MacKenzie, Paul Eggert, and Jim Meyering. ``` ```bash adb shell du --version ``` ``` toybox 0.8.10-android ``` and checking files and folders quickly everything look correct. I guess that if it is incorrectly implemented, then the restoration will fail or some data will be missing. It seems that the restoration went through successfully and I do not notice any data missing, except those explicitly mentioned in [Benjamin-Loison/android/issues/46#issuecomment-2282301165](https://github.com/Benjamin-Loison/android/issues/46#issuecomment-2282301165). I will try to remember to come back to this issue if I notice later on some other data missing while they should not according to the mentioned Git issue comment.

Sign in to join this conversation.

No Label

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Benjamin_Loison/adb#11

adb push folder seems to consume as much memory as we try to push #11

`adb push` folder seems to consume as much memory as we try to push #11