Benjamin_Loison/Robust_image_source_identification_on_modern_smartphones
1
0
Fork 0
Code Issues 69 Pull Requests Packages Projects Releases 1 Wiki Activity

Optimize execution speed #62

New Issue
Open
opened 2024-04-30 02:34:41 +02:00 by Benjamin_Loison · 13 comments
Benjamin_Loison commented 2024-04-30 02:34:41 +02:00
Owner
Copy Link
Copy Source

Initially at #8.

https://github.com/JeffOwOSun/gpu-bm3d
https://github.com/OpenImageDenoise/oidn

Related to #53.

Initially at #8. https://github.com/JeffOwOSun/gpu-bm3d https://github.com/OpenImageDenoise/oidn Related to #53.
Benjamin_Loison commented 2024-04-30 02:48:17 +02:00
Author
Owner
Copy Link
Copy Source
from utils import silentTqdm
tqdm = silentTqdm

Need builtins import otherwise get:

Traceback (most recent call last):
  File "/home/benjamin/robust_image_source_identification_on_modern_smartphones/datasets/raise/./split_and_compare_prnus_of_subgroups.py", line 45, in <module>
    print('Alpha')
  File "/home/benjamin/robust_image_source_identification_on_modern_smartphones/datasets/raise/utils.py", line 145, in print
    print(datetime.now(), *toPrint)
  File "/home/benjamin/robust_image_source_identification_on_modern_smartphones/datasets/raise/utils.py", line 145, in print
    print(datetime.now(), *toPrint)
  File "/home/benjamin/robust_image_source_identification_on_modern_smartphones/datasets/raise/utils.py", line 145, in print
    print(datetime.now(), *toPrint)
  [Previous line repeated 996 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
```py from utils import silentTqdm tqdm = silentTqdm ``` Need `builtins` import otherwise get: ``` Traceback (most recent call last): File "/home/benjamin/robust_image_source_identification_on_modern_smartphones/datasets/raise/./split_and_compare_prnus_of_subgroups.py", line 45, in <module> print('Alpha') File "/home/benjamin/robust_image_source_identification_on_modern_smartphones/datasets/raise/utils.py", line 145, in print print(datetime.now(), *toPrint) File "/home/benjamin/robust_image_source_identification_on_modern_smartphones/datasets/raise/utils.py", line 145, in print print(datetime.now(), *toPrint) File "/home/benjamin/robust_image_source_identification_on_modern_smartphones/datasets/raise/utils.py", line 145, in print print(datetime.now(), *toPrint) [Previous line repeated 996 more times] RecursionError: maximum recursion depth exceeded while calling a Python object ```
Benjamin_Loison commented 2024-04-30 02:51:19 +02:00
Author
Owner
Copy Link
Copy Source
2024-04-30 02:52:09.202336 Alpha
2024-04-30 02:52:10.477927 Beta
2024-04-30 02:52:31.878341 Charlie
2024-04-30 02:52:31.878389 Delta
2024-04-30 02:52:53.226386 Echo
2024-04-30 02:52:53.259885 Alpha
2024-04-30 02:52:54.524432 Beta

+            print('Alpha')
             singleColorChannelImages = {color: rescaleIfNeeded(getColorChannel(imageFilePath, color), minColor, maxColor) for color in Color}
+            print('Beta')
             multipleColorsImage = mergeSingleColorChannelImagesAccordingToBayerFilter(singleColorChannelImages)
+            print('Charlie')
 
             if computeExtremes:
                 minColor, maxColor = updateExtremes(multipleColorsImage, minColor, maxColor)
                 continue
 
             singleColorChannelDenoisedImages = {color: denoise(singleColorChannelImages[color], DENOISER) for color in Color}
+            print('Delta')
            multipleColorsDenoisedImage = mergeSingleColorChannelImagesAccordingToBayerFilter(singleColorChannelDenoisedImages)
+           print('Echo')

             singleColorChannelImages = {color: rescaleIfNeeded(getColorChannel(imageFilePath, color), minColor, maxColor) for color in Color}
+            #plt.imshow(singleColorChannelImages[Color.RED])
+            #plt.show()
+            #exit(1)
+            print('Beta')
             multipleColorsImage = mergeSingleColorChannelImagesAccordingToBayerFilter(singleColorChannelImages)
+            plt.imshow(multipleColorsImage)
+            plt.show()
+            exit(1)

Thanks to the Stack Overflow answer 5347492:

2024-04-30 03:08:07.431852 Alpha
2024-04-30 03:08:08.714293 Beta
2024-04-30 03:08:08.760981 Charlie
2024-04-30 03:08:08.761045 Delta
2024-04-30 03:08:08.807088 Echo
2024-04-30 03:08:08.842309 Alpha
2024-04-30 03:08:10.095110 Beta
2024-04-30 03:08:10.151470 Charlie
2024-04-30 03:08:10.151533 Delta
2024-04-30 03:08:10.194094 Echo
2024-04-30 03:08:10.372008 Alpha

So reduced 21 seconds per merge to 1 second.

``` 2024-04-30 02:52:09.202336 Alpha 2024-04-30 02:52:10.477927 Beta 2024-04-30 02:52:31.878341 Charlie 2024-04-30 02:52:31.878389 Delta 2024-04-30 02:52:53.226386 Echo 2024-04-30 02:52:53.259885 Alpha 2024-04-30 02:52:54.524432 Beta ``` ```diff + print('Alpha') singleColorChannelImages = {color: rescaleIfNeeded(getColorChannel(imageFilePath, color), minColor, maxColor) for color in Color} + print('Beta') multipleColorsImage = mergeSingleColorChannelImagesAccordingToBayerFilter(singleColorChannelImages) + print('Charlie') if computeExtremes: minColor, maxColor = updateExtremes(multipleColorsImage, minColor, maxColor) continue singleColorChannelDenoisedImages = {color: denoise(singleColorChannelImages[color], DENOISER) for color in Color} + print('Delta') multipleColorsDenoisedImage = mergeSingleColorChannelImagesAccordingToBayerFilter(singleColorChannelDenoisedImages) + print('Echo') ``` ```diff singleColorChannelImages = {color: rescaleIfNeeded(getColorChannel(imageFilePath, color), minColor, maxColor) for color in Color} + #plt.imshow(singleColorChannelImages[Color.RED]) + #plt.show() + #exit(1) + print('Beta') multipleColorsImage = mergeSingleColorChannelImagesAccordingToBayerFilter(singleColorChannelImages) + plt.imshow(multipleColorsImage) + plt.show() + exit(1) ``` Thanks to [the Stack Overflow answer 5347492](https://stackoverflow.com/a/5347492): ``` 2024-04-30 03:08:07.431852 Alpha 2024-04-30 03:08:08.714293 Beta 2024-04-30 03:08:08.760981 Charlie 2024-04-30 03:08:08.761045 Delta 2024-04-30 03:08:08.807088 Echo 2024-04-30 03:08:08.842309 Alpha 2024-04-30 03:08:10.095110 Beta 2024-04-30 03:08:10.151470 Charlie 2024-04-30 03:08:10.151533 Delta 2024-04-30 03:08:10.194094 Echo 2024-04-30 03:08:10.372008 Alpha ``` So reduced 21 seconds per merge to 1 second.
Benjamin_Loison referenced this issue from a commit 2024-04-30 03:36:33 +02:00
Simplify loops
Benjamin_Loison commented 2024-04-30 03:38:26 +02:00
Author
Owner
Copy Link
Copy Source

Related to issues/25#issuecomment-1476.

Related to [issues/25#issuecomment-1476](https://gitea.lemnoslife.com/Benjamin_Loison/Robust_image_source_identification_on_modern_smartphones/issues/25#issuecomment-1476).
Benjamin_Loison added the enhancementepicmedium priority labels 2024-04-30 04:11:54 +02:00
Benjamin_Loison commented 2024-05-13 16:37:48 +02:00
Author
Owner
Copy Link
Copy Source

It does not seem that can only load part of a raw image with rawpy, if even possible at all.

In getRawColorChannel:

rawImageVisible = raw.raw_image_visible.copy()[:100, :100]

or

rawImageVisible = raw.raw_image_visible[:100, :100].copy()

do not result in significant execution speed increase.

So only loading the specified color channel might not help too.

However, can keep the crop loaded into memory such that do not have to load the whole image when want to work with it. Assuming loading the whole raw images is the issue, can just load to memory getSingleColorChannelImages.

Could save the crops as Numpy arrays on hard disk if want to avoid loading time.
An interesting thing would be to save the images in a format such that can load only part of the image.
The question is what data structure do I precisely want to store this way?

getSingleColorChannelImages calls getColorChannel which calls getRawColorChannel. So a dict or similar seems to make sense. Maybe partial read is not compatible with dict and will have to make a file per color channel.

It does not seem that can only load part of a raw image with rawpy, if even possible at all. In `getRawColorChannel`: ```python rawImageVisible = raw.raw_image_visible.copy()[:100, :100] ``` or ```python rawImageVisible = raw.raw_image_visible[:100, :100].copy() ``` do not result in significant execution speed increase. So only loading the specified color channel might not help too. However, can keep the crop loaded into memory such that do not have to load the whole image when want to work with it. Assuming loading the whole raw images is the issue, can just load to memory `getSingleColorChannelImages`. Could save the crops as Numpy arrays on hard disk if want to avoid loading time. An interesting thing would be to save the images in a format such that can load only part of the image. The question is what data structure do I precisely want to store this way? `getSingleColorChannelImages` calls `getColorChannel` which calls `getRawColorChannel`. So a `dict` or similar seems to make sense. Maybe partial read is not compatible with `dict` and will have to make a file per color channel.
Benjamin_Loison commented 2024-05-13 16:42:56 +02:00
Author
Owner
Copy Link
Copy Source

Thought about using benchmark_raw_images_loading.py:

#!/usr/bin/env python

from tqdm import tqdm

IMAGES_CAMERAS_FOLDER = {
    'RAISE': 'flat-field/nef',
    'Rafael 23/04/24': 'rafael/230424',
}

for color in
getColorChannel(imageFilePath, color)

but in fact just using time and adding an exit to current algorithm was faster to use and almost as precise.

mmap_mode{None, ‘r+’, ‘r’, ‘w+’, ‘c’}, optional

If not None, then memory-map the file, using the given mode (see numpy.memmap for a detailed description of the modes). A memory-mapped array is kept on disk. However, it can be accessed and sliced like any ndarray. Memory mapping is especially useful for accessing small fragments of large files without reading the entire file into memory.

Source: https://numpy.org/doc/1.26/reference/generated/numpy.load.html

https://numpy.org/doc/1.26/reference/generated/numpy.memmap.html#numpy.memmap

https://numpy.org/doc/1.26/reference/generated/numpy.save.html

Someone asked this question as the Stack Overflow question 42727412.

Thought about using `benchmark_raw_images_loading.py`: ```python #!/usr/bin/env python from tqdm import tqdm IMAGES_CAMERAS_FOLDER = { 'RAISE': 'flat-field/nef', 'Rafael 23/04/24': 'rafael/230424', } for color in getColorChannel(imageFilePath, color) ``` but in fact just using `time` and adding an `exit` to current algorithm was faster to use and almost as precise. > mmap_mode{None, ‘r+’, ‘r’, ‘w+’, ‘c’}, optional > > If not None, then memory-map the file, using the given mode (see numpy.memmap for a detailed description of the modes). A memory-mapped array is kept on disk. However, it can be accessed and sliced like any ndarray. **Memory mapping is especially useful for accessing small fragments of large files without reading the entire file into memory.** Source: https://numpy.org/doc/1.26/reference/generated/numpy.load.html https://numpy.org/doc/1.26/reference/generated/numpy.memmap.html#numpy.memmap https://numpy.org/doc/1.26/reference/generated/numpy.save.html Someone asked this question as [the Stack Overflow question 42727412](https://stackoverflow.com/q/42727412).
Benjamin_Loison commented 2024-05-13 17:19:44 +02:00
Author
Owner
Copy Link
Copy Source

With all colors:

time ./benchmark_load_part_of_images.py

Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [01:58<00:00,  1.19s/it]
Camera:   0%|                                                                                                        | 0/2 [01:58<?, ?it/s]

real	1m58.753s
user	1m57.897s
sys	0m1.744s

With only one color:

time ./benchmark_load_part_of_images.py

Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:29<00:00,  3.41it/s]
Camera:   0%|                                                                                                        | 0/2 [00:29<?, ?it/s]

real	0m29.580s
user	0m29.782s
sys	0m0.729s

Should have at least a factor 4 improvement as only consider a single color.

With all colors: ```bash time ./benchmark_load_part_of_images.py ``` ``` Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [01:58<00:00, 1.19s/it] Camera: 0%| | 0/2 [01:58<?, ?it/s] real 1m58.753s user 1m57.897s sys 0m1.744s ``` With only one color: ```bash time ./benchmark_load_part_of_images.py ``` ``` Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:29<00:00, 3.41it/s] Camera: 0%| | 0/2 [00:29<?, ?it/s] real 0m29.580s user 0m29.782s sys 0m0.729s ``` Should have at least a factor 4 improvement as only consider a single color.
Benjamin_Loison commented 2024-05-13 17:56:55 +02:00
Author
Owner
Copy Link
Copy Source

Related to Benjamin_Loison/rawpy/issues/2.

Related to [Benjamin_Loison/rawpy/issues/2](https://codeberg.org/Benjamin_Loison/rawpy/issues/2).
Benjamin_Loison commented 2024-05-13 18:08:23 +02:00
Author
Owner
Copy Link
Copy Source

Loading all colors on both cameras:

time ./benchmark_load_part_of_images.py
Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [01:57<00:00,  1.17s/it]
Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 259/259 [02:32<00:00,  1.69it/s]
Camera: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [04:30<00:00, 135.14s/it]

real	4m30.510s
user	4m3.505s
sys	0m15.271s

For saving:

time ./benchmark_load_part_of_images.py
OPERATION = <Operation.SAVE: 2>
Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [01:59<00:00,  1.20s/it]
Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 259/259 [02:23<00:00,  1.81it/s]
Camera: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [04:23<00:00, 131.63s/it]

real	4m23.483s
user	4m8.537s
sys	0m15.810s

With .npy loading:

time ./benchmark_load_part_of_images.py
OPERATION = <Operation.LOAD_NPY: 3>
RESOLUTION = 100
Image: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 200/200 [00:00<00:00, 1441.65it/s]
Image: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 518/518 [00:00<00:00, 1000.87it/s]
Camera: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  3.04it/s]

real	0m0.886s
user	0m0.961s
sys	0m0.848s

Note that could consider a middle image (per image resolution) crop to reduce possible issues like vignetting.
Can also consider random location crop.

Loading all colors on both cameras: ``` time ./benchmark_load_part_of_images.py Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [01:57<00:00, 1.17s/it] Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 259/259 [02:32<00:00, 1.69it/s] Camera: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [04:30<00:00, 135.14s/it] real 4m30.510s user 4m3.505s sys 0m15.271s ``` For saving: ``` time ./benchmark_load_part_of_images.py OPERATION = <Operation.SAVE: 2> Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [01:59<00:00, 1.20s/it] Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 259/259 [02:23<00:00, 1.81it/s] Camera: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [04:23<00:00, 131.63s/it] real 4m23.483s user 4m8.537s sys 0m15.810s ``` With `.npy` loading: ``` time ./benchmark_load_part_of_images.py OPERATION = <Operation.LOAD_NPY: 3> RESOLUTION = 100 Image: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 200/200 [00:00<00:00, 1441.65it/s] Image: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 518/518 [00:00<00:00, 1000.87it/s] Camera: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 3.04it/s] real 0m0.886s user 0m0.961s sys 0m0.848s ``` Note that could consider a middle image (per image resolution) crop to reduce possible issues like [vignetting](https://en.wikipedia.org/wiki/Vignetting). Can also consider random location crop.
Benjamin_Loison commented 2024-05-13 18:21:55 +02:00
Author
Owner
Copy Link
Copy Source

Well I was only saving the last color.

time ./benchmark_load_part_of_images.py
OPERATION = <Operation.SAVE: 2>
Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [02:05<00:00,  1.26s/it]
Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 259/259 [02:49<00:00,  1.53it/s]
Camera: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [04:54<00:00, 147.33s/it]

real	4m54.894s
user	4m27.517s
sys	0m21.637s
Well I was only saving the last color. ``` time ./benchmark_load_part_of_images.py OPERATION = <Operation.SAVE: 2> Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [02:05<00:00, 1.26s/it] Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 259/259 [02:49<00:00, 1.53it/s] Camera: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [04:54<00:00, 147.33s/it] real 4m54.894s user 4m27.517s sys 0m21.637s ```
Benjamin_Loison commented 2024-05-13 18:43:11 +02:00
Author
Owner
Copy Link
Copy Source

Saving each color channel in a different file:

OPERATION = <Operation.SAVE: 2>
Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [02:30<00:00,  1.50s/it]
Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 259/259 [04:20<00:00,  1.01s/it]
Camera: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [06:50<00:00, 205.27s/it]

real	6m50.766s
user	6m26.275s
sys	0m24.466s
Saving each color channel in a different file: ``` OPERATION = <Operation.SAVE: 2> Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [02:30<00:00, 1.50s/it] Image: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 259/259 [04:20<00:00, 1.01s/it] Camera: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [06:50<00:00, 205.27s/it] real 6m50.766s user 6m26.275s sys 0m24.466s ```
Benjamin_Loison commented 2024-05-13 18:55:53 +02:00
Author
Owner
Copy Link
Copy Source

My tests seem biased due to a cache.

My tests seem biased due to a cache.
Benjamin_Loison commented 2024-05-15 19:15:21 +02:00
Author
Owner
Copy Link
Copy Source

For Rafael 23/04/24 estimated PRNU with mean denoiser:

mean_rafael_230424_mean_multiple_colors

image

400x400 (per color channel) centered crop:

mean_rafael_230424_mean_multiple_colors_crop

6024x4024 so 800x800 centered crop is from (2612, 1612) to (3412, 2412).

Could use same color scale to ease comparing both images.

For Rafael 23/04/24 estimated PRNU with mean denoiser: ![mean_rafael_230424_mean_multiple_colors](/attachments/e4291cc4-3e69-4300-989b-265581d02b1a) ![image](/attachments/24f581d2-d2c5-4946-83d6-3e9fc596d4ff) 400x400 (per color channel) centered crop: ![mean_rafael_230424_mean_multiple_colors_crop](/attachments/3e01d2dd-e838-4e64-a456-e4042524d4c1) 6024x4024 so 800x800 centered crop is from (2612, 1612) to (3412, 2412). Could use same color scale to ease comparing both images.
mean_rafael_230424_mean_multiple_colors_crop.png
1.5 MiB
image.png
1.4 MiB
mean_rafael_230424_mean_multiple_colors.png
55 MiB
Benjamin_Loison commented 2024-05-15 19:18:23 +02:00
Author
Owner
Copy Link
Copy Source

On the crop we seem to particularly see the lines artifacts, see #70.

On the crop we seem to particularly see the lines artifacts, see #70.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something is not working
 Context-Adaptive Interpolator
duplicate
This issue or pull request already exists
 enhancement
New feature
 epic
A task that is going to take more than a day to complete.
 help wanted
Need some help
 high priority
invalid
Something is wrong
 left for future work
low priority
medium
A task that should take less than a day to complete.
 medium priority
meta
question
More information is needed
 quick
A task that should take less than two hours to complete.
 wontfix
This won't be fixed
No labels enhancement epic medium priority
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
Benjamin_Loison (Benjamin Loison)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Benjamin_Loison/Robust_image_source_identification_on_modern_smartphones#62