Benjamin_Loison/YouTube_captions_search_engine
1
0
Fork 0
Code Issues 16 Pull Requests Packages Projects Releases 2 Wiki Activity
33 Commits 1 Branch 3 Tags
f6c11b54f31634d19317269562d1fb03dd361a2e
Commit Graph

20 Commits

Author SHA1 Message Date
Benjamin_Loison f6c11b54f3 Fix #26: Keep efficient search algorithm while keeping order (notably of the starting set) 2023-01-14 15:14:24 +01:00
Benjamin_Loison 27cd5c3a64 Fix #24: Stop using macros for user inputs to notably make releases 2023-01-08 18:26:20 +01:00
Benjamin_Loison eb805f5ced Fix #6: Add support for multiple keys to be resilient against exceeded quota errors 2023-01-08 17:59:08 +01:00
Benjamin_Loison d6f6b26361 Fix #23: YouTube Data API v3 PlaylistItems: list endpoint returns playlistNotFound error for regular uploads ones 2023-01-08 16:31:57 +01:00
Benjamin_Loison b3779fe49a Fix #20: YouTube Data API v3 returns rarely suddenly commentsDisabled error which involves an unwanted method switch 
Also modified compression command, as I got `sh: 1: zip: Argument list too long` when compressing the 248,868 json files of the French most subscribers channel.
 2023-01-08 15:43:27 +01:00
Benjamin_Loison 71e4bd95a9 Fix #9: Make sure that in case of error returned by the YouTube Data API v3 the algorithm treats it correctly 
Note that in case of error the algorithm used to skip the received content, as if just no `items` were in it.
 2023-01-06 20:55:32 +01:00
Benjamin_Loison baec8fcb6c #7: Remove remaining undefined behavior due to missing mutex use 2023-01-06 18:00:51 +01:00
Benjamin_Loison 773f86c551 Fix #17: Add to stdout live statistics of the number of comments treated per second 2023-01-06 17:55:16 +01:00
Benjamin_Loison 292dd8919e Add try/catch around json parser 
As got:
```
terminate called after throwing an instance of 'nlohmann::detail::parse_error'
terminate called recursively
  what():  [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal
terminate called recursively
```
 2023-01-06 00:31:05 +01:00
Benjamin_Loison 9d5c9fde2a #2: Add compression to channels/ folder 
Can use following Python script to compress existing uncompressed
`channels/` folder.

```py

import os, shutil

path = 'channels/'

os.chdir(path)

d = next(os.walk('.'))[1]
for channelIndex, channelId in enumerate(d):
    print(f'{channelIndex} / {len(d)}: {channelId}')
    shutil.make_archive(channelId, 'zip', channelId)
    shutil.rmtree(channelId)
```
 2023-01-04 03:06:33 +01:00
Benjamin_Loison e4b4ce21a2 Fix #7: Add multi-threading 2023-01-03 04:56:19 +01:00
Benjamin_Loison a2990c7699 Fix #8: Support comments disabled channels 
Tested with `UCWIdqSQekeGmUWlSFeCiEnA` which treated correctly the 36 comments of the only comments enabled video `3F8dFt8LsXY`.

Note that this commit doesn't support comments disabled channels with more than 20,000 videos.
 2023-01-03 02:56:07 +01:00
Benjamin_Loison 923c14a77b #2: Add data logging 2023-01-02 19:46:32 +01:00
Benjamin_Loison 73a9dea023 Apply astyle formatting to main.cpp 2023-01-02 18:31:16 +01:00
Benjamin_Loison 938ae4b0fb Fix #4: Provide a version relying on the no-key service of https://yt.lemnoslife.com 2023-01-02 18:30:18 +01:00
Benjamin_Loison c50a82df1b Make compatible with Debian 
More precise ly make compatible with `gcc version 10.2.1 20210110 (Debian 10.2.1-6)`
 2023-01-02 18:23:30 +01:00
Benjamin_Loison 36f1fb9e83 Add progression save and use spaces instead of tabs 2022-12-22 06:18:22 +01:00
Benjamin_Loison 934954092a Add time to logging 2022-12-22 05:47:16 +01:00
Benjamin_Loison eaae954e1b Add resilience to missing authorChannelId in main.cpp 2022-12-22 05:41:38 +01:00
Benjamin_Loison 2ffc1d0e5d Add main.cpp, Makefile and channelsToTreat.txt 
Note that running this algorithm end up with channel [`UC-99odscxh1xxTyxHyXuRrg`](https://www.youtube.com/channel/UC-99odscxh1xxTyxHyXuRrg) and more precisely the video [`Tq5aPNzfYcg`](https://www.youtube.com/watch?v=Tq5aPNzfYcg) and more precisely the comment [`Ugx-TlSq6SNCbOX04mx4AaABAg`](https://www.youtube.com/watch?v=Tq5aPNzfYcg&lc=Ugx-TlSq6SNCbOX04mx4AaABAg) [which doesn't have any author](https://yt.lemnoslife.com/noKey/comments?part=snippet&id=Ugx-TlSq6SNCbOX04mx4AaABAg)...
 2022-12-22 05:20:32 +01:00