Benjamin_Loison/YouTube_captions_search_engine
1
0
Fork 0
Code Issues 16 Pull Requests Packages Projects Releases 2 Wiki Activity
57 Commits 1 Branch 3 Tags
2179e9b6f4fc1042b66187047364faf33f9a6b66
Commit Graph

57 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Benjamin_Loison 2179e9b6f4 Add channels.php adding support for (file in) zip download 2023-02-07 22:39:43 +01:00
Benjamin_Loison e9b77369fb #31: Add zip files search 2023-02-07 20:15:36 +01:00
Benjamin_Loison b45384bab7 Comment WebSocket mechanism to work with an arbitrary number of independent send 2023-02-07 18:14:49 +01:00
Benjamin_Loison 126cc75dc6 Make WebSocket able to manage arbitrary feedback to end-user 
While previous implementation was able to send two independent messages, now we can send an arbitrary amount of independent messages.
 2023-02-07 17:25:17 +01:00
Benjamin_Loison 7302679a81 Make websockets.php able to proceed blocking treatments 2023-02-07 01:22:26 +01:00
Benjamin_Loison 0dba8e0c7d Make a WebSocket example work with crawler.yt.lemnoslife.com 2023-01-31 01:05:09 +01:00
Benjamin_Loison 155d372186 Run php-cs-fixer fix --rules=@PSR12 websocket.php 2023-01-31 00:57:06 +01:00
Benjamin_Loison bd184bd0f0 Rename chat.php to websocket.php 2023-01-30 22:24:02 +01:00
Benjamin_Loison 0193f05143 Copy-pasted the README.md quick example of ratchetphp/Ratchet 
https://github.com/ratchetphp/Ratchet/tree/5012dc954541b40c5599d286fd40653f5716a38f#a-quick-example
 2023-01-30 22:19:04 +01:00
Benjamin_Loison 931b2df563 Add static website/index.php 2023-01-30 22:14:05 +01:00
Benjamin_Loison 0f4b89ccd9 Correct typo: the channel tab is LIVE, not LIVES 2023-01-25 01:00:29 +01:00
Benjamin_Loison 4e162e34c3 Add comment in README.md about the usage of --no-keys or generating a YouTube Data API v3 key 2023-01-22 15:41:13 +01:00
Benjamin_Loison 10e8811817 Introduce {,MAIN_}EXIT_WITH_ERROR macros for exitting with an error 2023-01-22 15:17:14 +01:00
Benjamin_Loison 0f15bb0235 #11: Add the discovering of channels having commented on ended livestreams 2023-01-22 15:15:27 +01:00
Benjamin_Loison bdb4e6443a #11: Add current livestreams support to discover channels 2023-01-22 04:00:11 +01:00
Benjamin_Loison d2391e5d54 Instead of looping on items where we expect only one to be, we just use items[0] 2023-01-22 02:19:26 +01:00
Benjamin_Loison 993d0b9771 Make PRINT not requiring to precise threadId 2023-01-22 02:04:03 +01:00
Benjamin_Loison 0fcb5a0426 #11: Treat COMMUNITY post comments to discover channels 2023-01-22 01:37:32 +01:00
Benjamin_Loison 57200da482 Add in README.md the fact that as documented in #30, this algorithm is only known to be working fin on Linux 2023-01-21 22:20:45 +01:00
Benjamin_Loison a0880c79bb #11: Update channel CHANNELS tab treatment following YouTube-operational-API/issues/121 closure 2023-01-21 02:24:42 +01:00
Benjamin_Loison 10c5c1d605 #11: Add the treatment of channels' tab, but only postpone unlisted videos treatment 2023-01-15 14:56:44 +01:00
Benjamin_Loison 51a70f6e54 #7: Make commentsCount and requestsPerChannel compatible with multithreading 2023-01-15 14:31:55 +01:00
Benjamin_Loison aa97c94bf8 #11: Add a first iteration for the CHANNELS retrieval 2023-01-15 02:19:31 +01:00
Benjamin_Loison d1b84335d1 #11: Add --youtube-operational-api-instance-url parameter and use exit(EXIT_{SUCCESS, FAILURE}) instead of exit({0, 1}) 2023-01-15 00:49:32 +01:00
Benjamin_Loison 6ce29051c0 Fix #26: Keep efficient search algorithm while keeping order (notably of the starting set) 2023-01-14 15:14:24 +01:00
Benjamin_Loison ad9f96b33c Fix #24: Stop using macros for user inputs to notably make releases 0.0.0 2023-01-08 18:26:20 +01:00
Benjamin_Loison d498c86058 Fix #6: Add support for multiple keys to be resilient against exceeded quota errors 2023-01-08 17:59:08 +01:00
Benjamin_Loison 1ee767abbc Fix #23: YouTube Data API v3 PlaylistItems: list endpoint returns playlistNotFound error for regular uploads ones 2023-01-08 16:31:57 +01:00
Benjamin_Loison 7e35a6473a Fix #20: YouTube Data API v3 returns rarely suddenly commentsDisabled error which involves an unwanted method switch 
Also modified compression command, as I got `sh: 1: zip: Argument list too long` when compressing the 248,868 json files of the French most subscribers channel.
 2023-01-08 15:43:27 +01:00
Benjamin_Loison ba37d6a111 Make all Python scripts executable and add findAlreadyTreatedCommentsCount.py to find how many comments were already treated 2023-01-07 15:45:31 +01:00
Benjamin_Loison 5a7e5b6f78 Add a note about the timing percentage of findLatestTreatedCommentsForChannelsBeingTreated.py going backward 2023-01-07 15:35:12 +01:00
Benjamin_Loison e3cab4c204 Fix #9: Make sure that in case of error returned by the YouTube Data API v3 the algorithm treats it correctly 
Note that in case of error the algorithm used to skip the received content, as if just no `items` were in it.
 2023-01-06 20:55:32 +01:00
Benjamin_Loison 156a621413 Fix #15: Provide an algorithm to retrieve the list of 100 French channels with most subscribers (and provide it too) 2023-01-06 18:06:00 +01:00
Benjamin_Loison fdfec17817 #7: Remove remaining undefined behavior due to missing mutex use 2023-01-06 18:00:51 +01:00
Benjamin_Loison 3ef5fa0707 Fix #17: Add to stdout live statistics of the number of comments treated per second 2023-01-06 17:55:16 +01:00
Benjamin_Loison 0259dfb3fb Fix #16: Provide an algorithm to determine the progress of retrieving comments for huge YouTube channels 2023-01-06 17:51:00 +01:00
Benjamin_Loison b2fafb721c #1: Add GNU AGPLv3 license 2023-01-06 16:09:12 +01:00
Benjamin_Loison 01394769fd Add try/catch around json parser 
As got:
```
terminate called after throwing an instance of 'nlohmann::detail::parse_error'
terminate called recursively
  what():  [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal
terminate called recursively
```
 2023-01-06 00:31:05 +01:00
Benjamin_Loison 5d13bd3c44 Modify removeChannelsBeingTreated.py to be more resilient against not existing files in the treatment process 2023-01-04 03:10:28 +01:00
Benjamin_Loison 512485b1b8 #2: Add compression to channels/ folder 
Can use following Python script to compress existing uncompressed
`channels/` folder.

```py

import os, shutil

path = 'channels/'

os.chdir(path)

d = next(os.walk('.'))[1]
for channelIndex, channelId in enumerate(d):
    print(f'{channelIndex} / {len(d)}: {channelId}')
    shutil.make_archive(channelId, 'zip', channelId)
    shutil.rmtree(channelId)
```
 2023-01-04 03:06:33 +01:00
Benjamin_Loison a4a282642d Make #7: Add multi-threading compatible with my Debian setup 2023-01-04 02:51:40 +01:00
Benjamin_Loison dde52da8c8 Add {removeChannelsBeingTreated, findTreatedChannelWithMost{Comments, Subscribers}}.py 2023-01-04 02:41:07 +01:00
Benjamin_Loison 2a33be9272 Fix #7: Add multi-threading 2023-01-03 04:56:19 +01:00
Benjamin_Loison ad3e90fe92 Fix #8: Support comments disabled channels 
Tested with `UCWIdqSQekeGmUWlSFeCiEnA` which treated correctly the 36 comments of the only comments enabled video `3F8dFt8LsXY`.

Note that this commit doesn't support comments disabled channels with more than 20,000 videos.
 2023-01-03 02:56:07 +01:00
Benjamin_Loison b12fa15288 #2: Add data logging 2023-01-02 19:46:32 +01:00
Benjamin_Loison 0675314fe6 Apply astyle formatting to main.cpp 2023-01-02 18:31:16 +01:00
Benjamin_Loison dfd9ee9c41 Fix #4: Provide a version relying on the no-key service of https://yt.lemnoslife.com 2023-01-02 18:30:18 +01:00
Benjamin_Loison 68800a25a0 Make compatible with Debian 
More precise ly make compatible with `gcc version 10.2.1 20210110 (Debian 10.2.1-6)`
 2023-01-02 18:23:30 +01:00
Benjamin_Loison 7a1eac5e40 Add progression save and use spaces instead of tabs 2022-12-22 06:18:22 +01:00
Benjamin_Loison 273537bc8d Add time to logging 2022-12-22 05:47:16 +01:00