Benjamin_Loison/YouTube_captions_search_engine
1
0
Fork 0
Code Issues 16 Pull Requests Packages Projects Releases 2 Wiki Activity
59 Commits 1 Branch 3 Tags
4fa433495b706a42439606d099cbec44e55cbc62
Commit Graph

59 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Benjamin_Loison 4fa433495b Restore ability to download whole archives 
As API keys aren't written in the first line of JSON files.
 2023-02-07 23:01:26 +01:00
Benjamin_Loison a0ba474fcc Remove ability in channels.php to download whole archive for not leaking API keys used 2023-02-07 22:42:24 +01:00
Benjamin_Loison b4cb072770 Add channels.php adding support for (file in) zip download 2023-02-07 22:39:43 +01:00
Benjamin_Loison fda8fc728e #31: Add zip files search 2023-02-07 20:15:36 +01:00
Benjamin_Loison 82e597f205 Comment WebSocket mechanism to work with an arbitrary number of independent send 2023-02-07 18:14:49 +01:00
Benjamin_Loison 03c2566a20 Make WebSocket able to manage arbitrary feedback to end-user 
While previous implementation was able to send two independent messages, now we can send an arbitrary amount of independent messages.
 2023-02-07 17:25:17 +01:00
Benjamin_Loison a116b29df9 Make websockets.php able to proceed blocking treatments 2023-02-07 01:22:26 +01:00
Benjamin_Loison 1fe92ec2d0 Make a WebSocket example work with crawler.yt.lemnoslife.com 2023-01-31 01:05:09 +01:00
Benjamin_Loison 411a3db465 Run php-cs-fixer fix --rules=@PSR12 websocket.php 2023-01-31 00:57:06 +01:00
Benjamin_Loison 08b465753d Rename chat.php to websocket.php 2023-01-30 22:24:02 +01:00
Benjamin_Loison 45c5d8a940 Copy-pasted the README.md quick example of ratchetphp/Ratchet 
https://github.com/ratchetphp/Ratchet/tree/5012dc954541b40c5599d286fd40653f5716a38f#a-quick-example
 2023-01-30 22:19:04 +01:00
Benjamin_Loison 668aa608ed Add static website/index.php 2023-01-30 22:14:05 +01:00
Benjamin_Loison c746d43ddf Correct typo: the channel tab is LIVE, not LIVES 2023-01-25 01:00:29 +01:00
Benjamin_Loison 05cd243abd Add comment in README.md about the usage of --no-keys or generating a YouTube Data API v3 key 2023-01-22 15:41:13 +01:00
Benjamin_Loison 9d40fef429 Introduce {,MAIN_}EXIT_WITH_ERROR macros for exitting with an error 2023-01-22 15:17:14 +01:00
Benjamin_Loison d34fade0cd #11: Add the discovering of channels having commented on ended livestreams 2023-01-22 15:15:27 +01:00
Benjamin_Loison 68b1f9a77f #11: Add current livestreams support to discover channels 2023-01-22 04:00:11 +01:00
Benjamin_Loison c17a33d181 Instead of looping on items where we expect only one to be, we just use items[0] 2023-01-22 02:19:26 +01:00
Benjamin_Loison 59dc5676cc Make PRINT not requiring to precise threadId 2023-01-22 02:04:03 +01:00
Benjamin_Loison 548a797ee8 #11: Treat COMMUNITY post comments to discover channels 2023-01-22 01:37:32 +01:00
Benjamin_Loison 46ef8146f8 Add in README.md the fact that as documented in #30, this algorithm is only known to be working fin on Linux 2023-01-21 22:20:45 +01:00
Benjamin_Loison 4133faad41 #11: Update channel CHANNELS tab treatment following YouTube-operational-API/issues/121 closure 2023-01-21 02:24:42 +01:00
Benjamin_Loison fced9e0a3a #11: Add the treatment of channels' tab, but only postpone unlisted videos treatment 2023-01-15 14:56:44 +01:00
Benjamin_Loison f114aac0cf #7: Make commentsCount and requestsPerChannel compatible with multithreading 2023-01-15 14:31:55 +01:00
Benjamin_Loison 7456685f2b #11: Add a first iteration for the CHANNELS retrieval 2023-01-15 02:19:31 +01:00
Benjamin_Loison 270c48da02 #11: Add --youtube-operational-api-instance-url parameter and use exit(EXIT_{SUCCESS, FAILURE}) instead of exit({0, 1}) 2023-01-15 00:49:32 +01:00
Benjamin_Loison f6c11b54f3 Fix #26: Keep efficient search algorithm while keeping order (notably of the starting set) 2023-01-14 15:14:24 +01:00
Benjamin_Loison 27cd5c3a64 Fix #24: Stop using macros for user inputs to notably make releases 2023-01-08 18:26:20 +01:00
Benjamin_Loison eb805f5ced Fix #6: Add support for multiple keys to be resilient against exceeded quota errors 2023-01-08 17:59:08 +01:00
Benjamin_Loison d6f6b26361 Fix #23: YouTube Data API v3 PlaylistItems: list endpoint returns playlistNotFound error for regular uploads ones 2023-01-08 16:31:57 +01:00
Benjamin_Loison b3779fe49a Fix #20: YouTube Data API v3 returns rarely suddenly commentsDisabled error which involves an unwanted method switch 
Also modified compression command, as I got `sh: 1: zip: Argument list too long` when compressing the 248,868 json files of the French most subscribers channel.
 2023-01-08 15:43:27 +01:00
Benjamin_Loison 3ae0f4e924 Make all Python scripts executable and add findAlreadyTreatedCommentsCount.py to find how many comments were already treated 2023-01-07 15:45:31 +01:00
Benjamin_Loison 3758405f52 Add a note about the timing percentage of findLatestTreatedCommentsForChannelsBeingTreated.py going backward 2023-01-07 15:35:12 +01:00
Benjamin_Loison 71e4bd95a9 Fix #9: Make sure that in case of error returned by the YouTube Data API v3 the algorithm treats it correctly 
Note that in case of error the algorithm used to skip the received content, as if just no `items` were in it.
 2023-01-06 20:55:32 +01:00
Benjamin_Loison 34bbc216f6 Fix #15: Provide an algorithm to retrieve the list of 100 French channels with most subscribers (and provide it too) 2023-01-06 18:06:00 +01:00
Benjamin_Loison baec8fcb6c #7: Remove remaining undefined behavior due to missing mutex use 2023-01-06 18:00:51 +01:00
Benjamin_Loison 773f86c551 Fix #17: Add to stdout live statistics of the number of comments treated per second 2023-01-06 17:55:16 +01:00
Benjamin_Loison f436007836 Fix #16: Provide an algorithm to determine the progress of retrieving comments for huge YouTube channels 2023-01-06 17:51:00 +01:00
Benjamin_Loison dfbf38b071 #1: Add GNU AGPLv3 license 2023-01-06 16:09:12 +01:00
Benjamin_Loison 292dd8919e Add try/catch around json parser 
As got:
```
terminate called after throwing an instance of 'nlohmann::detail::parse_error'
terminate called recursively
  what():  [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal
terminate called recursively
```
 2023-01-06 00:31:05 +01:00
Benjamin_Loison dab4c8ff69 Modify removeChannelsBeingTreated.py to be more resilient against not existing files in the treatment process 2023-01-04 03:10:28 +01:00
Benjamin_Loison 9d5c9fde2a #2: Add compression to channels/ folder 
Can use following Python script to compress existing uncompressed
`channels/` folder.

```py

import os, shutil

path = 'channels/'

os.chdir(path)

d = next(os.walk('.'))[1]
for channelIndex, channelId in enumerate(d):
    print(f'{channelIndex} / {len(d)}: {channelId}')
    shutil.make_archive(channelId, 'zip', channelId)
    shutil.rmtree(channelId)
```
 2023-01-04 03:06:33 +01:00
Benjamin_Loison f201ae7a91 Make #7: Add multi-threading compatible with my Debian setup 2023-01-04 02:51:40 +01:00
Benjamin_Loison 4cae7e09d1 Add {removeChannelsBeingTreated, findTreatedChannelWithMost{Comments, Subscribers}}.py 2023-01-04 02:41:07 +01:00
Benjamin_Loison e4b4ce21a2 Fix #7: Add multi-threading 2023-01-03 04:56:19 +01:00
Benjamin_Loison a2990c7699 Fix #8: Support comments disabled channels 
Tested with `UCWIdqSQekeGmUWlSFeCiEnA` which treated correctly the 36 comments of the only comments enabled video `3F8dFt8LsXY`.

Note that this commit doesn't support comments disabled channels with more than 20,000 videos.
 2023-01-03 02:56:07 +01:00
Benjamin_Loison 923c14a77b #2: Add data logging 2023-01-02 19:46:32 +01:00
Benjamin_Loison 73a9dea023 Apply astyle formatting to main.cpp 2023-01-02 18:31:16 +01:00
Benjamin_Loison 938ae4b0fb Fix #4: Provide a version relying on the no-key service of https://yt.lemnoslife.com 2023-01-02 18:30:18 +01:00
Benjamin_Loison c50a82df1b Make compatible with Debian 
More precise ly make compatible with `gcc version 10.2.1 20210110 (Debian 10.2.1-6)`
 2023-01-02 18:23:30 +01:00