Benjamin_Loison/YouTube_captions_search_engine
1
0
Fork 0
Code Issues 16 Pull Requests Packages Projects Releases 2 Wiki Activity
108 Commits 1 Branch 3 Tags
master
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Benjamin_Loison 02ee625cc7 Replace re.match with re.fullmatch 
See [Benjamin-Loison/cpython/issues/43](https://github.com/Benjamin-Loison/cpython/issues/43).
 2024-10-16 18:11:21 +02:00
Benjamin_Loison 7ee5667628 Avoid shell command injection in main.cpp 2023-07-30 13:14:00 +02:00
Benjamin_Loison 5185bcb6a7 Replace "\n" with n in Python scripts 2023-07-28 14:52:41 +02:00
Benjamin_Loison b60bb22112 Simplify scripts/findTreatedChannelWithMostSubscribers.py 2023-07-28 11:40:07 +02:00
Benjamin_Loison 654769b80b Simplify scripts/findLatestTreatedCommentsForChannelsBeingTreated.py 2023-07-28 11:32:52 +02:00
Benjamin_Loison 3ca58b2466 Add presentation video link to README.md 2023-02-26 17:10:00 +01:00
Benjamin_Loison 20025e02b4 Use compiled regex in the for loops for the website 2023-02-26 16:01:56 +01:00
Benjamin_Loison e493eaeb49 Add an optimization to the website when providing a channel id as a file path filter 2023-02-26 15:56:16 +01:00
Benjamin_Loison e1aff6f469 #35: Move Python scripts to scripts/ and describe the project structure in README.md 2023-02-26 15:12:06 +01:00
Benjamin_Loison ff5542d8b0 Add sudo apt install nginx to README.md for hosting the website 2023-02-25 15:55:24 +01:00
Benjamin_Loison 65bc8853e6 Make the website support regex for both search and path filtering 2023-02-24 15:38:51 +01:00
Benjamin_Loison d3c87d3b6f Use with open(filePath) as f: instead of f = open(filePath) in search.py 2023-02-24 15:15:42 +01:00
Benjamin_Loison 58f25a114e #44: Enable end-users to filter path for searches 2023-02-24 15:12:07 +01:00
Benjamin_Loison 3bba97e90c Make search.py search across displayed captions. 
Otherwise `Linux, is in millions of computers` doesn't match the not automatically generated caption of [`o8NPllzkFhE`](https://www.youtube.com/watch?v=o8NPllzkFhE). Note to be confused with the search across captions that already used to work with for instance `is in millions of computers, it`.
 2023-02-24 14:46:00 +01:00
Benjamin_Loison 9d433ba2f3 Remove unused setFromVector function 2023-02-23 23:50:07 +01:00
Benjamin_Loison 572f9e121f Precise in README.md in which folder each command has to be ran 2023-02-23 23:48:40 +01:00
Benjamin_Loison 02c3d74eb6 Add support for channelsToTreat to be empty 
It's the case when providing a single channel in `channels.txt` for
instance.
 2023-02-23 23:45:36 +01:00
Benjamin_Loison 95e96d08e1 Advertize pip instead of apt in README.md to install the latest version of yt-dlp 2023-02-23 23:16:36 +01:00
Benjamin_Loison df3af2780b #19: Detail how to run the website and reference channels.txt on it 2023-02-23 23:12:18 +01:00
Benjamin_Loison 68cd27c263 Fix #19: Improve documentation and code comments 2023-02-23 22:50:30 +01:00
Benjamin_Loison f44ee4b3c1 #44: Allow arbitrary end-user requests 2023-02-22 17:48:24 +01:00
Benjamin_Loison 7626c7bad1 Fix #51: These last days the algorithm seems to not treat completely firstly the starting set of channels before treating discovered channels 
I verified that this commit solves the issue by treating only `CHANNELS` tab of the channels in `channels.txt`.
 2023-02-22 04:09:35 +01:00
Benjamin_Loison 76e41f2f7b #48: Stop relying on echo, tee and /dev/null for redirecting compression command to debug/ 2023-02-22 03:47:06 +01:00
Benjamin_Loison 8d4f31d106 #48: Redirect compression command echo to /dev/null 2023-02-22 03:37:07 +01:00
Benjamin_Loison c0db4eb437 Fix #48: Redirect compression execution logs for not having them overlapping PRINTs 2023-02-22 03:27:49 +01:00
Benjamin_Loison e23a0fc4c7 #48: Modify removeChannelsBeingTreated.py to temporarily solve the issue 2023-02-19 02:04:28 +01:00
Benjamin_Loison e5dd476490 #35: Make the not automatically generated captions correctly downloaded 2023-02-17 16:57:11 +01:00
Benjamin_Loison fd75d27b99 Change the EXIT_WITH_ERROR to PRINT for channels not having an enumerable uploads playlist 2023-02-16 12:21:28 +01:00
Benjamin_Loison cb7b68342a Make the first channel of channels.txt being treated again, solve temporary empty response from YouTube Data API v3 issue and temporarily remove sanity check failing very rarely #39 2023-02-14 23:15:07 +01:00
Benjamin_Loison e166fdb4e5 Fix #31: List all occurrences of search within video captions 2023-02-14 02:56:11 +01:00
Benjamin_Loison 8d34cf33ae Fix #31: Make a website with a search engine notably based on the captions extracted 2023-02-14 02:00:23 +01:00
Benjamin_Loison 09f7675bf7 #31: Make search within captions not limited by line wrapping 2023-02-14 01:32:36 +01:00
Benjamin_Loison 4449d488c9 Fix #38: Add a loading message with progress on end-user interface 2023-02-14 01:08:05 +01:00
Benjamin_Loison 34c0d03587 #31: Add a first search only captions support 2023-02-14 00:59:37 +01:00
Benjamin_Loison 8d20327b67 Add .gitignore to ignore {keys, channels}.txt 2023-02-13 06:18:42 +01:00
Benjamin_Loison 09c2a8eafe Make the COMMUNITY tab process not infinitely loop 
Related to https://github.com/Benjamin-Loison/YouTube-operational-API/issues/49
 2023-02-13 06:17:23 +01:00
Benjamin_Loison 9cd7c57e2f Add link to channels/ to index.php 2023-02-13 05:55:44 +01:00
Benjamin_Loison 94ad823e3b Modify website to support new sub-folders architecture 2023-02-13 05:45:08 +01:00
Benjamin_Loison 454503271e Fix #37: Use a number of channels seen (possibly repeated) instead of YouTube Data API v3 Comment(Thread): resource 2023-02-12 16:31:27 +01:00
Benjamin_Loison 54fe40e588 Add logging to exec and make it crashless, requests and captions folders support for compressing, clean captions support for videos being livestreams and videos starting with - 2023-02-12 16:24:16 +01:00
Benjamin_Loison 8cf5698051 Move YouTube API requests logging to requests/ channel sub-folder 2023-02-10 20:17:49 +01:00
Benjamin_Loison 04c59eb025 Fix #13: Add captions extraction 
I was about to commit in addition:

```c++
// Due to videos with automatically generated captions but being set to `Off` by default aren't retrieved with `--sub-langs '.*orig'`.
// My workaround is to first call YouTube Data API v3 Captions: list endpoint with `part=snippet` and retrieve the language that has `"trackKind": "asr"` (automatic speech recognition) in `snippet`.
/*json data = getJson(threadId, "captions?part=snippet&videoId=" + videoId, true, channelToTreat),
     items = data["items"];
for(const auto& item : items)
{
    json snippet = item["snippet"];
    if(snippet["trackKind"] == "asr")
    {
        string language = snippet["language"];
        cmd = cmdCommonPrefix + "--write-auto-subs --sub-langs '" + language + "-orig' --sub-format ttml --convert-subs vtt" + cmdCommonPostfix;
        exec(threadId, cmd);
        // As there should be a single automatic speech recognized track, there is no need to go through all tracks.
        break;
    }
}*/
```

Instead of:

```c++
cmd = cmdCommonPrefix + "--write-auto-subs --sub-langs '.*orig' --sub-format ttml --convert-subs vtt" + cmdCommonPostfix;
exec(threadId, cmd);
```

But I realized that, as the GitHub comment I was about to add to https://github.com/yt-dlp/yt-dlp/issues/2655, I was
wrong:

> `yt-dlp --cookies cookies.txt --sub-langs 'en.*,.*orig' --write-auto-subs https://www.youtube.com/watch?v=tQqDBySHYlc` work as expected. Many thanks again.
>
> ```
> 'subtitleslangs': ['en.*','.*orig'],
> 'writeautomaticsub': True,
> ```
>
> Work as expected too. Thank you
>
> Very sorry for the video sample. I even not watched it.

Thank you for this workaround. However note that videos having automatically generated subtitles but being set to `Off` by default aren't retrieved with your method (example of such video: [`mozyXsZJnQ4`](https://www.youtube.com/watch?v=mozyXsZJnQ4)). My workaround is to first call [YouTube Data API v3](https://developers.google.com/youtube/v3) [Captions: list](https://developers.google.com/youtube/v3/docs/captions/list) endpoint with [`part=snippet`](https://developers.google.com/youtube/v3/docs/captions/list#part) and retrieve the [`language`](https://developers.google.com/youtube/v3/docs/captions#snippet.language) that has [`"trackKind": "asr"`](https://developers.google.com/youtube/v3/docs/captions#snippet.trackKind) (automatic speech recognition) in [`snippet`](https://developers.google.com/youtube/v3/docs/captions#snippet).
 2023-02-10 20:03:08 +01:00
Benjamin_Loison 4508b12b6c Correct the termination of COMMUNITY tab process due to missing page tokens 2023-02-10 00:37:28 +01:00
Benjamin_Loison ea604cce40 Remove the Content-Type: application/json HTTP header when retrieving urls.txt inside a .zip 2023-02-09 02:07:10 +01:00
Benjamin_Loison 01aac3f66e Add a verification that snippet/authorChannelId/value isn't null when using commentThreads for COMMUNITY 
As it can happen cf https://www.youtube.com/channel/UCWeg2Pkate69NFdBeuRFTAw/community?lc=UgwGfjNxGuwqP8qYPPN4AaABAg&lb=UgkxYiEAo9-b1vWPasxFy13f959rrctQpZwW
 2023-02-09 01:51:22 +01:00
Benjamin_Loison d0dee043c6 Append to channels.txt all channels mentioned in the Wiki 2023-02-08 16:28:44 +01:00
Benjamin_Loison d5c55e756a Add in urls.txt if the URL is related to YouTube Data API v3 or YouTube operational API 2023-02-08 16:05:03 +01:00
Benjamin_Loison 50306aff5d Fix #34: Correct JSON files by putting first line in another metadata file 2023-02-07 23:08:09 +01:00
Benjamin_Loison 4fa433495b Restore ability to download whole archives 
As API keys aren't written in the first line of JSON files.
 2023-02-07 23:01:26 +01:00
Benjamin_Loison a0ba474fcc Remove ability in channels.php to download whole archive for not leaking API keys used 2023-02-07 22:42:24 +01:00
Benjamin_Loison b4cb072770 Add channels.php adding support for (file in) zip download 2023-02-07 22:39:43 +01:00
Benjamin_Loison fda8fc728e #31: Add zip files search 2023-02-07 20:15:36 +01:00
Benjamin_Loison 82e597f205 Comment WebSocket mechanism to work with an arbitrary number of independent send 2023-02-07 18:14:49 +01:00
Benjamin_Loison 03c2566a20 Make WebSocket able to manage arbitrary feedback to end-user 
While previous implementation was able to send two independent messages, now we can send an arbitrary amount of independent messages.
 2023-02-07 17:25:17 +01:00
Benjamin_Loison a116b29df9 Make websockets.php able to proceed blocking treatments 2023-02-07 01:22:26 +01:00
Benjamin_Loison 1fe92ec2d0 Make a WebSocket example work with crawler.yt.lemnoslife.com 2023-01-31 01:05:09 +01:00
Benjamin_Loison 411a3db465 Run php-cs-fixer fix --rules=@PSR12 websocket.php 2023-01-31 00:57:06 +01:00
Benjamin_Loison 08b465753d Rename chat.php to websocket.php 2023-01-30 22:24:02 +01:00
Benjamin_Loison 45c5d8a940 Copy-pasted the README.md quick example of ratchetphp/Ratchet 
https://github.com/ratchetphp/Ratchet/tree/5012dc954541b40c5599d286fd40653f5716a38f#a-quick-example
 2023-01-30 22:19:04 +01:00
Benjamin_Loison 668aa608ed Add static website/index.php 2023-01-30 22:14:05 +01:00
Benjamin_Loison c746d43ddf Correct typo: the channel tab is LIVE, not LIVES 2023-01-25 01:00:29 +01:00
Benjamin_Loison 05cd243abd Add comment in README.md about the usage of --no-keys or generating a YouTube Data API v3 key 2023-01-22 15:41:13 +01:00
Benjamin_Loison 9d40fef429 Introduce {,MAIN_}EXIT_WITH_ERROR macros for exitting with an error 2023-01-22 15:17:14 +01:00
Benjamin_Loison d34fade0cd #11: Add the discovering of channels having commented on ended livestreams 2023-01-22 15:15:27 +01:00
Benjamin_Loison 68b1f9a77f #11: Add current livestreams support to discover channels 2023-01-22 04:00:11 +01:00
Benjamin_Loison c17a33d181 Instead of looping on items where we expect only one to be, we just use items[0] 2023-01-22 02:19:26 +01:00
Benjamin_Loison 59dc5676cc Make PRINT not requiring to precise threadId 2023-01-22 02:04:03 +01:00
Benjamin_Loison 548a797ee8 #11: Treat COMMUNITY post comments to discover channels 2023-01-22 01:37:32 +01:00
Benjamin_Loison 46ef8146f8 Add in README.md the fact that as documented in #30, this algorithm is only known to be working fin on Linux 2023-01-21 22:20:45 +01:00
Benjamin_Loison 4133faad41 #11: Update channel CHANNELS tab treatment following YouTube-operational-API/issues/121 closure 2023-01-21 02:24:42 +01:00
Benjamin_Loison fced9e0a3a #11: Add the treatment of channels' tab, but only postpone unlisted videos treatment 2023-01-15 14:56:44 +01:00
Benjamin_Loison f114aac0cf #7: Make commentsCount and requestsPerChannel compatible with multithreading 2023-01-15 14:31:55 +01:00
Benjamin_Loison 7456685f2b #11: Add a first iteration for the CHANNELS retrieval 2023-01-15 02:19:31 +01:00
Benjamin_Loison 270c48da02 #11: Add --youtube-operational-api-instance-url parameter and use exit(EXIT_{SUCCESS, FAILURE}) instead of exit({0, 1}) 2023-01-15 00:49:32 +01:00
Benjamin_Loison f6c11b54f3 Fix #26: Keep efficient search algorithm while keeping order (notably of the starting set) 2023-01-14 15:14:24 +01:00
Benjamin_Loison 27cd5c3a64 Fix #24: Stop using macros for user inputs to notably make releases 2023-01-08 18:26:20 +01:00
Benjamin_Loison eb805f5ced Fix #6: Add support for multiple keys to be resilient against exceeded quota errors 2023-01-08 17:59:08 +01:00
Benjamin_Loison d6f6b26361 Fix #23: YouTube Data API v3 PlaylistItems: list endpoint returns playlistNotFound error for regular uploads ones 2023-01-08 16:31:57 +01:00
Benjamin_Loison b3779fe49a Fix #20: YouTube Data API v3 returns rarely suddenly commentsDisabled error which involves an unwanted method switch 
Also modified compression command, as I got `sh: 1: zip: Argument list too long` when compressing the 248,868 json files of the French most subscribers channel.
 2023-01-08 15:43:27 +01:00
Benjamin_Loison 3ae0f4e924 Make all Python scripts executable and add findAlreadyTreatedCommentsCount.py to find how many comments were already treated 2023-01-07 15:45:31 +01:00
Benjamin_Loison 3758405f52 Add a note about the timing percentage of findLatestTreatedCommentsForChannelsBeingTreated.py going backward 2023-01-07 15:35:12 +01:00
Benjamin_Loison 71e4bd95a9 Fix #9: Make sure that in case of error returned by the YouTube Data API v3 the algorithm treats it correctly 
Note that in case of error the algorithm used to skip the received content, as if just no `items` were in it.
 2023-01-06 20:55:32 +01:00
Benjamin_Loison 34bbc216f6 Fix #15: Provide an algorithm to retrieve the list of 100 French channels with most subscribers (and provide it too) 2023-01-06 18:06:00 +01:00
Benjamin_Loison baec8fcb6c #7: Remove remaining undefined behavior due to missing mutex use 2023-01-06 18:00:51 +01:00
Benjamin_Loison 773f86c551 Fix #17: Add to stdout live statistics of the number of comments treated per second 2023-01-06 17:55:16 +01:00
Benjamin_Loison f436007836 Fix #16: Provide an algorithm to determine the progress of retrieving comments for huge YouTube channels 2023-01-06 17:51:00 +01:00
Benjamin_Loison dfbf38b071 #1: Add GNU AGPLv3 license 2023-01-06 16:09:12 +01:00
Benjamin_Loison 292dd8919e Add try/catch around json parser 
As got:
```
terminate called after throwing an instance of 'nlohmann::detail::parse_error'
terminate called recursively
  what():  [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal
terminate called recursively
```
 2023-01-06 00:31:05 +01:00
Benjamin_Loison dab4c8ff69 Modify removeChannelsBeingTreated.py to be more resilient against not existing files in the treatment process 2023-01-04 03:10:28 +01:00
Benjamin_Loison 9d5c9fde2a #2: Add compression to channels/ folder 
Can use following Python script to compress existing uncompressed
`channels/` folder.

```py

import os, shutil

path = 'channels/'

os.chdir(path)

d = next(os.walk('.'))[1]
for channelIndex, channelId in enumerate(d):
    print(f'{channelIndex} / {len(d)}: {channelId}')
    shutil.make_archive(channelId, 'zip', channelId)
    shutil.rmtree(channelId)
```
 2023-01-04 03:06:33 +01:00
Benjamin_Loison f201ae7a91 Make #7: Add multi-threading compatible with my Debian setup 2023-01-04 02:51:40 +01:00
Benjamin_Loison 4cae7e09d1 Add {removeChannelsBeingTreated, findTreatedChannelWithMost{Comments, Subscribers}}.py 2023-01-04 02:41:07 +01:00
Benjamin_Loison e4b4ce21a2 Fix #7: Add multi-threading 2023-01-03 04:56:19 +01:00
Benjamin_Loison a2990c7699 Fix #8: Support comments disabled channels 
Tested with `UCWIdqSQekeGmUWlSFeCiEnA` which treated correctly the 36 comments of the only comments enabled video `3F8dFt8LsXY`.

Note that this commit doesn't support comments disabled channels with more than 20,000 videos.
 2023-01-03 02:56:07 +01:00
Benjamin_Loison 923c14a77b #2: Add data logging 2023-01-02 19:46:32 +01:00
Benjamin_Loison 73a9dea023 Apply astyle formatting to main.cpp 2023-01-02 18:31:16 +01:00
Benjamin_Loison 938ae4b0fb Fix #4: Provide a version relying on the no-key service of https://yt.lemnoslife.com 2023-01-02 18:30:18 +01:00
Benjamin_Loison c50a82df1b Make compatible with Debian 
More precise ly make compatible with `gcc version 10.2.1 20210110 (Debian 10.2.1-6)`
 2023-01-02 18:23:30 +01:00
Benjamin_Loison 36f1fb9e83 Add progression save and use spaces instead of tabs 2022-12-22 06:18:22 +01:00
Benjamin_Loison 934954092a Add time to logging 2022-12-22 05:47:16 +01:00