Make a website with a search engine notably based on the captions extracted #31

New Issue

Benjamin_Loison · 2023-01-25T22:32:37+01:00

Benjamin_Loison commented

2023-01-25 22:32:37 +01:00

Could propose two options:

filtering only on captions
filtering on all data retrieved (~~as an archive with matched files~~, make sure to remove/censor API keys - currently doing so disable the ability to download whole archives)

Could also add channels.txt (done) and logs retrieval (make sure that there isn't anything secret in the latter, it's not the case cf at least 1. and 2., it's not much a problem as it's a small file that we can post-process, I am waiting to have one consequent to verify experimentally that only these two occurrences are problematic).

Could propose two options: - filtering only on captions - filtering on all data retrieved (~~as an archive with matched files~~, make sure to remove/censor API keys - currently doing so disable the ability to download whole archives) Could also add `channels.txt` (done) and logs retrieval (make sure that there isn't anything secret in the latter, it's not the case cf at least [1.](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/5a1df71bb97a7e45242756f895f04d7590f966ca/main.cpp#L711) and [2.](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/5a1df71bb97a7e45242756f895f04d7590f966ca/main.cpp#L716), it's not much a problem as it's a *small* file that we can post-process, I am waiting to have one consequent to verify experimentally that only these two occurrences are problematic).

Benjamin_Loison added the

epic

enhancement

captions

labels 2023-01-25 22:32:37 +01:00

Benjamin_Loison commented

2023-01-30 20:58:30 +01:00

Introduce https://crawler.yt.lemnoslife.com for this purpose.

The plan is to use WebSocket, as it's perfectly adapted and compatible.

Introduce https://crawler.yt.lemnoslife.com for this purpose. The plan is to use WebSocket, as it's perfectly adapted and [compatible](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API#browser_compatibility).

Benjamin_Loison added the

high priority

label 2023-01-30 21:17:48 +01:00

Benjamin_Loison referenced this issue from a commit

2023-02-07 20:15:39 +01:00

#31: Add zip files search

Benjamin_Loison referenced this issue

2023-02-10 17:01:27 +01:00

Try to find a logic to YouTube search concerning captions #27

Benjamin_Loison added this to the 0.0.1 milestone 2023-02-10 17:14:13 +01:00

~~Benjamin_Loison referenced this issue 2023-02-10 20:21:36 +01:00~~

Make a not pre-release release #25

Benjamin_Loison added a new dependency 2023-02-10 20:22:09 +01:00

#25 Make a not pre-release release

Benjamin_Loison commented

2023-02-13 06:02:31 +01:00

Should add vtt parsing to not be limited to line wrapping.

Should add an option to only search through captions.

Should also update findLatestTreatedCommentsForChannelsBeingTreated.py with all features to better evaluate algorithm progress.

Should add `vtt` parsing to not be limited to line wrapping. Should add an option to only search through captions. Should also update [`findLatestTreatedCommentsForChannelsBeingTreated.py`](https://gitea.lemnoslife.com/Benjamin_Loison/YouTube_captions_search_engine/src/commit/fa7da64879b2f8180dcf0b6bd964f9d9af3e2dde/findLatestTreatedCommentsForChannelsBeingTreated.py) with all features to better evaluate algorithm progress.

Benjamin_Loison added the

website

label 2023-02-13 22:56:45 +01:00

Benjamin_Loison referenced this issue from a commit

2023-02-14 00:59:43 +01:00

#31: Add a first search only captions support

Benjamin_Loison commented

2023-02-14 01:07:27 +01:00

Working at PHP level wasn't making much sense:

$searchOnlyCaptions = str_starts_with($msg, 'search-only-captions ');
$msg = substr($msg, strpos($msg, ' ') + 1);

Working at PHP level wasn't making much sense: ```php $searchOnlyCaptions = str_starts_with($msg, 'search-only-captions '); $msg = substr($msg, strpos($msg, ' ') + 1); ```

Benjamin_Loison commented

2023-02-14 01:29:21 +01:00

Could add later on a link directly to the YouTube video timestamp, however as we aren't limited to line wrapping, it's not easy to implement efficiently this feature.

Benjamin_Loison referenced this issue from a commit

2023-02-14 01:32:38 +01:00

#31: Make search within captions not limited by line wrapping

Benjamin_Loison referenced this issue from a commit

2023-02-14 02:00:26 +01:00

Fix #31: Make a website with a search engine notably based on the captions extracted

Benjamin_Loison closed this issue

2023-02-14 02:00:26 +01:00

Benjamin_Loison commented

2023-02-14 02:01:44 +01:00

We could list all occurrences within a video.

Benjamin_Loison reopened this issue

2023-02-14 02:01:44 +01:00

Benjamin_Loison referenced this issue from a commit

2023-02-14 02:56:13 +01:00

Fix #31: List all occurrences of search within video captions

Benjamin_Loison closed this issue

2023-02-14 02:56:13 +01:00

Benjamin_Loison referenced this issue

2023-02-14 16:34:50 +01:00

It seems that the website interleaves user request responses #41

Benjamin_Loison commented

2023-02-17 01:02:41 +01:00

Note that maybe the returned match timestamps aren't as precise as we can (maybe it returns the previous beginning timestamp caption for instance). This should be ideally investigated.

Benjamin_Loison referenced this issue from a commit

2023-03-24 23:46:27 +01:00

#31: Add zip files search

Benjamin_Loison referenced this issue from a commit

2023-03-24 23:46:27 +01:00

#31: Add a first search only captions support

Benjamin_Loison referenced this issue from a commit

2023-03-24 23:46:27 +01:00

#31: Make search within captions not limited by line wrapping

Benjamin_Loison referenced this issue from a commit

2023-03-24 23:46:27 +01:00

Fix #31: Make a website with a search engine notably based on the captions extracted